Code covered by the BSD License  

Highlights from
deprecated -- Light Field Toolbox v0.2 -- v0.3 now available

image thumbnail

deprecated -- Light Field Toolbox v0.2 -- v0.3 now available

by

 

26 Apr 2013 (Updated )

A set of tools for working with light field (aka plenoptic) imagery in Matlab

LFUtilDecodeLytroFolder( InputPath, FileOptions, DecodeOptions, RectOptions )
% LFUtilDecodeLytroFolder - decode and optionally colour correct and rectify Lytro light fields
%
% Usage:
%
%     LFUtilDecodeLytroFolder
%     LFUtilDecodeLytroFolder( InputPath )
%     LFUtilDecodeLytroFolder( InputPath, FileOptions, DecodeOptions, RectOptions )
%     LFUtilDecodeLytroFolder( InputPath, [], [], RectOptions )
%
%
% All parameters are optional and take on default values as set in the "Defaults" section at the top
% of the implementation. As such, this can be called as a function or run directly by editing the
% code. When calling as a function, pass an empty array "[]" to omit a parameter.
%
% As released, the default values are set up to match the naming conventions of LFP Reader v2.0.0.
%
% This function demonstrates decoding and optionally colour-correction and rectification of 2D
% lenselet images into 4D light fields. It recursively crawls through a prescribed folder and its
% subfolders, operating on each light field. It can be used incrementally: previously-decoded light
% fields can be subsequently colour-corected, rectified, or both. Previously-completed tasks will
% not be re-applied. A filename pattern can be provided to work on individual files. All paths and
% naming are configurable.
%
% Decoding and rectification follow the process described in:
%
% [1] D. G. Dansereau, O. Pizarro, and S. B. Williams, "Decoding, calibration and rectification for
% lenselet-based plenoptic cameras," in Computer Vision and Pattern Recognition (CVPR), IEEE
% Conference on. IEEE, Jun 2013.
%
% Decoding requires that an appropriate database of white images be created using
% LFUtilProcessWhiteImages. Rectification similarly requires a calibration database be created using
% LFUtilProcessCalibrations.
% 
% To decode a single light field, it is simplest to include a file specification in InputPath (see
% below). It is also possible to call LFDecodeLytroImage directly.
%
% Colour correction employs the metadata associated with each Lytro picture. It also applies
% histogram-based contrast adjustment. It calls the functions LFColourCorrect and LFHistEqualize.
%
% Rectification employs a calibration info file to rectify the light field, correcting for lens
% distortion, making pixels square, and yielding an intrinsics matrix which allows easy conversion
% from a pixel index [i,j,k,l] to a ray [s,t,u,v]. A calibration info file is generated by
% processing a series of checkeboard images, following the calibration procedure described in
% LFToolbox.pdf. A calibration only applies to one specific camera at one specific zoom and focus
% setting, and decoded using one specific lenselet grid model. The tool LFUtilProcessCalibrations is
% used to build a database of rectifications, and LFSelectFromDatabase isused to select a
% calibration appropriate to a given light field.
%
% This function was written to deal with Lytro imagery, but adapting it to operate with other
% lenselet-based cameras should be straightforward. For more information on the decoding process,
% refer to LFDecodeLenseletImageSimple, [1], and LFToolbox.pdf.
%
% Some optional parameters are not used or documented at this level -- see each of LFCalRectifyLF,
% LFDecodeLytroImage, LFDecodeLenseletImageSimple, and LFColourCorrect for further information.
%
%
% Inputs -- all are optional, see code below for default values :
%
%     InputPath :  Path to folder containing light fields -- note the function operates recursively,
%                  i.e. it will search all sub-folders for light fields. Inputpath may optionally
%                  include a base filename specification. To do this, pass a cell array, in which
%                  the first element is the input path, and the second is the filename spec. Note
%                  that the file search is still recursive, even if a filename spec is provided. See
%                  the examples, below.
%
%
%     FileOptions : struct controlling file naming and saving
%
%               .SaveResult : Set to false to perform a "dry run"
%
%                .ForceRedo : If true previous results are ignored and decoding starts from scratch
%
%         .SaveFnamePattern : String defining the pattern used in generating the output filename;
%                             sprintf is used to complete this pattern, such that %s gets replaced
%                             with the base name of the input light field
%
%        .ThumbFnamePattern : As with SaveFnamePattern, defines the name of the output thumbnail
%                             image
%
%
%     DecodeOptions : struct controlling the decoding process
%
%                    .OptionalTasks : Cell array containing any combination of 'ColourCorrect' and
%                                     'Rectify'; an empty array "{}" means no additional tasks are
%                                     requested; case sensitive
%
%        .LenseletImageFnamePattern : Pattern used to locate input files -- the pattern %s stands in
%                                     for the base filename
%
%                 .ColourHistThresh : Threshold used by LFHistEqualize in optional colour correction
%
%           .WhiteImageDatabasePath : Full path to the white images database, as created by
%                                     LFUtilProcessWhiteImages
%
%
%     RectOptions : struct controlling the optional rectification process
%
%         .CalibrationDatabaseFname : Full path to the calibration file database, as created by
%                                     LFUtilProcessCalibrations;
%
%
% Example:
%
%   LFUtilDecodeLytroFolder
% 
%     Run from the top level of the 'Samples' folder will decode all the light fields in all the
%     sub-folders, with default settings as set up in the opening section of the code. The
%     calibration database created by LFUtilProcessWhiteImages is expected to be in
%     'Cameras/CaliCalibrationDatabase.mat' by default.
% 
%   LFUtilDecodeLytroFolder([], [], struct('OptionalTasks', 'ColourCorrect'))
%   LFUtilDecodeLytroFolder('.', [], struct('OptionalTasks', 'ColourCorrect'))
%
%     Either of these, run from the top level of the 'Samples' folder will decode and colour correct
%     all light fields in that folder and its sub-folders.
%
%   DecodeOptions.OptionalTasks = {'ColourCorrect', 'Rectify'};
%   LFUtilDecodeLytroFolder([], [], DecodeOptions)
% 
%     Will perform both colour correction and rectification.
% 
%   LFUtilDecodeLytroFolder({'Images','IMG_0002'})
%   LFUtilDecodeLytroFolder({'Images','*_0002'})
%   LFUtilDecodeLytroFolder({'.', '*2'})
%
%     Any of these, run from the top level of the 'Samples' folder, will only decode
%     IMG_0002__frame.raw. Note that if more than one file exists in the tree with a filename base
%     ending in '2', the third example will result in all of them being decoded.
% 
%
% See also:  LFDecodeLytroImage, LFDecodeLenseletImageSimple, LFSelectFromDatabase,
% LFUtilProcessWhiteImages, LFUtilProcessCalibrations, LFUtilCalLenseletCam, LFColourCorrect,
% LFCalRectifyLF

% Part of LF Toolbox v0.2 released 27-May-2013
% Copyright (c) 2013, Donald G. Dansereau

function LFUtilDecodeLytroFolder( InputPath, FileOptions, DecodeOptions, RectOptions )

%---Defaults---
InputPath = LFDefaultVal( 'InputPath', '.' );

FileOptions = LFDefaultField('FileOptions', 'SaveResult', true);
FileOptions = LFDefaultField('FileOptions', 'ForceRedo', false);
FileOptions = LFDefaultField('FileOptions', 'SaveFnamePattern', '%s__Decoded.mat');
FileOptions = LFDefaultField('FileOptions', 'ThumbFnamePattern', '%s__Decoded_Thumb.png');

DecodeOptions = LFDefaultField('DecodeOptions', 'OptionalTasks', {}); % 'ColourCorrect', 'Rectify'
DecodeOptions = LFDefaultField('DecodeOptions', 'LenseletImageFnamePattern', '%s__frame.raw');
DecodeOptions = LFDefaultField('DecodeOptions', 'ColourHistThresh', 0.01);
DecodeOptions = LFDefaultField(...
    'DecodeOptions', 'WhiteImageDatabasePath', fullfile('Cameras','WhiteImageDatabase.mat'));

RectOptions = LFDefaultField(...
    'RectOptions', 'CalibrationDatabaseFname', fullfile('Cameras','CalibrationDatabase.mat'));

% Used to decide if two lenselet grid models are "close enough"... if they're not a warning is raised
RectOptions = LFDefaultField( 'RectOptions', 'MaxGridModelDiff', 1e-8 );

% Massage a single-element OptionalTasks list to behave as a cell array
while( ~iscell(DecodeOptions.OptionalTasks) )
    DecodeOptions.OptionalTasks = {DecodeOptions.OptionalTasks};
end

% Break any wildcards / file specifications out of InputPath
InputFileSpec = '*'; % gets overriden below, if a file spec is provided
if( iscell(InputPath) )
    if( numel(InputPath) >= 2 )
        InputFileSpec = InputPath{2};
    end
    InputPath = InputPath{1};
end

%---Build a regular expression for stripping the base filename out of the full raw filename---
BaseFnamePattern = regexp(DecodeOptions.LenseletImageFnamePattern, '%s', 'split');
BaseFnamePattern = cell2mat({BaseFnamePattern{1}, '(.*)', BaseFnamePattern{2}});

%---Crawl folder structure locating raw lenselet images---
fprintf('Searching for raw lenselet images...\n');
FileList = LFFindFilesRecursive( InputPath, sprintf(DecodeOptions.LenseletImageFnamePattern, InputFileSpec) );
fprintf('Found :\n');
disp(FileList')

%---Process each raw lenselet file---
% Store options so we can reset them for each file
OrigDecodeOptions = DecodeOptions;
OrigRectOptions = RectOptions;

for( iFile = 1:length(FileList) )

    %---Start from orig options, avoids values bleeding between iterations---
    DecodeOptions = OrigDecodeOptions;
    RectOptions = OrigRectOptions;
    
    %---Find current / base filename---
    LFFnameBase = FileList{iFile};
    LFFnameBase = fullfile(InputPath, LFFnameBase);
    
    LFFnameBase = regexp(LFFnameBase, BaseFnamePattern, 'tokens');
    LFFnameBase = LFFnameBase{1}{1};
    fprintf('\n---%s [%d / %d]...\n', LFFnameBase, iFile, length(FileList));
    
    %---Decode---
    fprintf('Decoding...\n');
    
    % First check if a decoded file already exists
    [SDecoded, FileExists, CompletedTasks, TasksRemaining, SaveFname] = CheckIfExists( ...
        LFFnameBase, DecodeOptions, FileOptions.SaveFnamePattern, FileOptions.ForceRedo );
    
    if( ~FileExists )
        % No previous result, decode
        [LF, LFMetadata, WhiteImageMetadata, LenseletGridModel, DecodeOptions] = ...
            LFDecodeLytroImage( LFFnameBase, DecodeOptions );
        fprintf('Decode complete\n');
        CompletedTasks = {'Decode'};
    elseif( isempty(TasksRemaining) )
        % File exists, and nothing more to do
        continue;
    else
        % File exists and tasks remain: unpack previous decoding results
        [LF, LFMetadata, WhiteImageMetadata, LenseletGridModel, DecodeOptions] = LFStruct2Var( ...
            SDecoded, 'LF', 'LFMetadata', 'WhiteImageMetadata', 'LenseletGridModel', 'DecodeOptions' );
        clear SDecoded
    end
    
    %---Display image info---
    fprintf('\nWhite image / LF Picture:\n');
    fprintf('%s, %s\n', DecodeOptions.WhiteImageInfo.Fname, LFFnameBase);
    fprintf('Serial:\t%s\t%s\n', WhiteImageMetadata.SerialData.camera.serialNumber, LFMetadata.SerialData.camera.serialNumber);
    fprintf('Lambda:\t%d\t\t%d\n', WhiteImageMetadata.devices.lens.infinityLambda, LFMetadata.devices.lens.infinityLambda);
    fprintf('Zoom:\t%d\t\t%d\n', WhiteImageMetadata.devices.lens.zoomStep, LFMetadata.devices.lens.zoomStep);
    fprintf('Focus:\t%d\t\t%d\n\n', WhiteImageMetadata.devices.lens.focusStep, LFMetadata.devices.lens.focusStep);
    
    %---Display thumbnail---
    Thumb = DispThumb(LF, LFFnameBase, CompletedTasks);
    
    %---Optionally colour correct---
    if( ismember( 'ColourCorrect', TasksRemaining ) )
        LF = ColourCorrect( LF, LFMetadata, DecodeOptions );
        CompletedTasks = [CompletedTasks, 'ColourCorrect'];
        fprintf('Done\n');
        
        %---Display thumbnail---
        Thumb = DispThumb(LF, LFFnameBase, CompletedTasks);
    end
    
    %---Optionally rectify---
    if( ismember( 'Rectify', TasksRemaining ) )
        [LF, RectOptions] = Rectify( LF, LFMetadata, DecodeOptions, RectOptions, LenseletGridModel );
        CompletedTasks = [CompletedTasks, 'Rectify'];
        
        %---Display thumbnail---
        Thumb = DispThumb(LF, LFFnameBase, CompletedTasks);
    end
    
    %---Check that all tasks are completed---
    UnrecognizedTasks = find(~ismember(TasksRemaining, CompletedTasks));
    if( ~isempty(UnrecognizedTasks) )
        UnrecTaskList = [];
        for( i=UnrecognizedTasks )
            UnrecTaskList = [UnrecTaskList, ' ', TasksRemaining{UnrecognizedTasks}];
        end
        warning(['Unrecognized tasks requested in DecodeOptions.OptionalTasks: ', UnrecTaskList]);
    end
    
    %---Optionally save---
    if( FileOptions.SaveResult )
        if( isfloat(LF) )
            LF = uint16( LF .* double(intmax('uint16')) );
        end
        ThumbFname = sprintf(FileOptions.ThumbFnamePattern, LFFnameBase);
        fprintf('Saving to:\n\t%s,\n\t%s...\n', SaveFname, ThumbFname);
        TimeStamp = datestr(now,'ddmmmyyyy_HHMMSS');
        GeneratedByInfo = struct('mfilename', mfilename, 'time', TimeStamp, 'VersionStr', 'v0.2 released 27-May-2013');
        
        save('-v7.3', SaveFname, 'GeneratedByInfo', 'LF', 'LFMetadata', 'WhiteImageMetadata', 'LenseletGridModel', 'DecodeOptions', 'RectOptions');
        imwrite(Thumb, ThumbFname);
    end
end
end

%---------------------------------------------------------------------------------------------------
function  [SDecoded, FileExists, CompletedTasks, TasksRemaining, SaveFname] = ...
    CheckIfExists( LFFnameBase, DecodeOptions, SaveFnamePattern, ForceRedo )

SDecoded = [];
FileExists = false;
SaveFname = sprintf(SaveFnamePattern, LFFnameBase);

if( ~ForceRedo && exist(SaveFname, 'file') )
    %---Task previously completed, check if there's more to do---
    FileExists = true;
    fprintf( '    %s already exists\n', SaveFname );
    CompletedTasks = {'Decode'}; % at least this much is done
    
    PrevDecodeOptions = load( SaveFname, 'DecodeOptions' );
    PrevOptionalTasks = PrevDecodeOptions.DecodeOptions.OptionalTasks;
    CompletedTasks = [CompletedTasks, PrevOptionalTasks];
    TasksRemaining = find(~ismember(DecodeOptions.OptionalTasks, PrevOptionalTasks));
    if( ~isempty(TasksRemaining) )
        %---Additional tasks remain---
        TasksRemaining = {DecodeOptions.OptionalTasks{TasksRemaining}};  % by name
        fprintf('    Additional tasks remain, loading existing file...\n');
        
        SDecoded = load( SaveFname );
        AllTasks = [SDecoded.DecodeOptions.OptionalTasks, TasksRemaining];
        SDecoded.DecodeOptions.OptionalTasks = AllTasks;
        
        %---Convert to float as this is what subsequent operations require---
        OrigClass = class(SDecoded.LF);
        SDecoded.LF = cast( SDecoded.LF, SDecoded.DecodeOptions.Precision ) ./ ...
            cast( intmax(OrigClass), SDecoded.DecodeOptions.Precision );
        fprintf('Done\n');
    else
        %---No further tasks... move on---
        fprintf( '    No further tasks requested\nProceeding to next file\n');
        TasksRemaining = {};
    end
else
    %---File doesn't exist, all tasks remain---
    TasksRemaining =  DecodeOptions.OptionalTasks;
    CompletedTasks = {};
end
end

%---------------------------------------------------------------------------------------------------
function Thumb = DispThumb( LF, CurFname, CompletedTasks)
Thumb = squeeze(LF(floor(end/2),floor(end/2),:,:,:)); % including weight channel for hist equalize
Thumb = uint8(LFHistEqualize(Thumb).*double(intmax('uint8')));
Thumb = Thumb(:,:,1:3); % strip off weight channel
LFDispSetup(Thumb);
Title = CurFname;

for( i=1:length(CompletedTasks) )
    Title = [Title, ', ', CompletedTasks{i}];
end

title(Title, 'Interpreter', 'none');
drawnow
end

%---------------------------------------------------------------------------------------------------
function LF = ColourCorrect( LF, LFMetadata, DecodeOptions )
fprintf('Applying colour correction... ');

%---Format the Lytro metadata as required by LFColourCorrect---
ColMatrix = reshape(LFMetadata.image.color.ccmRgbToSrgbArray, 3,3);
ColBalance = [...
    LFMetadata.image.color.whiteBalanceGain.r, ...
    LFMetadata.image.color.whiteBalanceGain.gb, ...
    LFMetadata.image.color.whiteBalanceGain.b ];
Gamma = LFMetadata.image.color.gamma^0.5;

%---Weight channel is not used by colour correction, so strip it out---
LFWeight = LF(:,:,:,:,4);
LF = LF(:,:,:,:,1:3);

%---Apply the color conversion and saturate---
LF = LFColourCorrect( LF, ColMatrix, ColBalance, Gamma );

%---Put the weight channel back---
LF(:,:,:,:,4) = LFWeight;

%---Apply histogram-based contrast adjustment---
LF = LFHistEqualize( LF, DecodeOptions.ColourHistThresh );

end

%---------------------------------------------------------------------------------------------------
function [LF, RectOptions] = Rectify( LF, LFMetadata, DecodeOptions, RectOptions, LenseletGridModel )
fprintf('Applying rectification... ');
%---Load cal info---
fprintf('Selecting calibration...\n');

DesiredCam = struct('CamSerial', LFMetadata.SerialData.camera.serialNumber, ...
    'ZoomStep', LFMetadata.devices.lens.zoomStep, ...
    'FocusStep', LFMetadata.devices.lens.focusStep );
CalFileInfo = LFSelectFromDatabase( DesiredCam, RectOptions.CalibrationDatabaseFname );
PathToDatabase = fileparts( RectOptions.CalibrationDatabaseFname );
RectOptions.CalInfoFname = CalFileInfo.Fname;
CalInfo = LFReadMetadata( fullfile(PathToDatabase, RectOptions.CalInfoFname) );
fprintf('Loading %s\n', RectOptions.CalInfoFname);

%---Check that the decode options and calibration info are a good match---
fprintf('\nCalibration / LF Picture (ideally these match exactly):\n');
fprintf('Serial:\t%s\t%s\n', CalInfo.CamInfo.CamSerial, LFMetadata.SerialData.camera.serialNumber);
fprintf('Zoom:\t%d\t\t%d\n', CalInfo.CamInfo.ZoomStep, LFMetadata.devices.lens.zoomStep);
fprintf('Focus:\t%d\t\t%d\n\n', CalInfo.CamInfo.FocusStep, LFMetadata.devices.lens.focusStep);

if( ~strcmp(CalInfo.CamInfo.CamSerial, LFMetadata.SerialData.camera.serialNumber) )
    warning('Calibration is for a different camera, rectification may be invalid.');
end
if( CalInfo.CamInfo.ZoomStep ~= LFMetadata.devices.lens.zoomStep || ...
        CalInfo.CamInfo.FocusStep ~= LFMetadata.devices.lens.focusStep )
    warning('Zoom / focus mismatch -- for significant deviations rectification may be invalid.');
end

if( ~all(abs(struct2array(CalInfo.LenseletGridModel) - struct2array(LenseletGridModel)) < RectOptions.MaxGridModelDiff) )
    warning(['Lenselet grid models differ -- ideally the same grid model and white image are ' ...
        ' used to decode during calibration and rectification']);
end

%---Perform rectification---
[LF, RectOptions] = LFCalRectifyLF( LF, CalInfo, RectOptions );
end

Contact us