Develop Custom Datastore

This example shows how to implement a custom datastore for file-based data. Use this framework only when writing your own custom datastore interface. Otherwise, for standard file formats, such as images or spreadsheets, use an existing datastore from MATLAB®. For more information, see Getting Started with Datastore.

Overview

To build your custom datastore interface, use the custom datastore classes and objects. Then, use the custom datastore to bring your data into MATLAB and leverage the MATLAB big data capabilities such as tall, MapReduce, and Hadoop®.

Designing your custom datastore involves inheriting from one or more abstract classes and implementing the required methods. The specific classes and methods you need depend on your processing needs.

Processing Needs

Classes

Datastore for Serial Processing in MATLAB

matlab.io.Datastore

See, Implement Datastore for Serial Processing

Datastore with support for Parallel Computing Toolbox™ and MATLAB Parallel Server™

matlab.io.Datastore and matlab.io.datastore.Partitionable

See, Add Support for Parallel Processing

Datastore with support for Hadoop

matlab.io.Datastore and matlab.io.datastore.HadoopLocationBased

See, Add Support for Hadoop

Datastore with support for shuffling samples in a datastore in random order

matlab.io.Datastore and matlab.io.datastore.Shuffleable

See, Add Support for Shuffling

Start by implementing datastore for serial processing, and then add support for parallel processing, Hadoop, and suffling.

Implement Datastore for Serial Processing

To implement a custom datastore named MyDatastore, create a script MyDatastore.m. The script must be on the MATLAB path and should contain code that inherits from the appropriate class and defines the required methods. The code for creating a datastore for serial processing in MATLAB must:

  • Inherit from the base class matlab.io.Datastore.

  • Define these methods: hasdata, read, reset, and progress.

  • Define additional properties and methods based on your data processing and analysis needs.

For a sample implementation, follow these steps.

StepsImplementation

Inherit from the base class Datastore.

classdef MyDatastore < matlab.io.Datastore
    
    properties (Access = private)
        CurrentFileIndex double
        FileSet matlab.io.datastore.DsFileSet
    end

Add this property to create a datastore on one machine that works seamlessly on another machine or cluster that possibly has a different file system or operating system.

Add methods to get and set this property in the methods section.

    % Property to support saving, loading, and processing of
    % datastore on different file system machines or clusters. 
    % In addition, define the methods get.AlternateFileSystemRoots()
    % and set.AlternateFileSystemRoots() in the methods section. 
    properties(Dependent)
        AlternateFileSystemRoots
    end

Implement the function MyDatastore that creates the custom datastore.

    methods % begin methods section
        
        function myds = MyDatastore(location,altRoots)
            myds.FileSet = matlab.io.datastore.DsFileSet(location,...
                'FileExtensions','.bin', ...
                'FileSplitSize',8*1024);
            myds.CurrentFileIndex = 1;
            
            if nargin == 2
                 myds.AlternateFileSystemRoots = altRoots;
            end

            reset(myds);
        end

Implement the hasdata method.

        function tf = hasdata(myds)
            % Return true if more data is available.
            tf = hasfile(myds.FileSet);
        end

Implement the read method.

This method uses MyFileReader, which is a function that you must create to read your proprietary file format .

See Create Function to Read Your Proprietary File Format.

        function [data,info] = read(myds)
            % Read data and information about the extracted data.
            if ~hasdata(myds)
                error(sprintf(['No more data to read.\nUse the reset ',...
                    'method to reset the datastore to the start of ' ,...
                    'the data. \nBefore calling the read method, ',...
                    'check if data is available to read ',...
                    'by using the hasdata method.']))
                
            end
            
            fileInfoTbl = nextfile(myds.FileSet);
            data = MyFileReader(fileInfoTbl);
            info.Size = size(data);
            info.FileName = fileInfoTbl.FileName;
            info.Offset = fileInfoTbl.Offset;
            
            % Update CurrentFileIndex for tracking progress
            if fileInfoTbl.Offset + fileInfoTbl.SplitSize >= ...
                    fileInfoTbl.FileSize
                myds.CurrentFileIndex = myds.CurrentFileIndex + 1 ;
            end
            
            
        end

Implement the reset method.

        function reset(myds)
            % Reset to the start of the data.
            reset(myds.FileSet);
            myds.CurrentFileIndex = 1;
        end

Define the methods to get and set the AlternateFileSystemRoots property.

You must reset the datastore in the set method.

        % Before defining these methods, add the AlternateFileSystemRoots 
        % property in the properties section 
    
        % Getter for AlternateFileSystemRoots property
        function altRoots = get.AlternateFileSystemRoots(myds)
            altRoots = myds.FileSet.AlternateFileSystemRoots;
        end

        % Setter for AlternateFileSystemRoots property
        function set.AlternateFileSystemRoots(myds,altRoots)
            try
              % The DsFileSet object manages the AlternateFileSystemRoots
              % for your datastore
              myds.FileSet.AlternateFileSystemRoots = altRoots;

              % Reset the datastore
              reset(myds);  
            catch ME
              throw(ME);
            end
        end
        
    end

Implement the progress method.

    methods (Hidden = true)
        function frac = progress(myds)
            % Determine percentage of data read from datastore
            if hasdata(myds) 
               frac = (myds.CurrentFileIndex-1)/...
                             myds.FileSet.NumFiles; 
            else 
              frac = 1;  
            end 
        end
    end 

Implement the copyElement method when you use the DsFileSet object as a property in your datastore.

    methods (Access = protected)
        % If you use the DsFileSet object as a property, then 
        % you must define the copyElement method. The copyElement
        % method allows methods such as readall and preview to 
        % remain stateless
        function dscopy = copyElement(ds)
            dscopy = copyElement@matlab.mixin.Copyable(ds);
            dscopy.FileSet = copy(ds.FileSet);
        end
    end

End the classdef section.

end

Create Function to Read Your Proprietary File Format

The implementation of the read method of your custom datastore uses a function called MyFileReader. You must create this function to read your custom or proprietary data. Build this function using DsFileReader object and its methods. For instance, create a function that reads binary files.

function data = MyFileReader(fileInfoTbl)
% create a reader object using the FileName
reader = matlab.io.datastore.DsFileReader(fileInfoTbl.FileName);

% seek to the offset
seek(reader,fileInfoTbl.Offset,'Origin','start-of-file');

% read fileInfoTbl.SplitSize amount of data
data = read(reader,fileInfoTbl.SplitSize);
end

Add Support for Parallel Processing

To add support for parallel processing with Parallel Computing Toolbox and MATLAB Parallel Server, update your implementation code in MyDatastore.m to:

For a sample implementation, follow these steps.

StepsImplementation

Update the classdef section to inherit from the Partitionable class.

classdef MyDatastore < matlab.io.Datastore & ...
                       matlab.io.datastore.Partitionable
     .
     .
     . 
  

Add the definition for partition to the methods section.

 methods
     .
     .
     .
     function subds = partition(myds,n,ii)
         subds = copy(myds);
         subds.FileSet = partition(myds.FileSet,n,ii);
         reset(subds);         
     end
 end

Add definition for maxpartitions to the methods section.

 methods (Access = protected)
     function n = maxpartitions(myds) 
       n = maxpartitions(myds.FileSet); 
     end     
 end 

End classdef.

end  

Add Support for Hadoop

To add support for Hadoop, update your implementation code in MyDatastore.m to:

For a sample implementation, follow these steps.

StepsImplementation

Update the classdef section to inherit from the HadoopLocationBased class.

classdef MyDatastore < matlab.io.Datastore & ...
                       matlab.io.datastore.HadoopLocationBased 
     .
     .
     . 
  

Add the definition for getLocation, initializeDatastore, and isfullfile(optional) to the methods section.

 methods (Hidden = true)
     .
     .
     .   

     function initializeDatastore(myds,hadoopInfo)
        import matlab.io.datastore.DsFileSet;
        myds.FileSet = DsFileSet(hadoopInfo,...
             'FileSplitSize',myds.FileSet.FileSplitSize);
        reset(myds);         
     end 
     
     function loc = getLocation(myds)
        loc = myds.FileSet;         
     end 
     
     % isfullfile method is optional
     function tf = isfullfile(myds)
        tf = isequal(myds.FileSet.FileSplitSize,'file');          
     end 
 
 end

End the classdef section.

end

Add Support for Shuffling

To add support for shuffling, update your implementation code in MyDatastore.m to:

For a sample implementation, follow these steps.

StepsImplementation

Update the classdef section to inherit from the Shuffleable class.

classdef MyDatastore < matlab.io.Datastore & ...
                      matlab.io.datastore.Shuffleable 
     .
     .
     . 
  

Add the definition for shuffle to the existing methods section.

  methods

        % previously defined methods
        .
        .
        . 
   
        function dsNew = shuffle(ds)
            % dsNew = shuffle(ds) shuffles the files and the
            % corresponding labels in the datastore.
            
            % Create a copy of datastore
            dsNew = copy(ds);
            dsNew.Datastore = copy(ds.Datastore);
            fds = dsNew.Datastore;
            
            % Shuffle files and corresponding labels
            numObservations = dsNew.NumObservations;
            idx = randperm(numObservations);
            fds.Files = fds.Files(idx);
            dsNew.Labels = dsNew.Labels(idx);
        end

  end


End the classdef section.

end

Validate Custom Datastore

After following instructions presented here, the implementation step of your custom datastore is complete. Before using this custom datastore, qualify it using the guidelines presented in Testing Guidelines for Custom Datastores.

See Also

| | | | |

Related Topics