File Exchange

image thumbnail

newmatic

version 0.2.2 (211 KB) by Keith Ma
Create new MAT-files optimized for partial reading and writing of large arrays

12 Downloads

Updated 05 Mar 2020

GitHub view license on GitHub

See the demo at: http://www.allnans.com/newmatic/newmatic_demo.html

About
--------

This tool solves a few problems I ran into when using "matfile" to manage partial IO, namely:

+ *Inability to control data chunking*: Arrays in MAT-files are saved in "chunks". The most natural and performant way to read/write to these arrays is to access one chunk at a time. MATLAB chooses the chunk size automatically and it may (read: almost certainly) not be the most efficient choice for your workload. The recommended solution is to use third-party tools to
"repack" your MAT-file after you create it (see section "Accelerate Save and Load Operations for Version 7.3 MAT-Files" at https://www.mathworks.com/help/matlab/import_export/mat-file-versions.html). This should make us all sad.

+ *Repetitive and cryptic array size initialization (see: https://www.mathworks.com/help/matlab/import_export/troubleshooting-file-size-increases-unexpectdly-when-growing-an-array.html) Assigning a value to the very last element in the array serves a similar purpose as pre-allocating arrays for in-memory variables. The recommended approach is fine, but not easy for the uninitiated to parse and ugly to look at.

Fortunately, (a) MAT-files are HDF5 formatted under the hood, and (b) MATLAB includes utilities for working with HDF5 files directly (i.e., without any external tools). newmatic provides an alternative interface for creating "customized" MAT-files that perform well for partial IO.

Additional notes:

+ newmatic only allocates arrays. If you want to include other data types (strings, cell arrays, whatever), you can add them to the file created by newmatic in the usual way.

+ This initial version of newmatic depends on the system CLI h5repack (see: https://support.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Repack), a future update will drop this dependency in favor of the native MATLAB hdf5 interface.

Example Usage
---------------------

There are two key functions in this package:

+ newmatic_variable is used to define the name, data type, array size, and chunk size for variables created by newmatic

+ newmatic creates a MAT-file with the specified variables

Here is a small example demonstrating how to create a MAT-file with two arrays (created by newmatic) and a cell array (not created by newmatic).

mat = newmatic(...
'my-mat-file.mat', ...
newmatic_variable('x', 'double', [1000, 1000, 20], [1000, 1000, 1]), ...
newmatic_variable('y', 'double', [1000, 10, 10], [5000, 10, 10]));
mat.z = {'a', 'cell', 'array'};

Cite As

Keith Ma (2020). newmatic (https://github.com/keithfma/newmatic), GitHub. Retrieved .

Comments and Ratings (3)

Keith: Thanks! I downloaded your latest version and it works fine

Keith Ma

@Stephen: It looks like the syntax I used is not supported everywhere (e.g., https://www.mathworks.com/matlabcentral/answers/363587-error-to-this-line-functions-cannot-be-indexed-using-or-indexing). I switch to a more verbose, but less fragile syntax in version 0.2.2 . Good luck!

I get the following error when running 'newmatic_demo'
Error: Functions cannot be indexed using {} or . indexing.

Error in newmatic_demo (line 66)
native_complete_file_size = dir(native_complete_file).bytes/1024/1024;
I am running 2018b but I do have 2019b on another machine. Please advise

Updates

0.2.2

Improve compatibility in demo script

0.2.1

Logical variables are read as correct datatype

0.2.0

Drop h5repack dependency in favor of MATLAB's native HDF5 utilities. This change also yields a bug performance improvement for large files (a critical use case).

0.1.1

Improved FEX description

MATLAB Release Compatibility
Created with R2019b
Compatible with any release
Platform Compatibility
Windows macOS Linux
Tags Add Tags