File Exchange

image thumbnail

Save MAT files more quickly

version 1.0 (1.84 KB) by

Saves large numeric arrays more quickly than Matlab's built-in save command

4.71429
8 Ratings

50 Downloads

Updated

View License

When working with large arrays, Matlab's 'save' command can be very slow, presumably because Matlab attempts to use compression. This function provides a much faster alternative, at the cost of larger files. It only improves speed for variables that are plain arrays; elements of structures or cell arrays are not altered.

Example:
% First make some data
>> ops = struct('algorithm', 'greedy');
>> A = int32(randi(20, 1000, 1200, 40));
>> B = randn(500, 1800, 60);
% Now save it
>> tic; save /tmp/test ops A B; toc
Elapsed time is 22.980294 seconds.
>> tic; savefast /tmp/test ops A B; toc
Elapsed time is 0.571098 seconds.

Happy saving!

Comments and Ratings (22)

adi

adi (view profile)

Hi! does it work with version v7.3? I need to save and load parts of a variable from a mat file, and the files are larger than 2 Gib.

Daniel

Daniel (view profile)

How can we append files to data already saved?

He Qihong

It doesn't work correctly in Matlab2014b. It will makes Matlab2014b crash.

Benedikt

Coming back to the complex-number issue discussed by Andre and Raj. A small addition e.g.
isnum(cellfun(@(x)~isreal(x),vars))=0

saves all complex numbers using the default (slow) command instead of hdf5 without compression. This would have saved me some trouble. Maybe add a comment to the file description as well.

All in all the plugin is very nice.

Tim Holy

Tim Holy (view profile)

@arnold, does it work if you list variables explicitly? It's possible it's just missing a variant that saves all the variables in your workspace.

arnold

arnold (view profile)

I don't seem to understand the syntax, it doesn't behave like 'save'.

savefast('test.mat');

Does produce a file with the appropriate name but it doesn't seem to contain anything. Loading it does not restore any of the workspace variables.

Tim Holy

Tim Holy (view profile)

@zhang, the problem is that '-v6' has some serious limitations, for example it can't store arrays bigger than 2GB. It's also not as portable to other languages, etc; '-v7.3' is based on HDF5, so many different languages can read the files.

zhang

zhang (view profile)

This is unnecessary. Use the build-in save(filename, variables,'-v6') is even much faster than this file.

e.g., test showed that:
if not use '-v6', it takes 7.7 seconds for a file.
Using this file, it's 0.73 seconds.
But using the '-v6' option, it's ONLY 0.17 seconds!

Tim Holy

Tim Holy (view profile)

@yaron, off the top of my head I have no idea. What is the error message, and what are version(s) of Matlab on each?

yaron

yaron (view profile)

Great contribution - but I have an issue with using it on a mac. For some reason, while the files load on a PC, the save fast .mat files cause an error on my mac. Any idea for how to fix this?

Tim Holy

Tim Holy (view profile)

Ali, it can only handle certain types, I didn't try to reverse engineer the entire format.

But see the comment from Raj, immediately below yours, about hlp_serialize.

Ali

Ali (view profile)

Thank you for providing this code. I tried to use it for fast saving of complex-sparse matrices and it ends with the following error:

Error using hdf5lib2
The HDF5 library encountered an unknown error.
Error in H5D.write (line 72)

do you think it there is a solution to this error or it is not applicable to large complex sparse data format?
Thanks

Raj

Raj (view profile)

Andre..may be your problem might be resolved if you first serialize the data using hlp_serialize(https://www.mathworks.com/matlabcentral/fileexchange/34564-fast-serializedeserialize) and store using savefast function and later deserialize the data. This worked even when I had complex vector is a structure...definitely improved saving files with marginal increase in memory!

Raj

Raj (view profile)

Andre Kuehne

Andre Kuehne

Great submission, however it will not work for complex (real/imaginary) datatypes! The Matlab highlevel hdf5 routines apparently do not support complex datatypes and thus only the real part is saved. This is probably not easily fixed, as you would need to utilize low-level hdf5 functions.

Tim Holy

Tim Holy (view profile)

Robin, that's good to know about parfor. Thanks for providing a version that works under such circumstances.

Robin

Robin (view profile)

This is great. One problem was that 'evalin' doesn't work properly within a parfor loop. So I made a minor modification to allow 'save -struct' style saving which works fine in parfor:
https://gist.github.com/robince/5974172

Tim Holy

Tim Holy (view profile)

Hi Jeremy,

Your performance benefit will depend a lot on your hard drive configuration. In my case I was writing to an SSD RAID. But on my laptop (with a single SSD) I can write in about 2.1 sec, which is still a 10x benefit. What kind of performance benefit are you getting?

Jeremy

Jeremy (view profile)

I am not seeing anywhere close to that save time using your A,B,ops sample. I'm running 2012b.

Your example is showing that you saved ~600MB in .57 sec. That seems a bit suspect as it is over 1GB/sec.

Robin

Robin (view profile)

MATLAB Release
MATLAB 8.0 (R2012b)

Download apps, toolboxes, and other File Exchange content using Add-On Explorer in MATLAB.

» Watch video