When working with large arrays, Matlab's 'save' command can be very slow, presumably because Matlab attempts to use compression. This function provides a much faster alternative, at the cost of larger files. It only improves speed for variables that are plain arrays; elements of structures or cell arrays are not altered.
% First make some data
>> ops = struct('algorithm', 'greedy');
>> A = int32(randi(20, 1000, 1200, 40));
>> B = randn(500, 1800, 60);
% Now save it
>> tic; save /tmp/test ops A B; toc
Elapsed time is 22.980294 seconds.
>> tic; savefast /tmp/test ops A B; toc
Elapsed time is 0.571098 seconds.
Surely the fact that it cannot handle complex data should be mentioned in the documentation, preferably in bold text and on the first line?
Having happily used it on only real valued data for the last year, I've just wasted more time than I am willing to describe here conducting analysis on what I thought was the absolute value of a complex number, only to discover it's the real component only.
This is otherwise a fantastic script, and a great contribution to the community, but I have spent the last few days thinking I was going insane, before I was finally able to track down the source of error. Thankfully before publication...
Hi! does it work with version v7.3? I need to save and load parts of a variable from a mat file, and the files are larger than 2 Gib.
How can we append files to data already saved?
It doesn't work correctly in Matlab2014b. It will makes Matlab2014b crash.
Coming back to the complex-number issue discussed by Andre and Raj. A small addition e.g.
saves all complex numbers using the default (slow) command instead of hdf5 without compression. This would have saved me some trouble. Maybe add a comment to the file description as well.
All in all the plugin is very nice.
@arnold, does it work if you list variables explicitly? It's possible it's just missing a variant that saves all the variables in your workspace.
I don't seem to understand the syntax, it doesn't behave like 'save'.
Does produce a file with the appropriate name but it doesn't seem to contain anything. Loading it does not restore any of the workspace variables.
@zhang, the problem is that '-v6' has some serious limitations, for example it can't store arrays bigger than 2GB. It's also not as portable to other languages, etc; '-v7.3' is based on HDF5, so many different languages can read the files.
This is unnecessary. Use the build-in save(filename, variables,'-v6') is even much faster than this file.
e.g., test showed that:
if not use '-v6', it takes 7.7 seconds for a file.
Using this file, it's 0.73 seconds.
But using the '-v6' option, it's ONLY 0.17 seconds!
@yaron, off the top of my head I have no idea. What is the error message, and what are version(s) of Matlab on each?
Great contribution - but I have an issue with using it on a mac. For some reason, while the files load on a PC, the save fast .mat files cause an error on my mac. Any idea for how to fix this?
Ali, it can only handle certain types, I didn't try to reverse engineer the entire format.
But see the comment from Raj, immediately below yours, about hlp_serialize.
Thank you for providing this code. I tried to use it for fast saving of complex-sparse matrices and it ends with the following error:
Error using hdf5lib2
The HDF5 library encountered an unknown error.
Error in H5D.write (line 72)
do you think it there is a solution to this error or it is not applicable to large complex sparse data format?
Andre..may be your problem might be resolved if you first serialize the data using hlp_serialize(https://www.mathworks.com/matlabcentral/fileexchange/34564-fast-serializedeserialize) and store using savefast function and later deserialize the data. This worked even when I had complex vector is a structure...definitely improved saving files with marginal increase in memory!
Great submission, however it will not work for complex (real/imaginary) datatypes! The Matlab highlevel hdf5 routines apparently do not support complex datatypes and thus only the real part is saved. This is probably not easily fixed, as you would need to utilize low-level hdf5 functions.
Robin, that's good to know about parfor. Thanks for providing a version that works under such circumstances.
This is great. One problem was that 'evalin' doesn't work properly within a parfor loop. So I made a minor modification to allow 'save -struct' style saving which works fine in parfor:
Your performance benefit will depend a lot on your hard drive configuration. In my case I was writing to an SSD RAID. But on my laptop (with a single SSD) I can write in about 2.1 sec, which is still a 10x benefit. What kind of performance benefit are you getting?
I am not seeing anywhere close to that save time using your A,B,ops sample. I'm running 2012b.
Your example is showing that you saved ~600MB in .57 sec. That seems a bit suspect as it is over 1GB/sec.