Code covered by the BSD License  

Highlights from
Save MAT files more quickly

4.6

4.6 | 6 ratings Rate this file 90 Downloads (last 30 days) File Size: 1.84 KB File ID: #39721

Save MAT files more quickly

by

 

Saves large numeric arrays more quickly than Matlab's built-in save command

| Watch this File

File Information
Description

When working with large arrays, Matlab's 'save' command can be very slow, presumably because Matlab attempts to use compression. This function provides a much faster alternative, at the cost of larger files. It only improves speed for variables that are plain arrays; elements of structures or cell arrays are not altered.

Example:
% First make some data
>> ops = struct('algorithm', 'greedy');
>> A = int32(randi(20, 1000, 1200, 40));
>> B = randn(500, 1800, 60);
% Now save it
>> tic; save /tmp/test ops A B; toc
Elapsed time is 22.980294 seconds.
>> tic; savefast /tmp/test ops A B; toc
Elapsed time is 0.571098 seconds.

Happy saving!

Required Products MATLAB
MATLAB release MATLAB 8.0 (R2012b)
Tags for This File   Please login to tag files.
Please login to add a comment or rating.
Comments and Ratings (18)
07 Sep 2014 Tim Holy

@arnold, does it work if you list variables explicitly? It's possible it's just missing a variant that saves all the variables in your workspace.

06 Sep 2014 arnold

I don't seem to understand the syntax, it doesn't behave like 'save'.

savefast('test.mat');

Does produce a file with the appropriate name but it doesn't seem to contain anything. Loading it does not restore any of the workspace variables.

07 Jun 2014 Tim Holy

@zhang, the problem is that '-v6' has some serious limitations, for example it can't store arrays bigger than 2GB. It's also not as portable to other languages, etc; '-v7.3' is based on HDF5, so many different languages can read the files.

07 Jun 2014 zhang

This is unnecessary. Use the build-in save(filename, variables,'-v6') is even much faster than this file.

e.g., test showed that:
if not use '-v6', it takes 7.7 seconds for a file.
Using this file, it's 0.73 seconds.
But using the '-v6' option, it's ONLY 0.17 seconds!

05 May 2014 Tim Holy

@yaron, off the top of my head I have no idea. What is the error message, and what are version(s) of Matlab on each?

05 May 2014 yaron

Great contribution - but I have an issue with using it on a mac. For some reason, while the files load on a PC, the save fast .mat files cause an error on my mac. Any idea for how to fix this?

10 Jan 2014 Tim Holy

Ali, it can only handle certain types, I didn't try to reverse engineer the entire format.

But see the comment from Raj, immediately below yours, about hlp_serialize.

10 Jan 2014 Ali

Thank you for providing this code. I tried to use it for fast saving of complex-sparse matrices and it ends with the following error:

Error using hdf5lib2
The HDF5 library encountered an unknown error.
Error in H5D.write (line 72)

do you think it there is a solution to this error or it is not applicable to large complex sparse data format?
Thanks

03 Dec 2013 Raj

Andre..may be your problem might be resolved if you first serialize the data using hlp_serialize(https://www.mathworks.com/matlabcentral/fileexchange/34564-fast-serializedeserialize) and store using savefast function and later deserialize the data. This worked even when I had complex vector is a structure...definitely improved saving files with marginal increase in memory!

03 Dec 2013 Raj  
09 Oct 2013 Andre Kuehne  
10 Sep 2013 Andre Kuehne

Great submission, however it will not work for complex (real/imaginary) datatypes! The Matlab highlevel hdf5 routines apparently do not support complex datatypes and thus only the real part is saved. This is probably not easily fixed, as you would need to utilize low-level hdf5 functions.

11 Jul 2013 Tim Holy

Robin, that's good to know about parfor. Thanks for providing a version that works under such circumstances.

11 Jul 2013 Robin

This is great. One problem was that 'evalin' doesn't work properly within a parfor loop. So I made a minor modification to allow 'save -struct' style saving which works fine in parfor:
https://gist.github.com/robince/5974172

18 Jun 2013 Tim Holy

Hi Jeremy,

Your performance benefit will depend a lot on your hard drive configuration. In my case I was writing to an SSD RAID. But on my laptop (with a single SSD) I can write in about 2.1 sec, which is still a 10x benefit. What kind of performance benefit are you getting?

18 Jun 2013 Jeremy

I am not seeing anywhere close to that save time using your A,B,ops sample. I'm running 2012b.

Your example is showing that you saved ~600MB in .57 sec. That seems a bit suspect as it is over 1GB/sec.

17 Jun 2013 Robin  
01 Jun 2013 Chien-Tung Hsu  

Contact us