MATLAB function "save" and "-V7.3"

665 views (last 30 days)
carl howell
carl howell on 9 Sep 2011
Commented: Walter Roberson on 11 Jun 2023
Running R2010a, 64bit(win64). I have a large structure. Up to this point save has worked fine. Efficient, fast, great... I have recently added new data to this structure. not so cool... Now I am getting the message:
Warning: Variable 'iceChart' cannot be saved to a MAT-file whose version is older than 7.3. To save this variable, use the -v7.3 switch. Skipping...
Unfortunately “-v7.3” appears to have some form of compression algorithm embedded, saving and loading these “-v7.3” mat files istaking significant time. Apparently there was a “-nocompression” flag that worked in previous versions of save. But this is no longer supported??? My options are at present seem to be just regenerating the entire data set during each procession session OR I can hack this by having multiple mat files using the old ‘–v6’ flag... I don't particularly like either of these options...
Appreciate any advice on this matter...
Regards Carl

Accepted Answer

Jan
Jan on 9 Sep 2011
Your observation is correct:
  • With the -v7.3 flag you can store variables > 2GB with compression
  • Without the -v7.3 flag (e.g. if the default version is set to -v7 or lower) you have no (or less) compression, but cannot store large arrays.
It is surprising, that the compression cannot be disabled manually, because it can be very time-consuming. But this was not the case when the v7.3 format has been developped, because the hard disks have been slower and more expensive. Therefore a compression has been a good idea.
Can you use HDF5 files?
  2 Comments
Malcolm Lidierth
Malcolm Lidierth on 9 Oct 2011
@Jan
See within. Compression appears to be applied only for the first variable in a v7.3 file. The HDF Groups HDFView program shows no compression on other variables. So
dumvar=0;
save(filename,'dumvar','-v7.3');
seems to act as a switch.
Image Analyst
Image Analyst on 7 Apr 2014
maysam said (in a flag, which I removed):
working for me very well. thanks

Sign in to comment.

More Answers (9)

Yair Altman
Yair Altman on 10 Apr 2013
Since R2008b, v7.3 compression seems to be done only (and always) for [large?] numeric data, but never for non-numeric data, regardless of their relative placement within the file. The release notes state that this change took place in R2008a, but in practice I still see uncompressed data in R2008a. Compression is only used extensively starting in R2008b.
The internal implementation of v7.3 seems to have changed over the years, making the compression more efficient. Still, the end result is not significantly different than the one suggested above (read on).
For improved performance you can save an HDF5 without compression (Deflate=0) using either the low-level HDF5 primitives, or using the savefast utility (which does it internally). This provides nearly the same performance as v6, while preserving the HDF5 benefits (ability to save objects, 2GB+).
Note that the resulting file size may be much larger compared to v6, especially if the data contains cell or struct arrays. So for simple data types (non-objects, <2GB) v6 might still be preferable due to the reduced I/O and cross-platform compatibility. The only exception to this rule might be if you're constantly accessing parts of data in the file, where v7.3's ability might be more efficient (R2011b and newer).
As a side-note, I find it highly ironic that the proprietary v6/v7 MAT format is more cross-platform-compatible (due to its documentation), compared to the v7.3 format that is supposed to rely on the standard HDF5 format and yet is sufficiently different (in a non-documented manner) to make this format unusable on any platform outside Matlab.
  2 Comments
justin
justin on 22 Oct 2014
How can I access my saved data in Matlab fold to use those data to run the script
Walter Roberson
Walter Roberson on 10 May 2016
K E comments in response to Justin:
I don't understand this comment

Sign in to comment.


Malcolm Lidierth
Malcolm Lidierth on 8 Oct 2011
TMW do seem to struggle with file formats
I agree with @Jan above that not having a switch to disable compression in both v7 and 7.3 is problematic.
As @Walter points out, v7 can be fast - but this is context-dependent. Problems arise because information about the variables disc class, size etc AND NAME is compressed along with the data. The v6 and v7 file format uses a linked list of data entries. If you have 26 variables named A...Z stored in that order, you will need to access each of A to Y to find Z. In v6, that is less of an issue but in v7 you will need to decompress each preceding variable to find the one you are after. This means
[1] data stored earlier in the file can be accessed more quickly and
[2] for v7, large data entries should be stored last.
For both v6 and v7, there is a problem with using -append. Both are conservative about using disc space and will squeeze the file to recover any space that is freed. So if you have a uint16 scalar called 'A' and replace its value using
save(filename, 'A', '-append);
up to 2Gb of data will be shifted upwards in the file before A is appended to the result. This squeeze seems to be done always - even when all conditions are met for the new entry just to overwrite the old one.
Note that v6 and v7 formats are both "Level 5". I would not recommend it, but you can mix save -appends with -v6 and -v7 to the same file without getting an immediate error. Whether or not you might eventually is another matter.
Two functions in my MAT-file Utilities http://www.mathworks.com/matlabcentral/fileexchange/12250 on the FEX may help:
where
is a whos like function but provides information about the class of data on disc and byte offset in the file. That info can be used to do low-level i/o on a -v6 file or to memory map it.
VarRename
can be used to rename a variable in the file to prevent its data area being reclaimed with a subsequent save -append.
I am presently updating these utilities to provide some easier to use functions e.g. getMap which simply returns a standard memmapfile object e.g.
map=getMap(filename, '/mydata/data/xdata');
There is, however, no good way to support v7 files - their design makes them useful only when the file is always to be loaded in its entirety or is small.
I am also adding v7.3 and HDF5 support. As v7.3 is HDF5-based you can use standard HDF5 tools such as HDFView http://www.hdfgroup.org/products/hdf5_tools/ to examine the content. The problems reported above might arise from the "chunk" size being inappropriate rather than gzip compression per se. I had hoped v7.3 would improve on v7 but from what is posted above it looks as though that may not be the case.
Rather disappointingly, TMW support tell me that the v7.3 format "...is not pure HDF5 however and we never claimed that these files will be readable by any HDF5 libraries".
So it looks as though the pre-2004 v6 will often be the best option as it's
  • the fastest
  • documented and supported outside of MATLAB - as is v7 - e.g. in Octave, Python and R amongst others.
  3 Comments
Malcolm Lidierth
Malcolm Lidierth on 9 Oct 2011
Correction:
Compression is sometimes used with v7.3. It looks though, as only the first variable written to each file is compressed and all others are not. So, writing a small dummy variable to the file switches off compression for all subsequent variables.
Jurek Dziewierz
Jurek Dziewierz on 15 Feb 2012
Correction 2: it seems to me that compression depends on how big is the variable. For small ones there is no compression, for big ones (eg. 2MB) there is compression no matter where it is in the file.
As i am using matlab on a cluser with superfast storage system, compression is a serious hinderance to me!

Sign in to comment.


carl howell
carl howell on 12 Sep 2011
-v6 save has also been faster when writing or reading for the type of data that I work with... (Most probable reason: compression???)
It is annoying that there is no flag to turn off compression in the latest version of save -v7.3...
I don't have time to muck with this any longer... I will hack a load function that (using -v6) hides the fact that there are multiple mat files being loaded... ugly but computationally effective with the limitation at hand...
MATHWORKS GUYS: please add a no compression option for saving files larger than 2 G... As demonstrated here, compression is actually a hindrance for some people and/or applications...
Thanks all...
C

Walter Roberson
Walter Roberson on 11 Jul 2018
As of R2017a, the save() command supports the -nocompression flag for -v7.3 files (only)

Fangjun Jiang
Fangjun Jiang on 9 Sep 2011
Are you using the '-append' option of save? If you look at the help of save, '-v7.3' means Version 7.0 features plus support for data items greater than or equal to 2GB on 64-bit systems, which should be in your advantage.
Your MATLAB is R2010a which should be later than v7.3. That is why the warnning message sounds strange. It complaints that you try to save to a MAT-file older than 7.3. Can you try to just run save?
  2 Comments
carl howell
carl howell on 9 Sep 2011
Moved: DGM on 11 Jun 2023
~'append'... just a large structure... this is an example of what invokes the Warning message:
save('tmp.mat','iceChart');
this is what works on my data but takes a long time:
save('tmp.mat','iceChart','-v7.3');
Fangjun Jiang
Fangjun Jiang on 9 Sep 2011
Moved: DGM on 11 Jun 2023
Can you check the default version for MAT-files?
File > Preferences > General > MAT-Files

Sign in to comment.


Walter Roberson
Walter Roberson on 9 Sep 2011
To cross-check: is it happening just because of variable sizes, or is do the variable perhaps contain "new style" matlab OO objects? Those new objects were not supported by -v7 .
When -v7.3 was introduced, users did some tests and reported results that would be consistent with only the header information being compressed, with the bulk of the (large) data being saved in space very nearly as large as if they were uncompressed. As that could mean gigabytes of disk I/O, the -v7.3 option was often avoided when practical by users: the reduced disk space usage of the compressed -v7 files made reading much faster, even taking in to account the time required for decompression.
I believe a couple of people from TMW have indicated that -v7.3 files are much faster than they initially were; I have not seen any comparison timings.
  1 Comment
Jan
Jan on 9 Sep 2011
I'm writing a struct with 50 fields and subfields, which needs 5 to 20 MB memory to the local harddisk. When I use the v6 MAT-file format, the reading takes the half time compared to v7 and v7.3.

Sign in to comment.


Wenjuan Gong
Wenjuan Gong on 23 Apr 2012
Do I need to use certain parameter when I load this file? coz I had a problem loading the file, maybe it is not saved correctly.
  1 Comment
Walter Roberson
Walter Roberson on 23 Apr 2012
Please clarify which file you are referring to, and how you saved it. What MATLAB release are you using, and what problem are you seeing? Are you trying to save any object classes, or any handle graphics ?

Sign in to comment.


mursyidi  azaki
mursyidi azaki on 19 Oct 2015
how to check our matlab version is it R2010 =v7.3?
  1 Comment
Walter Roberson
Walter Roberson on 19 Oct 2015
At the MATLAB prompt give the command
version
Note that since version 7.3, there has been a Preference as to which save file format to use by default. If you need a particular save file format then it is best to specify it in the save() command.

Sign in to comment.


Sammy
Sammy on 11 Jun 2023
What is the difference of a compressed data or a saved data with no compression except for the storage space taken? Will the quality of data be compromised?
  2 Comments
Walter Roberson
Walter Roberson on 11 Jun 2023
The compression is lossless. The difference will be in the amount of storage and the time to write and the time to read.
Walter Roberson
Walter Roberson on 11 Jun 2023
Notes:
  • the compression used for -v7 files is different than what is used for -v7.3 files
  • -v7.3 files are almost always notably larger and slower than -v7 files, with the possible exception of mat files that contain pure double precision arrays. If you have mixed mixed data types such as struct or cell or table or objects, then -v7.3 can end up being much larger and slower.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!