MATLAB function "save" and "-V7.3"

Running R2010a, 64bit(win64). I have a large structure. Up to this point save has worked fine. Efficient, fast, great... I have recently added new data to this structure. not so cool... Now I am getting the message:

Warning: Variable 'iceChart' cannot be saved to a MAT-file whose version is older than 7.3. To save this variable, use the -v7.3 switch. Skipping...

Unfortunately “-v7.3” appears to have some form of compression algorithm embedded, saving and loading these “-v7.3” mat files istaking significant time. Apparently there was a “-nocompression” flag that worked in previous versions of save. But this is no longer supported??? My options are at present seem to be just regenerating the entire data set during each procession session OR I can hack this by having multiple mat files using the old ‘–v6’ flag... I don't particularly like either of these options...

Appreciate any advice on this matter...

Regards Carl

Answer 1

[Jan]

Your observation is correct:

• With the -v7.3 flag you can store variables > 2GB with compression
• Without the -v7.3 flag (e.g. if the default version is set to -v7 or lower) you have no (or less) compression, but cannot store large arrays.

It is surprising, that the compression cannot be disabled manually, because it can be very time-consuming. But this was not the case when the v7.3 format has been developped, because the hard disks have been slower and more expensive. Therefore a compression has been a good idea.

Can you use HDF5 files?

[Malcolm Lidierth]

@Jan

See within. Compression appears to be applied only for the first variable in a v7.3 file. The HDF Groups HDFView program shows no compression on other variables. So

dumvar=0;

save(filename,'dumvar','-v7.3');

seems to act as a switch.

[Image Analyst]

maysam said (in a flag, which I removed):

working for me very well. thanks

Answer 2

[Malcolm Lidierth]

TMW do seem to struggle with file formats

I agree with @Jan above that not having a switch to disable compression in both v7 and 7.3 is problematic.

As @Walter points out, v7 can be fast - but this is context-dependent. Problems arise because information about the variables disc class, size etc AND NAME is compressed along with the data. The v6 and v7 file format uses a linked list of data entries. If you have 26 variables named A...Z stored in that order, you will need to access each of A to Y to find Z. In v6, that is less of an issue but in v7 you will need to decompress each preceding variable to find the one you are after. This means

 [1] data stored earlier in the file can be accessed more quickly and
 [2] for v7, large data entries should be stored last.

For both v6 and v7, there is a problem with using -append. Both are conservative about using disc space and will squeeze the file to recover any space that is freed. So if you have a uint16 scalar called 'A' and replace its value using

 save(filename, 'A', '-append);

up to 2Gb of data will be shifted upwards in the file before A is appended to the result. This squeeze seems to be done always - even when all conditions are met for the new entry just to overwrite the old one.

Note that v6 and v7 formats are both "Level 5". I would not recommend it, but you can mix save -appends with -v6 and -v7 to the same file without getting an immediate error. Whether or not you might eventually is another matter.

Two functions in my MAT-file Utilities http://www.mathworks.com/matlabcentral/fileexchange/12250 on the FEX may help:

where

is a whos like function but provides information about the class of data on disc and byte offset in the file. That info can be used to do low-level i/o on a -v6 file or to memory map it.

VarRename

can be used to rename a variable in the file to prevent its data area being reclaimed with a subsequent save -append.

I am presently updating these utilities to provide some easier to use functions e.g. getMap which simply returns a standard memmapfile object e.g.

map=getMap(filename, '/mydata/data/xdata');

There is, however, no good way to support v7 files - their design makes them useful only when the file is always to be loaded in its entirety or is small.

I am also adding v7.3 and HDF5 support. As v7.3 is HDF5-based you can use standard HDF5 tools such as HDFView http://www.hdfgroup.org/products/hdf5_tools/ to examine the content. The problems reported above might arise from the "chunk" size being inappropriate rather than gzip compression per se. I had hoped v7.3 would improve on v7 but from what is posted above it looks as though that may not be the case.

Rather disappointingly, TMW support tell me that the v7.3 format "...is not pure HDF5 however and we never claimed that these files will be readable by any HDF5 libraries".

So it looks as though the pre-2004 v6 will often be the best option as it's

• the fastest
• documented and supported outside of MATLAB - as is v7 - e.g. in Octave, Python and R amongst others.

[Malcolm Lidierth]

A quick check shows that v7.3 is not using compression on R2011b. Its performance is pretty good.

[Malcolm Lidierth]

Correction:

Compression is sometimes used with v7.3. It looks though, as only the first variable written to each file is compressed and all others are not. So, writing a small dummy variable to the file switches off compression for all subsequent variables.

[Jurek Dziewierz]

Correction 2: it seems to me that compression depends on how big is the variable. For small ones there is no compression, for big ones (eg. 2MB) there is compression no matter where it is in the file.

As i am using matlab on a cluser with superfast storage system, compression is a serious hinderance to me!

Answer 3

[Yair Altman]

Since R2008b, v7.3 compression seems to be done only (and always) for [large?] numeric data, but never for non-numeric data, regardless of their relative placement within the file. The release notes state that this change took place in R2008a, but in practice I still see uncompressed data in R2008a. Compression is only used extensively starting in R2008b.

The internal implementation of v7.3 seems to have changed over the years, making the compression more efficient. Still, the end result is not significantly different than the one suggested above (read on).

For improved performance you can save an HDF5 without compression (Deflate=0) using either the low-level HDF5 primitives, or using the savefast utility (which does it internally). This provides nearly the same performance as v6, while preserving the HDF5 benefits (ability to save objects, 2GB+).

Note that the resulting file size may be much larger compared to v6, especially if the data contains cell or struct arrays. So for simple data types (non-objects, <2GB) v6 might still be preferable due to the reduced I/O and cross-platform compatibility. The only exception to this rule might be if you're constantly accessing parts of data in the file, where v7.3's ability might be more efficient (R2011b and newer).

As a side-note, I find it highly ironic that the proprietary v6/v7 MAT format is more cross-platform-compatible (due to its documentation), compared to the v7.3 format that is supposed to rely on the standard HDF5 format and yet is sufficiently different (in a non-documented manner) to make this format unusable on any platform outside Matlab.

[justin]

How can I access my saved data in Matlab fold to use those data to run the script

[Walter Roberson]

K E comments in response to Justin:

I don't understand this comment

Answer 4

[carl howell]

-v6 save has also been faster when writing or reading for the type of data that I work with... (Most probable reason: compression???)

It is annoying that there is no flag to turn off compression in the latest version of save -v7.3...

I don't have time to muck with this any longer... I will hack a load function that (using -v6) hides the fact that there are multiple mat files being loaded... ugly but computationally effective with the limitation at hand...

MATHWORKS GUYS: please add a no compression option for saving files larger than 2 G... As demonstrated here, compression is actually a hindrance for some people and/or applications...

Thanks all...

C

Answer 5

[Fangjun Jiang]

Are you using the '-append' option of save? If you look at the help of save, '-v7.3' means Version 7.0 features plus support for data items greater than or equal to 2GB on 64-bit systems, which should be in your advantage.

Your MATLAB is R2010a which should be later than v7.3. That is why the warnning message sounds strange. It complaints that you try to save to a MAT-file older than 7.3. Can you try to just run save?

Answer 6

[carl howell]

~'append'... just a large structure... this is an example of what invokes the Warning message:

save('tmp.mat','iceChart');

this is what works on my data but takes a long time:

save('tmp.mat','iceChart','-v7.3');

[Fangjun Jiang]

Can you check the default version for MAT-files?

File > Preferences > General > MAT-Files

Answer 7

[Walter Roberson]

To cross-check: is it happening just because of variable sizes, or is do the variable perhaps contain "new style" matlab OO objects? Those new objects were not supported by -v7 .

When -v7.3 was introduced, users did some tests and reported results that would be consistent with only the header information being compressed, with the bulk of the (large) data being saved in space very nearly as large as if they were uncompressed. As that could mean gigabytes of disk I/O, the -v7.3 option was often avoided when practical by users: the reduced disk space usage of the compressed -v7 files made reading much faster, even taking in to account the time required for decompression.

I believe a couple of people from TMW have indicated that -v7.3 files are much faster than they initially were; I have not seen any comparison timings.

[Jan]

I'm writing a struct with 50 fields and subfields, which needs 5 to 20 MB memory to the local harddisk. When I use the v6 MAT-file format, the reading takes the half time compared to v7 and v7.3.

Answer 8

[Wenjuan Gong]

Do I need to use certain parameter when I load this file? coz I had a problem loading the file, maybe it is not saved correctly.

[Walter Roberson]

Please clarify which file you are referring to, and how you saved it. What MATLAB release are you using, and what problem are you seeing? Are you trying to save any object classes, or any handle graphics ?

Answer 9

[mursyidi azaki]

how to check our matlab version is it R2010 =v7.3?

[Walter Roberson]

At the MATLAB prompt give the command

version

Note that since version 7.3, there has been a Preference as to which save file format to use by default. If you need a particular save file format then it is best to specify it in the save() command.

Answer 10

[Walter Roberson]

As of R2017a, the save() command supports the -nocompression flag for -v7.3 files (only)