Running R2010a, 64bit(win64). I have a large structure. Up to this point save has worked fine. Efficient, fast, great... I have recently added new data to this structure. not so cool... Now I am getting the message:
Warning: Variable 'iceChart' cannot be saved to a MAT-file whose version is older than 7.3. To save this variable, use the -v7.3 switch. Skipping...
Unfortunately “-v7.3” appears to have some form of compression algorithm embedded, saving and loading these “-v7.3” mat files istaking significant time. Apparently there was a “-nocompression” flag that worked in previous versions of save. But this is no longer supported??? My options are at present seem to be just regenerating the entire data set during each procession session OR I can hack this by having multiple mat files using the old ‘–v6’ flag... I don't particularly like either of these options...
Appreciate any advice on this matter...
No products are associated with this question.
Your observation is correct:
It is surprising, that the compression cannot be disabled manually, because it can be very time-consuming. But this was not the case when the v7.3 format has been developped, because the hard disks have been slower and more expensive. Therefore a compression has been a good idea.
Can you use HDF5 files?
TMW do seem to struggle with file formats
I agree with @Jan above that not having a switch to disable compression in both v7 and 7.3 is problematic.
As @Walter points out, v7 can be fast - but this is context-dependent. Problems arise because information about the variables disc class, size etc AND NAME is compressed along with the data. The v6 and v7 file format uses a linked list of data entries. If you have 26 variables named A...Z stored in that order, you will need to access each of A to Y to find Z. In v6, that is less of an issue but in v7 you will need to decompress each preceding variable to find the one you are after. This means
 data stored earlier in the file can be accessed more quickly and  for v7, large data entries should be stored last.
For both v6 and v7, there is a problem with using -append. Both are conservative about using disc space and will squeeze the file to recover any space that is freed. So if you have a uint16 scalar called 'A' and replace its value using
save(filename, 'A', '-append);
up to 2Gb of data will be shifted upwards in the file before A is appended to the result. This squeeze seems to be done always - even when all conditions are met for the new entry just to overwrite the old one.
Note that v6 and v7 formats are both "Level 5". I would not recommend it, but you can mix save -appends with -v6 and -v7 to the same file without getting an immediate error. Whether or not you might eventually is another matter.
Two functions in my MAT-file Utilities http://www.mathworks.com/matlabcentral/fileexchange/12250 on the FEX may help:
is a whos like function but provides information about the class of data on disc and byte offset in the file. That info can be used to do low-level i/o on a -v6 file or to memory map it.
can be used to rename a variable in the file to prevent its data area being reclaimed with a subsequent save -append.
I am presently updating these utilities to provide some easier to use functions e.g. getMap which simply returns a standard memmapfile object e.g.
There is, however, no good way to support v7 files - their design makes them useful only when the file is always to be loaded in its entirety or is small.
I am also adding v7.3 and HDF5 support. As v7.3 is HDF5-based you can use standard HDF5 tools such as HDFView http://www.hdfgroup.org/products/hdf5_tools/ to examine the content. The problems reported above might arise from the "chunk" size being inappropriate rather than gzip compression per se. I had hoped v7.3 would improve on v7 but from what is posted above it looks as though that may not be the case.
Rather disappointingly, TMW support tell me that the v7.3 format "...is not pure HDF5 however and we never claimed that these files will be readable by any HDF5 libraries".
So it looks as though the pre-2004 v6 will often be the best option as it's
Since R2008b, v7.3 compression seems to be done only (and always) for [large?] numeric data, but never for non-numeric data, regardless of their relative placement within the file. The release notes state that this change took place in R2008a, but in practice I still see uncompressed data in R2008a. Compression is only used extensively starting in R2008b.
The internal implementation of v7.3 seems to have changed over the years, making the compression more efficient. Still, the end result is not significantly different than the one suggested above (read on).
For improved performance you can save an HDF5 without compression (Deflate=0) using either the low-level HDF5 primitives, or using the savefast utility (which does it internally). This provides nearly the same performance as v6, while preserving the HDF5 benefits (ability to save objects, 2GB+).
Note that the resulting file size may be much larger compared to v6, especially if the data contains cell or struct arrays. So for simple data types (non-objects, <2GB) v6 might still be preferable due to the reduced I/O and cross-platform compatibility. The only exception to this rule might be if you're constantly accessing parts of data in the file, where v7.3's ability might be more efficient (R2011b and newer).
As a side-note, I find it highly ironic that the proprietary v6/v7 MAT format is more cross-platform-compatible (due to its documentation), compared to the v7.3 format that is supposed to rely on the standard HDF5 format and yet is sufficiently different (in a non-documented manner) to make this format unusable on any platform outside Matlab.
Are you using the '-append' option of save? If you look at the help of save, '-v7.3' means Version 7.0 features plus support for data items greater than or equal to 2GB on 64-bit systems, which should be in your advantage.
Your MATLAB is R2010a which should be later than v7.3. That is why the warnning message sounds strange. It complaints that you try to save to a MAT-file older than 7.3. Can you try to just run save?
~'append'... just a large structure... this is an example of what invokes the Warning message:
this is what works on my data but takes a long time:
To cross-check: is it happening just because of variable sizes, or is do the variable perhaps contain "new style" matlab OO objects? Those new objects were not supported by -v7 .
When -v7.3 was introduced, users did some tests and reported results that would be consistent with only the header information being compressed, with the bulk of the (large) data being saved in space very nearly as large as if they were uncompressed. As that could mean gigabytes of disk I/O, the -v7.3 option was often avoided when practical by users: the reduced disk space usage of the compressed -v7 files made reading much faster, even taking in to account the time required for decompression.
I believe a couple of people from TMW have indicated that -v7.3 files are much faster than they initially were; I have not seen any comparison timings.
-v6 save has also been faster when writing or reading for the type of data that I work with... (Most probable reason: compression???)
It is annoying that there is no flag to turn off compression in the latest version of save -v7.3...
I don't have time to muck with this any longer... I will hack a load function that (using -v6) hides the fact that there are multiple mat files being loaded... ugly but computationally effective with the limitation at hand...
MATHWORKS GUYS: please add a no compression option for saving files larger than 2 G... As demonstrated here, compression is actually a hindrance for some people and/or applications...