MATLAB Answers

Compress only selected variables when saving to .mat

8 views (last 30 days)
Oleg Komarov
Oleg Komarov on 25 Jun 2014
Commented: Oleg Komarov on 23 Jan 2015
I have two variables data and meta, which I am saving in a compressed .mat file (version '-v7' ). The data variable is usually 800mb uncompressed, while meta might not even be 1mb. I have lots of these .mat files and sometimes I just need to loop through all the meta variables. However, since the file is compressed, it still takes lots of time to load the meta variable alone , i.e. same time as if I were to load both variables.
Is it possible to selectively compress specific variables in a .mat file ? Alternative data designs?
Note : I already have an overall single meta which is basically the concatenation of the single smaller ones. However, I will need to abandon this approach because it does not scale well size-wise and performance-wise.
  6 Comments
Oleg Komarov
Oleg Komarov on 26 Jun 2014
So, matfile() does not improve the loading speed (but it also depends on the setup, I am currently testing with an SSD).
From my previous link, it seemed possible to use low level HDF5 API directly on .mat files, but it is not the case. Also, I do not have the time nor the intention to learn/rewrite everything with pure hdf5 (mat files being based on that).
I have not mentioned that meta is a dataset, and this might impact on the way it is stored, i.e. not contiguously, forcing to unzip/load many chunks. I will test serialization of the dataset, before saving it.

Sign in to comment.

Answers (1)

Jeremy
Jeremy on 23 Jan 2015
I know this is old, but I have been doing some similar work recently since I could tell it was taking longer than it should to load a small portion of the data using the matfile method. I also spent two days understanding how to use all the low level HDF5 commands only to find that it did not really help on the read side.
Then I realized that the issue is originating on the write side and the compression that occurs. the savefast utility in the file exchange saves using the high level hdf5 commands and this does NOT compress the data. This didn't quite work for me since I am saving complex numbers but I was able to use the same approach and I am now saving uncompressed v7.3 files and reading small portions of my matrix over 100 times faster!
If your matrix is just real numbers, you should be able to create a v7.3 file with your metadata and then use the simple high-level h5write command to save additional variables to the same file uncompressed.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!