Matlab fails saving on big cell-Arrays

After creating a big cell Array containing mixed content (logical, float and char Arrays), I want to save this cell Array to a file (3500x1000 cells). However it starts saving but never stops. The file becomes bigger and bigger.

I save using Matlab v. 7.5, HDF5-based (v73 file format) onto a NTFS hard-disk on a 64 bit win xp sp3 system.

I alreday encountered this error when handling objects, but cell-Arrays were never a problem.

Does a known issue exist? How can I avoid this problem?

[Titus Edelhofer]

When you look at you workspace (workspace browser, whos), and the file that get's saved: when the file continous to grow, does it grow significantly larger then what the memory in MATLAB shows?

[Vincent]

Yes. whos: 92MB, Saving uncompressed: 92MB, Saving compressed: > 3GB!

Answer 1

[Jan]

Your {3500x1000} cell contains 3'500'000 variables. Each variable has some overhead to describe the type and the dimensions. Let's assume it is 64 bytes. The the actual data must be written the file. If the cell elements contains array of about 100 elements of type double, you get:

3.5e6 * (64 + 100 * 8) % bytes

There is some overhead for the file format and some compressing, but both will not exceed a factor of 10 or 0.1. So you get a file size of about 3'024'000'000 bytes. Formatting, compressing and writing 3GB takes time. When you store such files on a network drive, it will take more time.

Please explain why you assume, that there is any error or a problem. Do you run a 64-bit version of Matlab? What does "bigger and bigger" mean in absolute numbers? Did you try to write an uncompressed file, which is much faster?

[Vincent]

The reason for which i assume Matlab has a problem is: I saved such a cell array about half a year ago and got a 23mb file. Reopening and saving it creates output of 2GB (Matlab is still "saving to mat file" in its status bar).

So perhaps I changed a setting and now I'm getting this behavior.

Now the fun fact: The uncompressed file (v6) results in a 90mb file, the compressed one in ~2GB. Kind of awkard...

[Jan]

I'm still convinced, that the MAT file formats are not well defined. Even the -v6 MAT files have been compressed, as longs as they do not contain more than 1 array of the type SINGLE -- at least if you save them from Matlab 6.5.1.

Actually HDF5 is an extremly well tested and established format, but sometimes I dream of creating a simple binary format, which allows appending, encryption on the fly, controllable LZO (fast) or LZMA (small files) compression and an open source implementation. But when I think of the effort, which is required to handle exceptions like user-defined objects, function-handles and fragmentation if existing variables are overwritten, I stop such dreams immediately until I get retired.