MATLAB Answers

Matlab fails saving on big cell-Arrays

9 views (last 30 days)
Vincent
Vincent on 23 Jul 2012
After creating a big cell Array containing mixed content (logical, float and char Arrays), I want to save this cell Array to a file (3500x1000 cells). However it starts saving but never stops. The file becomes bigger and bigger.
I save using Matlab v. 7.5, HDF5-based (v73 file format) onto a NTFS hard-disk on a 64 bit win xp sp3 system.
I alreday encountered this error when handling objects, but cell-Arrays were never a problem.
Does a known issue exist? How can I avoid this problem?
  2 Comments
Vincent
Vincent on 23 Jul 2012
Yes. whos: 92MB, Saving uncompressed: 92MB, Saving compressed: > 3GB!

Sign in to comment.

Answers (1)

Jan
Jan on 23 Jul 2012
Edited: Jan on 23 Jul 2012
Your {3500x1000} cell contains 3'500'000 variables. Each variable has some overhead to describe the type and the dimensions. Let's assume it is 64 bytes. The the actual data must be written the file. If the cell elements contains array of about 100 elements of type double, you get:
3.5e6 * (64 + 100 * 8) % bytes
There is some overhead for the file format and some compressing, but both will not exceed a factor of 10 or 0.1. So you get a file size of about 3'024'000'000 bytes. Formatting, compressing and writing 3GB takes time. When you store such files on a network drive, it will take more time.
Please explain why you assume, that there is any error or a problem. Do you run a 64-bit version of Matlab? What does "bigger and bigger" mean in absolute numbers? Did you try to write an uncompressed file, which is much faster?
  2 Comments
Jan
Jan on 23 Jul 2012
I'm still convinced, that the MAT file formats are not well defined. Even the -v6 MAT files have been compressed, as longs as they do not contain more than 1 array of the type SINGLE -- at least if you save them from Matlab 6.5.1.
Actually HDF5 is an extremly well tested and established format, but sometimes I dream of creating a simple binary format, which allows appending, encryption on the fly, controllable LZO (fast) or LZMA (small files) compression and an open source implementation. But when I think of the effort, which is required to handle exceptions like user-defined objects, function-handles and fragmentation if existing variables are overwritten, I stop such dreams immediately until I get retired.

Sign in to comment.

Tags

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!