Memory efficient options for storing multiple square matrices of unequal size

Hello MATLAB community,

this is my first time asking a question on these forums.

My problem is that in my codebase I require storage of many oddly shaped arrays that describe a block tridiagonal matrix. Ideally if the arrays were all the same dimension I'd use a sparse 3d array. Instead, I often find I have wildly different block sizes and sometimes these matrices are very sparse or in fact empty. In python (gasp) you can make a list that contains oddly sized matrices very easily but it does not seem so easy in MATLAB. The best solution I have arrived at is to use cells. Unfortunately the overhead for each cell is very large, especially when I have many sparse matrices, or many blocks of low dimension.

I'd love to hear any practicable ideas that the community has.

Cheers, Jesse

[per isakson]

There isn't anything better than a cell array. The overhead isn't that large! Spare or not is a different question.

[Walter Roberson]

The overhead per cell entry is about 100 - 112 bytes (depending on obscure factors)

MATLAB does not support 3D sparse arrays. You can simulate them using sub2ind and linear or 2D indexing

[Jesse]

Hi Walter Roberson, Per Isakson,

an overhead of 100-112 bytes is small on its own, but many grains of sand make a heap.

Consider that a singed floating point number requires itself 8 bytes. For a dense 4x4 matrix (128 bytes), this would be equivalent to increasing the storage space by a factor of 2. In my work, I can have many such small matrices, at several energy points. Let's say I wanted to store the identity matrx, now of course I could store it as a vector, but this is a thought experiment.

A sparse 10x10 identity matrix takes up 248 bytes. 10 copies of this matrix would therefore take up 2480 bytes.

Let's say I now store that as block diagonal matrices, one for each diagonal element. (8 + 112) bytes * 10 = 1200 bytes for one matrix, now we store each of these inside a cell and we have (1200+112)*10 = 13200 bytes. This is five times the storage requirements! In my work I can have many many more matrices over many different variables and it all adds up fast. You may argue that I have contrived the example with the most overhead, but I do actually have several matrices of this type, though not exclusively, I may have a 3x3, then a 1x1, then a 100x100, then a 2x2, and so on.

As far as I can tell, if I wanted to store these efficiently, I'd be better off inventing some kind of new naming structure and having thousands of individual matrices rather than using cells. I wanted to know if someone else had thought out this problem before.

Cheers, Jesse.