MATLAB Answers

Ingo
0

Best way to save big matrices

Asked by Ingo
on 29 Dec 2011
I'm looking for the best way to save big (both full and sparse) matrices in files. What I need to be able to do is:
- Read only certain entries from the file (certain matrix entries, that is)
- Create a sparse matrix in MATLAB from this file without creating a full matrix first (assuming the file contains a sparse matrix, of course)
The reason is quite simple: The nonsparse matrices will be too big to be stored into memory as a full matrix.
I read about the CDF file format which I'm not yet familiar with, so I don't know if it's possible to create a sparse matrix in Matlab from a CDF File?

  1 Comment

Ingo
on 29 Dec 2011
A little elaboration:
Given are n data points and now I want to calculate the (n x n) matrix containing the pairwise distance between all points. This is a full matrix, too big to store it in memory.
The next step would be to make this matrix sparse by, for example, ignoring all values less than a certain treshold. This sparse matrix can be stored in memory.

Sign in to comment.

1 Answer

Answer by the cyclist
on 29 Dec 2011
 Accepted Answer

The most recent versions of MATLAB have the matfile() command, which I believe does at least some of what you want.

  3 Comments

Ingo
on 29 Dec 2011
This seems like the perfect solution. I'm gonna give it a try as soon as possible. Thank you very much!
the cyclist
on 29 Dec 2011
If you do find it works out for you, please "accept" the solution, which may help future users with similar problems.
Ingo
on 30 Dec 2011
Hi. Sorry for being late on that!
Yes, matfile works for what I intended to do. However, for my purposes it is just way too slow. I'll have to stick with smaller data sets and try to create the resulting sparse matrices without ever calculating full matrices.
However, matfile() still does solve the problem.
Thanks again and Greetings
Ingo

Sign in to comment.