What is the largest size matrix that matlab can handle

I have a matrix that is 100M x 100M , where M stands for million. I need to read it into matlab to compute eigen values. I have a i7 Mac with 8GB. Is it sufficient or should I go for some solid state device with ~128GB or so..?

Also, all I need is just the product of eigenvalues of my 100M x 100M matrix. If there is a shortcut to compute the product directly without having to compute each and every eigen value, that is preferable.

If computing eigen values isn't feasible using matlab for the size of matrix I'm dealing with, please suggest a different package that can do it at least on a supercomputing (HPC) facility

Can SVD be used instead of eigen value computation to get the product of eigen values? Any SVD package that can speed up my computation?

thanks in advance!

[Cedric]

If your matrix is not sparse, you will need 625,000 solid state drives of 128GB each, just to store it. It might be a little challenging, without even talking about finding eigen v's. I think that you should first re-orient the general approach. Why do you need such a large matrix? What would be its structure (if it were possible to store it)?

[Shuster]

the information need to populate the 100M X 100M matrix is currently stored on disk in ~27GB (binary file), exploring all the redundancy. I can't tell much about the structure of matrix but as you can see from the storage space used on disk, it has huge number of zeros. Some entries are repeated versions of a particular entry. Such redundant entries were also not stored. But to compute eigen values or determinant, I thought I might have to construct the matrix first to give to Matlab

Answer 1

[James Tursa]

100M x 100M full matrix is over 71 PETA bytes! No, I don't think you have enough memory (or time) to do this. Is your matrix sparse? If not you need to reformulate your problem.

The product of the eigenvalues of a matrix is equal to the determinant of the matrix. Assuming you can even calculate it, what do you plan to do with this (probably very inaccurate) determinant anyway?

[Shuster]

Yes, my matrix is sparse. There are also some repeated stuff in the matrix, which if eliminated along with the zeros, needs only ~27GB of disk space for storage in binary format.

All I need is estimate of determinant. From what I understand determinant can't be calculated without calculating all the individual eigen values of matrix.

There are several configurations for which I need to compute determinant. The configuration which has maximum determinant is the best possible one based on some bayesian entropy principles

[Walter Roberson]

Do you need the actual determinants or only to know which one is maximum? (Not that I know myself how the latter might help, but perhaps someone might have an idea.)

[Shuster]

All I need to know is to find the maximum over a set of possible configurations (entries in matrix change according to configuration but dimension remains same)... that allows me to find out the best configuration Any additional information isn't of much use.

[Walter Roberson]

Do you need to know which one is maximum, or what the maximum value is as well?

[Shuster]

I don't need to know the maximum value. I know it has unique maximum. Just knowing which configuration has maximum helps

[Walter Roberson]

I wonder if the following would be of any assistance?

http://www.cs.rpi.edu/~drinep/random/MC_Det.pdf

[Shuster]

It is an interesting reference. They managed to calculate determinant of 1M x 1M on 133 MHz Pentium computer with 64 MB of memory in just ~23 min back in 1999! It has been almost a decade and so I see no reason why I can't compute determinant of 100M x 100M. The only difference is that they have various values of alpha for a fixed matrix but in my case matrix varies for a fixed alpha