Multiplication of gigantic matrices

Hi,

I am trying to multiply two gigantic matrices: [1585152 90]*[90 1585152]. MATLAB runs out of memory. Is there any workaround for this?

[Bruno Luong]

Just don't.

Keep both of these matrices as a coding of the product, then when you need to do algebra with the new matrix, use the alternate coding form with algorithms that are base of matrix x vector product, and never need to form explicitly the matrices product.

[Matt J]

You probably shouldn't do it, unless the matrix is sparse. I encourage you to explain what the resulting matrix will be used for so that indirect methods can be discussed.

[Askic V]

If your matrices contain a lot of zeros, then first use sparse function, for example As = sparse(A). This will save memory and multiplication can be more eficient.

For example:

clear
clc
v = 1:10000;
A = diag(v,2);

v = 2+randn(1,10000);
B = diag(v,2);

tic
C = A*B;
toc

Elapsed time is 15.971504 seconds.

As = sparse(A);
Bs = sparse(B);
tic
Cs = As*Bs;
toc

Elapsed time is 0.017916 seconds.

size(Cs)

ans = 1×2

10002 10002

whos

  Name          Size                   Bytes  Class     Attributes

  A         10002x10002            800320032  double              
  As        10002x10002               240024  double    sparse    
  B         10002x10002            800320032  double              
  Bs        10002x10002               240024  double    sparse    
  C         10002x10002            800320032  double              
  Cs        10002x10002               239992  double    sparse    
  ans           1x2                       16  double              
  v             1x10000                80000  double              

[John D'Errico]

Do you have the memory? The size of your result will be:

1585152*1585152

elements. Multiply that by 8, assuming doubles. Converting to rough gigabytes:

1585152*1585152*8/2^30

So only 18700 gigabytes, or 18.7 terabytes of RAM.

Again, do you have that much RAM? I'm not talking about disk space, where even that would be seriously big. Even in these days of large chunks of cheap RAM, 18 terabytes is flat out significantly big.

Unless your arrays are seriously sparse matrices, even converting them to sparse will not help much.

Far better is to do as has been suggested, to NOT do this. A little linear algebra will show you how to do much with the original arrays. And, worse, when you do that multiplication, you actually make things numerically pooly behaved. There are many good reasons to not do what you want to do. And one good reason why you CANNOT do it, because of the memory requirenemt.

Just because you have a formula that shows that product of matrices, does not mean it is a good idea to write code following that formula.

[Steven Lord]

What were you hoping to do with this extremely large matrix? There may be alternate approaches that don't require nearly so much memory. For example, if you were hoping to solve a system using the normal equations don't do that. Use mldivide, \ or one of the iterative methods for solving a system.

Answer 1

[Allen]

You can use datastores and tall arrays to work with large amounts of data. See references to both.

https://www.mathworks.com/help/matlab/datastore.html

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

[Bruno Luong]

See John's comment, it need 18.7 Tb to store it, without looking to time to work with huge matrix.

Explicit multiplication is a bad idea that will lead pratically to nowhere.