how can i distribute a large matrix?

I have a matrix of (20*2^20) and I can not deal with it . I want to distribute the matrix. How can I use ( distribute) and (gather) . please, the steps that can I write in the m-file.

[Geoff Hayes]

esraa - please clarify what you mean by distribute? What do you intend to do with this matrix?

[Stephen23]

@Geoff Hayes: Matrix distribution is the time-old tradition of giving freshly baked matrices to the neighbors to celebrate the new year. I guess the OP baked a very large matrix, and, as the title says, are now not sure how they should carry it around the neighborhood.

[Star Strider]

I always baked the complex ones, for the subtle interplay of flavours and that half the calories are imaginary.

[esraa]

I am studying the outage of electric feeders in a distribution system . the matrix represents the state of the n feeders (2^n states). so, the matrix is (20*2^20) all the elements are ones and zeroes . I want to write the matrix and identify the accepted states that satisfy certain conditions and other calculations. when I put n=6 (6 feeders) the output is displayed immediately but when I put n=20; the output is not displayed even after two days (matlab is busy).I want to distribute the matrix as in this link http://www.mathworks.com/videos/using-distributed-arrays-to-process-large-matrices-68757.html when I wrote ( pmode start 4) in the command window it takes along time (busy) how can I distribute the matrix?

[Star Strider]

You have to have the Parallel Computing Toolbox.

[Stephen23]

Have you tried the simpler steps of first checking the code that you are trying to run on the this matrix? Use profile to see which parts are taking up the most time, or have a look to see if you could vectorize any of the operations. Yes, they are small unimpressive words compared to "distributed arrays", but if you read those pages then they will make a huge difference to the speed of your code.

Making MATLAB code run faster is a topic which has been discussed many times before: have you searched this forum for advice? Did you read any of MATLAB's own advice on speeding up code?:

http://blogs.mathworks.com/loren/2008/06/25/speeding-up-matlab-applications/

Summary: before jumping onto distributed arrays, check that the code is already as lean-running as is reasonable.

[per isakson]

"matrix of (20*2^20)" &nbsp What is the size of that matrix? [(20*2^20),(20*2^20)] is huge, but is that what you mean?

Test on a five year old PC with 8GB:

    >> tic, m = randi([0,1],[5*2^10,4*2^10],'uint8');imagesc(m);whos,toc
      Name         Size                 Bytes  Class     Attributes

      ans          1x1                      8  double              
      m         5120x4096            20971520  uint8               
      n            1x1                      8  double              

    Elapsed time is 0.882252 seconds.

That's not a big matrix. I guess the problem is in your code.

[Image Analyst]

I think it's 20 rows by 2^20 = 1048576 columns. That's 20 million elements. If they're double (8 bytes per element), that's 160 MB. Not really that huge. We need to know what "I can not deal with it" really means.

[John D'Errico]

I recall seeing a recent question about an array of exactly that size, and asked by the same person. There was never any resolution there either. This simply is not that large of a matrix. The response made there by esraa was confusing.

[esraa]

matrix:
no of rows = 20
no of column = 2^20

May the problem be in my computer? if so, can I send the m-files to be run on another computer?

[Image Analyst]

Of course you can run your code on another computer. Go ahead.

Answer 1

[Image Analyst]

If a lot of it is empty/zero/unassigned, then try making it sparse. Otherwise, use it in bite-sized chunks.

Answer 2

[Stephen23]

The code that you uploaded is full of poor coding practice. Some examples of this are:

• using i and j as the names of variables (there are the names of the inbuilt imaginary unit ).
• over-complicated data structure: use a cell array instead of a structure.
• no vectorization : total reliance on loops.
• no array preallocation within those loops.

Using distributed arrays will not solve the slowness of this calculation, because it is the code itself that it poorly written. The last two points particularly need to be addresses to speed up your calculation.

If you tried to use the MATLAB debugger , then you would quickly find that the script does not even progress past the first set of nested loops. When I investigated why this is so slow, I found that the variable a is being enlarged on every loop iteration:

a = [0,1]
a = [0,1,0,1]
a = [0,1,0,1,0,1]
..
a = [0,1,0,1,0,1,0,1,.. ???]

given the rather obfuscated nature of how this vector is being generated I did not waste my time trying to figure out how large it might become (when I stopped it was 1x115444). Every time that you extend any variable in MATLAB it has to check the available memory, and move the variable if required. Suffice to say this can be a slow process.

Note also that you could generate this vector all in one go using the much simpler command

a(2:2:N) = 1;

for some scalar N. Try it and you will see how much faster MATLAB can be.

Until this code is vectorized where appropriate and plenty of array preallocation where not, then this code will always be slow, regardless of what computer you run it on.

[esraa]

thank you very much, how can I construct the matrix b, b: no. of column=20 and no. of rows=2^20=1048576.

the first column=[0 1 0 1 0 1 ........]
the second column=[0 0 1 1 0 0 1 1 0 0 1 1 ........]
the third column=[0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 ..........]
the fourth column=[0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1..................]

and so on. instead of the following command

for j=1:20

      for k=0:(2^(ns-j))-1
          a((2^(j-1))+1+k*(2^j):(2^j)+k*(2^j))=1;
      end
      b(j,:)=a;
      clear a
  end
  b=b';

to take less time compared with the commands I use