Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

To resolve issues starting MATLAB on Mac OS X 10.10 (Yosemite) visit: http://www.mathworks.com/matlabcentral/answers/159016

Out of memory when calculating very large Matrix (~100000 , 200000)

Asked by John R on 2 Nov 2011

I am trying calculate a very large matrix using a "parfor" loop, but am running out of memory while calculating it. The matrix is sparse, and I write to it column by column. I have looked at the suggestions in the matlab help to reduce memory usage and have implemented as many as I can. Are there any good tricks to reduce memory usage when using the parfor loop? (I use it after running matlabpool in the local configuration, it connects to all 4 cores on my processor)

Update: I tried Tim Davis' suggestion, but now am having issues with the parfor loop not liking my indexing, so I had to use 3 sliced variables instead of just 1...made it a tiny bit faster but I think it is using even more memory now.

Before Tim's suggestion:

%I initialize all of my variables

bins=1024;
parfor index=1:width*height
vals=zeros(bins*length(projectionAngles),1)
   for k=1:length(projectionAngles)
       binNumber=FindBinNumber(index k);
       for l=binNumber-1:1:binNumber+1
           vals(k*bins-(bins-l),1)=pixelArea(binNumber, index)
       end
   end
Aij(:,index)=vals;
end

After Tim's suggestion

parfor index=1:width*height
vals=zeros(bins*length(projAngles),1)
   for k=1:length(projAngles)
       binNumber=FindBinNumber(index k);
       for l=binNumber-1:1:binNumber+1
           row(1,length(projAngles)*3- ((length(projAngles)-(k-1))*3)) = uint16(k*bins-(bins-l));
            col(1,length(projAngles)*3-((length(projAngles)-(k-    1))*3)) = uint16(index);
            val(1,length(projAngles)*3-((length(projAngles)-(k-    1))*3))=pixelArea6_mex(x2,y2,center,num_proj_bins,m1,unit_rise(k),unit_run(k),l); 
        end 
    end
       row1(index,:)=row;
       col1(index,:)=col;
       val1(index,:)=val;
end
row1=reshape(row1,width*height*3*length(projAngles),1);
col1=reshape(col1,width*height*3*length(projAngles),1); val1=reshape(val1,width*height*3*length(projAngles),1); 
Aij=sparse(row1,col1 ,val1);

I don't think I did this right...Any suggestions on how to get rid of these three variables and still be able to run this in a parfor loop?

0 Comments

John R

Products

No products are associated with this question.

2 Answers

Answer by Tim Davis on 2 Nov 2011
Accepted answer

Don't attempt to update a sparse matrix that way. The best solution is to create a triplet form, which is an unordered list of row indices, column indices, and values (3 vectors I,J,X where the kth entry in the list is a(I(k),J(k))==X(k)). Then use 'sparse' to convert the triplet form to a sparse matrix all at once, via A = sparse (I,J,X).

1 Comment

John R on 2 Nov 2011

That makes a lot of sense. Thank you sir!

Tim Davis
Answer by Naz on 2 Nov 2011

Every time you add a new raw or column to your matrix, the program uses new memory because your old data + new data does not fit in the previous memory slot. You need to preallocate the memory for you matrix. Let's say that your final matrix zise will be 10000x200000, then you need to do this:

yourMatrix=zeros(100000,2000000);

In the loop you can access the needed column this way:

for n=1:2000000
yourMatrix(:,n)=newCoulumn;
end

2 Comments

John R on 2 Nov 2011

I have indeed pre-allocated. I did so using "spalloc" and I know the maximum number of elements that could possibly be in the matrix, so that shouldn't be an issue.

Jan Simon on 2 Nov 2011

Can you add the code to your original message? Without seeing the code, we can just guess. And if a guessed solution does not match your problem, as Naz' idea of pre-allocation, it is a waste of time.

Naz

Contact us