Thread Subject: Memory problems

Dear all,

I've the following code running for about 500 images (satellite composites of sea surface temperature):

for i=1:length(imageslist)
disp(num2str(i));
temp = imread(imageslist{i});
temp2=double(temp);
temp2(find(temp2==max(temp2(:))))=0;
temp2(find(temp2==0))=NaN;

mymatrix(:,:,i) = temp2;

It doesn't finish running the code with all the images and stops saying "out of memory". Then I've used:

mymatrix=zeros(365,525,506);%506 images of 365 rows and 525 columns each

to force matlab to use just the right amount of memory. Then the code runs fine but when I try to write mymatrix into a file it shows again the "out of memory" error. I'm using the following gwrite function to write mymatrix into a file:

function flag = gwrite(name,tab)

[imax,jmax,kmax] = size(tab);
valex = 9999;
nbmots = imax*jmax*kmax;

c4 = reshape(tab,imax*jmax*kmax,1);
c4(find(isnan(c4))) = valex;
[flag]=uwrite(name,c4,imax,jmax,kmax,valex,nbmots);

function [flag]=uwrite(file,c4,imax,jmax,kmax,valex,nbmots)
flag=0;
file;
dummy=0;
dummyf=0.;
dummy24=24;
iprec=4;
fid=fopen(file,'w','ieee-be');
if fid~=-1
ind=find(isnan(c4));
c4(ind)=valex;
if nbmots==-1
 nbmots=imax*jmax*kmax
end
for i=1:10
 fwrite(fid,dummy,'int32');
 fwrite(fid,dummy,'int32');
end
nl=fix((imax*jmax*kmax)/nbmots);
ir=imax*jmax*kmax-nbmots*nl;
dummyval2=4*nbmots;
fwrite(fid,dummy24,'int32');
fwrite(fid,imax,'int32');
fwrite(fid,jmax,'int32');
fwrite(fid,kmax,'int32');
fwrite(fid,iprec,'int32');
fwrite(fid,nbmots,'int32');
fwrite(fid,valex,'single');
fwrite(fid,dummy24,'int32');

if imax<0 | jmax<0 | kmax<0
 nl=0;
 ir=4;
 disp('Degenerated matrix');
end

ide=1;

for kl=1:nl
 fwrite(fid,4*nbmots,'int32');
 fwrite(fid,c4(ide:ide+nbmots-1),'single');
 fwrite(fid,4*nbmots,'int32');
 ide=ide+nbmots;
end

fwrite(fid,4*ir,'int32');
fwrite(fid,c4(ide:ide+ir-1),'single');
fwrite(fid,4*ir,'int32');

flag=1;
fclose(fid);
end

I don't think the variable is that big to run out with all the memory. Do you know what could be done to deal with those memory problems and allow writting the matrix into a file using gwrite?

Thank you very much for any help

regards
juckou

juckou wrote:
>Then I've used:
>
> mymatrix=zeros(365,525,506);%506 images of 365 rows and 525 columns each
>
> to force matlab to use just the right amount of memory. Then the code runs fine but when I try
> to write mymatrix into a file it shows again the "out of memory" error.

Your code to write out the data makes a complete copy of the data, so it is not surprising
that you run out of memory.

> function flag = gwrite(name,tab)
>
> [imax,jmax,kmax] = size(tab);
> valex = 9999;
> nbmots = imax*jmax*kmax;
>
> c4 = reshape(tab,imax*jmax*kmax,1);

Another way of writing the above is:

valex = 9999;
c4 = tab(:);

But I would leave c4 in the original shape instead of making a vector
out of it.

> c4(find(isnan(c4))) = valex;

Use logical indexing instead of find:

c4(isnan(c4)) = valex;

Or, more efficient:

valex = 9999;
c4 = tab;
c4(isnan(c4)) = valex;


> [flag]=uwrite(name,c4,imax,jmax,kmax,valex,nbmots);

Just call

flag = uwrite(name, c4);

As you would be passing in the full shape of c4, the rest of the
parameters can be deduced, except valex (and see the below about that.)

 
> function [flag]=uwrite(file,c4,imax,jmax,kmax,valex,nbmots)
> flag=0;
> file;
> dummy=0;
> dummyf=0.;
> dummy24=24;
> iprec=4;
> fid=fopen(file,'w','ieee-be');
> if fid~=-1
> ind=find(isnan(c4));
> c4(ind)=valex;

There can't be any nan in c4 at this point: you eliminated them all
before you passed c4 in to this routine. The only effect of this is part
of the code is to make yet another copy of the data.

> if nbmots==-1
> nbmots=imax*jmax*kmax
> end

[imax, jmax, kmax] = size(c4);
nbmots = numel(c4);

I suspect several parts of the rest of the code can be cleaned up, but I
have not checked them out in detail.

> I don't think the variable is that big to run out with all the memory.

You need enough -contiguous- memory. And you have to stop making unnecessary
copies of your variables.

--
.signature note: I am now avoiding replying to unclear or ambiguous postings.
Please review questions before posting them. Be specific. Use examples of what you mean,
of what you don't mean. Specify boundary conditions, and data classes and value
relationships -- what if we scrambled your data or used -Inf, NaN, or complex(rand,rand)?

Walter is right when i says: use logical indexing.

never copy big variables/objects so they are in a good shape. Theses steps are good in order to understand what you want to do. It is not efficient to have the information twice in your memory.

A general hint:

 use the "whos" at certain steps in yout function to check if there are variables you will not need anymore. After identifying these variables "clear" them. If you need these big variables later, then "save" and "clear" to "load" them later.