Recommended steps to deal with a slow loop

Hi,

If a "for loop" is slow and each iteration is taking 2 hours what are the recommended steps that I should follow to deal with this issue?

Answer 1

[Conrad]

I would recommend that you first focus on what happens inside the for loop... why is it taking so long? Get a complete breakdown of how much time is spend at every step, then you can decide where you can possibly optimise.

Further details would be great.

Answer 2

[Sean de Wolski]

• Listen to mlint. If it is warning you about a variable changing size within a loop, that will slow it down immensely
• Remove every calculation from every loop that does not depend on the loop.

[Danielle Leblanc]

below is my code. It is taking ages for each loop. I am a beginner and took me a lot of effort to write it . but still it is not feasible to implement it as it needs a lot of time to be executed and most probably my laptop will burn before running these codes. Any help would be a life saver. cd C:\Users\Danielle\Documents\MATLAB\CS\CSV

for i=1:114
%%Import the data, extracting spreadsheet dates in MATLAB datenum format
fullFileName=sprintf('%s%d%s', 'C:\Users\Danielle\Documents\MATLAB\CS\CSV\',i, '.csv') ;
fid = fopen(fullFileName, 'rt');
M=textscan(fid,'%s','collectoutput',1,'headerlines',0);
fclose(fid);  
X=M{1,1};
fieldnames = regexp( X(1,:), ',', 'split');
V= regexp( X(2:end,1), ',', 'split');
 ncol=[4 5 10 18];%numeric columns
 scol=[1 2 3 4 6 7 11 12 13];%string columns
 vtxt=fieldnames{1,1}(1,ncol);%numeric headers
 stxt=fieldnames{1,1}(1,scol);%string headers
variablestxt=fieldnames{1,1};
v2=nan(size(V,1)-1,size(ncol,2));
s2=num2cell(nan(size(V,1)-1,size(scol,2)));
for j=2:length(V)
try
v0=[V{j,1}];
v2(j-1,1)=datenum(v0(1,4),'mm/dd/yyyy');
str = [v0{1,4}, ' ', v0{1,5}];
v2(j-1,2) = datenum(str, 'mm/dd/yyyy HH:MM:SS');
v2(j-1,3:length(ncol))=str2double(v0(1,ncol(3:end)));
 s2(j-1,1:length(scol))=v0(1,scol);
catch
    'error' %#ok<NOPTS>
end
end
try
ID=unique(s2(:,1));
catch
xe3=find(cellfun(@(x)~ischar(x),s2(:,1)));
s2(xe3,:)=[];
v2(xe3,:)=[];
ID=unique(s2(:,1));
end
Count=0;
for k=1:length(ID)
    x0=find(ismember(s2(:,1),ID(k,1))); 
    v3=v2(x0,:);
    s3=s2(x0,:);
    x1=find(max(v3(:,2)));
    v(Count+1:Count+size(x1,1),1:size(v2,2))=v3(x1,:);
    s(Count+1:Count+size(x1,1),1:size(s2,2))=s3(x1,:);
    Count=Count+size(x1,1);
    xe=find(ismember(s(:,5),'C'));
    s(xe-1:xe,:)=[];
    v(xe-1:xe,:)=[];
end
filename=[num2str(i) '.mat'];
cd C:\Users\Danielle\Documents\MATLAB\CS\Step1
save (filename,'s','v','stxt','vtxt')

[Sean de Wolski]

Use the profiler to identify the bottlenecks. Just run two or three files with profiler and it will show you where to focus your efforts. Some obvious things:

• try/catch is slow. I would try and write the code to avoid those.
• Skip the call to find here:

 x0=find(ismember(s2(:,1),ID(k,1)));

• regexp can be slow, perhaps csvread or dlmread?
• Conversions to and from cell is slow (s2=num2cell(nan(size(V,1)-1,size(scol,2)));) avoid this if possible.

[Danielle Leblanc]

the profiler is showing that ismember and unique functions are th emost time consuming functions