Recommended steps to deal with a slow loop

Hi,

If a "for loop" is slow and each iteration is taking 2 hours what are the recommended steps that I should follow to deal with this issue?

Answer 1

I would recommend that you first focus on what happens inside the for loop... why is it taking so long? Get a complete breakdown of how much time is spend at every step, then you can decide where you can possibly optimise.

Further details would be great.

Answer 2

• Listen to mlint. If it is warning you about a variable changing size within a loop, that will slow it down immensely

• Remove every calculation from every loop that does not depend on the loop.

below is my code. It is taking ages for each loop. I am a beginner and took me a lot of effort to write it . but still it is not feasible to implement it as it needs a lot of time to be executed and most probably my laptop will burn before running these codes. Any help would be a life saver. cd C:\Users\Danielle\Documents\MATLAB\CS\CSV

for i=1:114
%% Import the data, extracting spreadsheet dates in MATLAB datenum format
fullFileName=sprintf('%s%d%s', 'C:\Users\Danielle\Documents\MATLAB\CS\CSV\',i, '.csv') ;
fid = fopen(fullFileName, 'rt');
M=textscan(fid,'%s','collectoutput',1,'headerlines',0);
fclose(fid);  
X=M{1,1};
fieldnames = regexp( X(1,:), ',', 'split');
V= regexp( X(2:end,1), ',', 'split');
 ncol=[4 5 10 18];%numeric columns
 scol=[1 2 3 4 6 7 11 12 13];%string columns
 vtxt=fieldnames{1,1}(1,ncol);%numeric headers
 stxt=fieldnames{1,1}(1,scol);%string headers
variablestxt=fieldnames{1,1};
v2=nan(size(V,1)-1,size(ncol,2));
s2=num2cell(nan(size(V,1)-1,size(scol,2)));
for j=2:length(V)
try
v0=[V{j,1}];
v2(j-1,1)=datenum(v0(1,4),'mm/dd/yyyy');
str = [v0{1,4}, ' ', v0{1,5}];
v2(j-1,2) = datenum(str, 'mm/dd/yyyy HH:MM:SS');
v2(j-1,3:length(ncol))=str2double(v0(1,ncol(3:end)));
 s2(j-1,1:length(scol))=v0(1,scol);
catch
    'error' %#ok<NOPTS>
end
end
try
ID=unique(s2(:,1));
catch
xe3=find(cellfun(@(x)~ischar(x),s2(:,1)));
s2(xe3,:)=[];
v2(xe3,:)=[];
ID=unique(s2(:,1));
end
Count=0;
for k=1:length(ID)
    x0=find(ismember(s2(:,1),ID(k,1))); 
    v3=v2(x0,:);
    s3=s2(x0,:);
    x1=find(max(v3(:,2)));
    v(Count+1:Count+size(x1,1),1:size(v2,2))=v3(x1,:);
    s(Count+1:Count+size(x1,1),1:size(s2,2))=s3(x1,:);
    Count=Count+size(x1,1);
    xe=find(ismember(s(:,5),'C'));
    s(xe-1:xe,:)=[];
    v(xe-1:xe,:)=[];
end
filename=[num2str(i) '.mat'];
cd C:\Users\Danielle\Documents\MATLAB\CS\Step1
save (filename,'s','v','stxt','vtxt')

Use the profiler to identify the bottlenecks. Just run two or three files with profiler and it will show you where to focus your efforts. Some obvious things:

• try/catch is slow. I would try and write the code to avoid those.

• Skip the call to find here:

 x0=find(ismember(s2(:,1),ID(k,1))); 

• regexp can be slow, perhaps csvread or dlmread?
• Conversions to and from cell is slow (s2=num2cell(nan(size(V,1)-1,size(scol,2))); ) avoid this if possible.

the profiler is showing that ismember and unique functions are th emost time consuming functions