How to preallocate memory for storing data in same mat file?

5 views (last 30 days)
Sunny
Sunny on 20 Oct 2018
Commented: Guillaume on 26 Oct 2018
Hi, I wrote the below code and I would like to preallocate memory so that the code will run faster. Once I preallocate I know that I cannot use append but need to index to store output. Can you suggest how to get output for code below?
Here the value of f is a 1*5449 double. Final output is 5449*5449 double.
clc;
n=1; %system order
m=1; %number of inputs
p=6;%number of outputs
Final = [];
for i = 1:7783
for j = 1:50
if exist(['ID_',num2str(i),'_file_',num2str(j),'_Variables','.mat'],'file')
load(['ID_',num2str(i),'_file_',num2str(j),'_Variables','.mat']);
A1 = A{1};
A1 = A1 / max(abs(eig(A1)));
B1 = B{1};
C1 = C{1};
index = 1;
for k = 1:7783
for l = 1:50
if exist(['ID_',num2str(k),'_file_',num2str(l),'_Variables','.mat'],'file')
load(['ID_',num2str(k),'_file_',num2str(l),'_Variables','.mat']);
A2 = A{1};
A2 = A2 / max(abs(eig(A2)));
B2 = B{1};
C2 = C{1};
f(index) = distance1_matlab(A1,A2,B1,B2,C1,C2);
index = index + 1;
end
end
end
Final = [Final;f];
end
end
end
save('Distance','Final');
  5 Comments
Guillaume
Guillaume on 21 Oct 2018
Well, yes it's going to be much faster. You're reading each file only once. You're still doing N^2 unnecessary eigs and related calculations. And nearly 99% of the files you test for existence don't exist, so it'd be faster to do a dir so the OS just tells you which files are there.
Finally, depending on what distance1_matlab does, it may well be that your 2nd loop is not needed.

Sign in to comment.

Accepted Answer

Guillaume
Guillaume on 21 Oct 2018
Depending on what distance1_matlab does, this code could be significantly improved.
I'm also assuming that all files that match the pattern ID_*_file_*_Variables.mat' need to be loaded.
filelist = dir('ID_*_file_*_Variables.mat'); %get list of files that exist
fileids = regexp({filelist.name}, 'ID_(\d+)_file_(\d+)_', 'tokens', 'once') %extract numeric ids as text
fileids = str2double(vertcat(fileids{:})); %and convert to numeric
%you may want to sort fileids and filelist to match the order of your original loops
%it's trivial to do. For now I assume it does not matter.
filedata = struct('A', cell(numel(filelist), 1), 'B', [], 'C', []); %preallocate structure to receive file content and final result
%note that A, B and C are very poor field names.
for fileiter = 1:numel(filelist)
filecontent = load(filelist(fileiter).name));
filedata(fileiter).A = filecontent.A{1} / max(abs(eig(A{1})));
filedata(fileiter).B = filecontent.B{1};
filedata(fileiter).C = filecontent.C{1};
end
[cartprod1, cartprod2] = ndgrid(filedata); %cartesian product of all files with themselves
distance = arrayfun(@(s1, s2) distance1_matlab(s1.A, s2.A, s1.B, s2.B, s1.C, s2.C), cartprod1, cartprod2); %assumes that the result of distance1_matlab is scalar
Note that that last line assumes distance1_matlab returns a scalar. If not, change it to:
distance = arrayfun(@(s1, s2) distance1_matlab(s1.A, s2.A, s1.B, s2.B, s1.C, s2.C), cartprod1, cartprod2, 'UniformOutput', false);
If you want the result in the same form as your original Final, then:
distance = distance(:); %if scalar result out of
distance = vertcat(distance{:}); %otherwise
  2 Comments
Guillaume
Guillaume on 26 Oct 2018
I doubt that using parfor for the loading loop would help much. The slow part of that is not the processor but the disk access. If anything, it's possible that parfor will slow things down as parallel threads compete for disk access. You'll only know if you try.
I don't know if the parallel toolbox can parallelise arrayfun (I don't have the toolbox). arrayfun is a for loop in disguise. Parallelising that code could certainly result in a speed-up
However, as I've said (twice now) depending on what distance_matlab does, it's likely that this 2nd loop/arrayfun is not needed at all and that the function can be vectorised. This would probably be the most efficient way to improve your code. Hence why I asked for the details of this function.

Sign in to comment.

More Answers (0)

Products


Release

R2018b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!