MATLAB Answers

0

what is an efficient way to save array of struct partially using the matfile() command

Asked by Konrad Rudin on 28 Nov 2017 at 15:57

To get around some memory issues, i want to save an array of large structs in every loop to disc using the matfile command. However, the write to disc increases in time for every loop, which was unexpected. To present the issue, i have created a minimum working example

vecSize = 20000;
numRuns = 1000;
B = genData(vecSize);
save('test','B','-v7.3');
clear B;
mfileh = matfile('test.mat','Writable', true);
time = NaN(numRuns,1);
for i = 1:numRuns
    tic;
    [nrows,~] = size(mfileh,'B');
    mfileh.B(nrows+1,:) = genData(vecSize);
    time(i) = toc;
end
function out = genData(vecSize)
out.a = zeros(1,vecSize/2);
out.b = zeros(1,vecSize/2);
end

When inspecting the time variable, one can see that the write time increases over the iterations (for my PC, it start wit haround 25ms and end with around 150ms). However, when saving a matrix to disc, the write time stays constant. Modifying the genData() function in the minimum working example to

vecSize = 20000;
numRuns = 1000;
B = genData(vecSize);
save('test','B','-v7.3');
clear B;
mfileh = matfile('test.mat','Writable', true);
time = NaN(numRuns,1);
for i = 1:numRuns
    tic;
    [nrows,~] = size(mfileh,'B');
    mfileh.B(nrows+1,:) = genData(vecSize);
    time(i) = toc;
end
function out = genData(vecSize)
out = zeros(1,vecSize);
end

results in a much faster write time (around 6ms on my PC), although the data to write is the same for both examples. Is it possible to efficiently save an array of struct partially using the matfile command? Please also consider, that fields of the large struct could also be structs themselves.

  0 Comments

Log in to comment.

0 Answers

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Win prizes and improve your MATLAB skills

Play today