what is an efficient way to save array of struct partially using the matfile() command

1 view (last 30 days)
To get around some memory issues, i want to save an array of large structs in every loop to disc using the matfile command. However, the write to disc increases in time for every loop, which was unexpected. To present the issue, i have created a minimum working example
vecSize = 20000;
numRuns = 1000;
B = genData(vecSize);
save('test','B','-v7.3');
clear B;
mfileh = matfile('test.mat','Writable', true);
time = NaN(numRuns,1);
for i = 1:numRuns
tic;
[nrows,~] = size(mfileh,'B');
mfileh.B(nrows+1,:) = genData(vecSize);
time(i) = toc;
end
function out = genData(vecSize)
out.a = zeros(1,vecSize/2);
out.b = zeros(1,vecSize/2);
end
When inspecting the time variable, one can see that the write time increases over the iterations (for my PC, it start wit haround 25ms and end with around 150ms). However, when saving a matrix to disc, the write time stays constant. Modifying the genData() function in the minimum working example to
vecSize = 20000;
numRuns = 1000;
B = genData(vecSize);
save('test','B','-v7.3');
clear B;
mfileh = matfile('test.mat','Writable', true);
time = NaN(numRuns,1);
for i = 1:numRuns
tic;
[nrows,~] = size(mfileh,'B');
mfileh.B(nrows+1,:) = genData(vecSize);
time(i) = toc;
end
function out = genData(vecSize)
out = zeros(1,vecSize);
end
results in a much faster write time (around 6ms on my PC), although the data to write is the same for both examples. Is it possible to efficiently save an array of struct partially using the matfile command? Please also consider, that fields of the large struct could also be structs themselves.

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!