large file slow running

3 views (last 30 days)
Joydeb Saha
Joydeb Saha on 20 Sep 2021
Answered: Jan on 20 Sep 2021
latmin=-90; %chooce your location
latmax=90;
longmin=-180; a=1;
longmax=180;
data=zeros(90,90);
for b=1:length(x(:,1))
if (x(b,8)>=longmin && x(b,8)<longmax && x(b,7)>=latmin && x(b,7)<latmax);
for col=1:13
data(a,col)=x(b,col);
end
a=a+1;
end
end
theres a file called x.mat (601760x13). What might be the reason it runs too slow
  1 Comment
KSSV
KSSV on 20 Sep 2021
What you are trying to achieve? It can be achived without loop. You need to give an explanation of your problem. What does x.mat have?

Sign in to comment.

Answers (1)

Jan
Jan on 20 Sep 2021
I guess that the iterative growing of the array data slows down the processing.
Remember that the growing of an array required to create a new array and to copy the foermer contents.
x = [];
for k = 1:1e6
x(k) = rand;
end
This creates a new array in each iteration. Although the final array needs 8MB only (8 bytes per double), Matlab has to allocate and to copy sum(1:1e6)*8 bytes, which are more than 4 TB ! The solution is easy: Allocate with the final size:
x = zeros(1, 1e6); % Pre-allocation
for k = 1:1e6
x(k) = rand;
end
In your case you do not know the needed size in advance. You tried to pre-allocate with data=zeros(90,90), but this is not the final size. You could determin the final size:
latmin = -90;
latmax = 90;
longmin = -180;
longmax = 180;
index = (x(:,8) >= longmin & x(:, 8) < longmax & x(:, 7) >= latmin && x(:,7) < latmax);
data = zeros(sum(index), 13);
a = 1;
for b = 1:size(x, 1) % nicer and faster than: length(x(:,1))
if index(b)
for col = 1:13
data(a, col) = x(b, col);
end
a = a + 1;
end
end
But it is easier, nicer and faster to perform the copy directly without a loop:
latmin = -90;
latmax = 90;
longmin = -180;
longmax = 180;
index = (x(:,8) >= longmin & x(:, 8) < longmax & x(:, 7) >= latmin && x(:,7) < latmax);
data = x(index, 1:13);

Categories

Find more on Dates and Time in Help Center and File Exchange

Tags

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!