Running Matrix with large amounts of data

suppose i have matrix Ui which has around 500,000 points and i have a code like this:

Ni=[];
Pi=[];
Ni=[100 100 100];
Pi=[100 100 100];
for i=1:length(Ui)
  u=Ui(1,:);
  [m,n]=size(Ni);
  [p,q]=size(Pi);
  for k=1:m
    dmin1=min(sqrt((u(1,1)-Ni(k,1))^2+(u(1,2)-Ni(k,2))^2));
  end
  for l=1:p
    dmin2=min(sqrt((u(1,1)-Pi(l,1))^2+(u(1,2)-Pi(l,2))^2));
  end
  [indx,d]=rangesearch(Ui(:,[1,2]),u(:,[1,2]),1.5);
  Vector=cell2mat(indx);
  for j=1:length(Vector)
    LocalMax=max(Ui(Vector(j),3));
  end
  if LocalMax==u(3)
      if dmin1>dmin2
        Ni(i,:)=u;
      end
      else
        if dmin1<=dmin2
            Pi(i,:)=u; 
        end 
      Ui(1,:)=[];
   if isempty(Ui)
      break; 
   end 
  end 
end
display(Pi);

When I tried it with Ui having 300 points it runs really fast however if I try it with 500,000 points it runs very slow it takes more than 3 hours. Is there a way to make the code run faster when it has 500,000 points?

Answer 1

[Jan]

Please mention the dimensions of Ui. "300 points" does not clarify both dimensions of a matrix. It helps to help you, if you post some working input data.

I do not see the purpose of:

  for k=1:m
     dmin1 = min(sqrt((u(1,1) - Ni(k,1))^2 + (u(1,2) - Ni(k,2))^2));
  end

The argument of min() is a scalar. Then the result is in every case:

  dmin1 = sqrt((u(1,1) - Ni(m,1))^2 + (u(1,2) - Ni(m,2))^2);  

The same happens for dmin2 and LocalMax. Let me guess, that instead of the loops you want:

  dmin1 = min(sqrt((u(1,1) - Ni(:,1)).^2 + (u(1,2) - Ni(:,2)).^2));

Letting an array shrink iteratively needs a lot of ressource: Ui(1,:)=[] . If you do this on a [500'000 x 3] array, this allocates 3 TB or RAM successively. Ni and Pi are growing iteratively, which suffers from the same problem. I cannot run your code and due to the useless for-loops I assume, it does not work correctly at all, but the speed can be improved by changing:

Ni=[100 100 100];
Pi=[100 100 100];

to

nUi = size(Ui, 1);
Ni = zeros(nUi, 3);   % Pre-allocation with maximum size
Pi = zeros(nUi, 3);
Ni(1, :) = [100 100 100];
Pi(1, :) = [100 100 100];

and

u=Ui(1,:);

to

u=Ui(i,:);

This is a strange construction:

for i=1:length(Ui)
    ...
    Ui(1,:)=[];
    if isempty(Ui)
        break;
    end
end

The FOR loop gets its limits when it is entered the first time. So the shrinking Ui does not change the limit. Then this would be better:

current = 1;
while current <= nUi
    u = Ui(currentm :);
    ...
    % Replace Ui(1,:)=[]; by:
    current = current + 1;
end

[A]

Hi, thank so much for your help, the Ui matirx is suppose to be (number_of_points,3) so in this case the number_of_points is 500,000. Here is the text file of some working data, on has number_of_points to be 300 i also did try it with the corrections you suggested but it still is taking a very long time to run, I did some breif calulation and it said it will take 4-5 days to finish it. Maybe cause there are 500,000 points? or is it some other function in my code that slowing it down?

[Jan]

Do you know the profiler? See:

doc profile

It helps to identify the bottlenecks of the code.

The iterativ shrinking or growing of arrays must be avoided for efficient code, but there can be other constructs also, which decrease the speed.

500'000 does not sound like a large array. Use the profile to find the section, which takes the most time. Then post this piece of code here with some working inputs, e.g. created by RAND.