How to impliment faster logical indexing?

I have several matrices that I need to add to one large matrix. The large matrix (300002x50) is split up by .001 seconds and the timing for the other 49 matrices (14250x2) are roughly .02 apart, but not uniformly distributed. I have tried 'find' function to index the entries from the smaller matricies into the larger matrix, but it was too slow. I have since tried:

            for a = 1:length(test)
                aaa = abs(AF1(:,1)-test(a,1))<10^-6;
                AF1(aaa,index)=test(a,2);
            end

Where 'test' is a 14250x2 double (time,data), AF1 is a 300002x50 double matrix and index is the column in AF1 that the data should be added. It was a bit faster, but it still takes up 99.3% (29 minutes) of the time. It works how I want it, but is there any way to impliment this faster?

The timings for each of the 49 smaller matricies are not the same, but should I be adding all of these into one matrix then merging it with AF1?

[per isakson]

Try replace

(aaa==1)

by

aaa

[dpb]

Are these monotonic time vectors in each? That is, is the result of

sum(aaa) == 1

for each pass? If so, would seem could at a minimum begin each search from the succeeding point and thus be reducing size each time.

If it's a "one to many" substitution have to think about it. I agree w/ per, there's no sense in adding another logical operation inside an already logical addressing vector so that's bound to help some.

A sample dataset to look at/'spearmint with might lead to others having more specific suggestions utilizing observed characteristics in the data as noted...

[Matthew Tourtelot]

Yes, the timing is monotonic. How could I easily impliment it so it would only check the succeeding entries?

[dpb]

"Easy"? not so sure otomh...as said, a relatively small sample dataset that is illustrative would likely stimulate the thought process...I can think of some possible ideas but it's generally a of these that only really get a good idea when have a concrete example to tackle. And, as an aside, if attach a file, make it a text file; my newsreader is having trouble with the links to other file formats so can't get to 'em...

Answer 1

[Kelly Kearney]

I often use interp1 with nearest-neighbor interpolation as a way of matching up not-exactly-matching values in one matrix with another.

The idea here is to calculate which row and column each value should end up in, then place them there all at once. The row is calculated by nearest-neighbor interpolation, and the column is set manually based on the dataset number.

The setup: AF1 holds time values in the first column, and NaN placeholders in everywhere else. The test matrices have time values in the first column and data values in the second column.

AF1 = nan(3002,4);
AF1(:,1) = (0:3001)*0.001;

test{1} = [rand(14260,1)*3 (1:14260)'];
test{2} = [rand(14260,1)*3 (1:14260)'];
test{3} = [rand(14260,1)*3 (1:14260)'];

nt = size(AF1,1);
nx = size(AF1,2)-1;

Method 1: Place the 3 test matrix values in AF1 all at once:

% Create time and value arrays.

t = cellfun(@(x) x(:,1), test, 'uni', 0);
t = cat(2, t{:});
x = cellfun(@(x) x(:,2), test, 'uni', 0);
x = cat(2, x{:});

% Calculate the row index of AF1 that holds the closest value to each time
% value in your other array.  The third input here can be any size.

ridx = interp1(AF1(:,1), 1:nt, t, 'nearest'); 

% Now figure out which column of your final AF1 matrix corresponds to each
% of the t values in you used as your third input above.  

cidx = ones(size(t,1),1)*(1:nx) + 1;

% Mask any values that are too far from the matched AF1 t-values (not
% within the 1e-6 toleranc you defined). 

dt = abs(reshape(AF1(ridx,1), size(t)) - t); 
mask = dt <= 1e-6;

% Translate the row-column subscript pairs into a single matrix index.
% You now have a matrix of indices the same size as your values matrix.

idx = sub2ind([nt nx+1], ridx, cidx);

% Now place your x values into the proper spots of AF1, based on the
% indices calculated above.

AF1(idx(mask)) = x(mask);

Method 2: Place test values one at a time in a loop:

for ii = 1:nx
    t = test{ii}(:,1);
    x = test{ii}(:,2);

      ridx = interp1(AF1(:,1), 1:nt, t, 'nearest'); 
      cidx = ones(size(ridx))*(ii+1);

      dt = abs(reshape(AF1(ridx,1), size(t)) - t); 
      mask = dt <= 1e-6;

      AF1(idx(mask)) = x(mask);  
  end

Method 1, placing everything at once, is faster, because you only need to call interp1 and sub2ind once.

[Matthew Tourtelot]

I can get this code to work with your example data, but when I try it with mine, I get an error with interp1. I noticed you used a 14260x49 data matrix for t. Can this be done with just 14250x2 matrices? I'm not sure I fully understand the code to modify it properly.

[Kelly Kearney]

Yes, the concept should still work if you looped over each of your 14260x2 matrices. I've edited the code above with a few more comments, to explain exactly what I'm doing at each step, and to show how it would work if you loop over your n x 2 inputs.

If you still get an error, post the details, and I can help you track it down.