Find NaN elements (NaN-islands) in the middle of a matrix and substitute them with linearly interpolated values.

This is repetitive 1D problem not 2D. We have a matrix (NxM) of N vectors with different length, M is the longest vector length. Shorter vectors are filled up with NaN at the end. Matrix can be big: N - 1000s, M - 100s. Sometimes vectors can have NaNs in the middle. They need to be substituted with linearly interpolated values from the nearest edges. This "NaN-islands" are rare and random.

So, for

A=[1  2  3   4  NaN  6   7   8   9  10;
   10 11 12  3   7  NaN NaN  13 14  NaN;
   6   8  2  5  NaN NaN NaN NaN NaN NaN;]

I would like to have

A=[1  2  3   4   5   6   7   8   9  10;
   10 11 12  3   7   9   11  13 14  NaN;
   6   8  2  5  NaN NaN NaN NaN NaN NaN;]

So far I have this working code but I would like to avoid at least the first loop as the biggest one and fires rarely. Any ideas?

for ind1=1:size(A,1) % loop for rows
    temp=bwconncomp(isnan(A(ind1,:)));
  if temp.NumObjects>1 || ~isnan(A(ind1,end)) % NaN in the middle
      for ind2=1:temp.NumObjects-1+(~isnan(A(ind1,end))) % loop all NaN-islands in the row
            tempCoor=temp.PixelIdxList{ind2};
            tMin=min(tempCoor); % index of the first NaN point
            tMax=max(tempCoor); % index of the last NaN point
            tN=(tMax-tMin)+1; % number of NaN points
            x=linspace(A(ind1,tMin-1),A(ind1,tMax+1),tN+2); % linear interpolation of missed values
            A(ind1,tMin:tMax)=x(2:end-1);
        end
    end
end

Speed is more important than memory consumption.

P.s. Code contains one bug: it does not handle rows with NaN in the middle only. I have not seen such case in my data but I'll work on it in any case. -> fixed, code updated.

Answer 1

[Image Analyst]

Have you tried isnan() and regionfill() in the Image Processing Toolbox:

A=[...
  1  2  3   4  NaN  6   7   8   9  10;
  10 11 12  3   7  NaN NaN  13 14  NaN;
  6   8  2  5  NaN NaN NaN NaN NaN NaN]
nanMap = isnan(A)
fixedA = regionfill(A, nanMap)

[Marina]

Thank you for the suggestion. However, regionfill provides different values (not linear interpolation) and also fixes NaN at the end while I need them.

fixedA =

      1.0000    2.0000    3.0000    4.0000    5.6667    6.0000    7.0000    8.0000    9.0000   10.0000
     10.0000   11.0000   12.0000    3.0000    7.0000    7.5756    9.3210   13.0000   14.0000   12.1519
      6.0000    8.0000    2.0000    5.0000    6.6605    7.9815    9.7084   11.8226   12.7594   12.4556

Will look if I can adapt it to my purposes. Thank you again.

[Marina]

Regionfill provides inward 2D interpolation (previous and next row contribute to calculation of values in current one) while my problem is repetitive 1D (all rows should be treated independently).

[Image Analyst]

Then loop row-by-row and use isnan() to get a map of where NANs are and use interp1(), something like

A=[...
  1  2  3   4  NaN  6   7   8   9  10;
  10 11 12  3   7  NaN NaN  13 14  NaN;
  6   8  2  5  NaN NaN NaN NaN NaN NaN]

[rows, columns] = size(A)
for row = 1 : rows
  goodColumns = ~isnan(A(row, :));
  goodA = A(row, goodColumns);
  % Get the x coordinates of the good columns.
  x = find(goodColumns);
  fixedRow = interp1(x, goodA, 1:columns);
  % Replace this row with the fixed one.
  A(row, :) = fixedRow;
end
% Print Final A to command window.
A

A =
     1     2     3     4     5     6     7     8     9    10
    10    11    12     3     7     9    11    13    14   NaN
     6     8     2     5   NaN   NaN   NaN   NaN   NaN   NaN

Answer 2

[Guillaume]

There was a cody problem recently very similar to this: replace nan values iteratively

This is the best scoring solution by Ly Cao (does not mean that it's the most efficient):

for i = find(isnan(x))'
   x(i) = interp1(x(x==x),i-0.5); 
end

Only works with a row vector, but you could adapt it.

[Tahir Ahsen]

Does not work!!!

[Guillaume]

As stated: Only works with a row vector.

In any case, this is all very outdated, nowadays you'd simply use fillmissing which does it all for you.

[Image Analyst]

Tahir, do you really have the very same problem as the original poster? (I doubt it) Why don't you post a complete description of your problem and attach your script and data, in a new separate question, so we can solve it for you?