I want to only interpolate shorter amounts of data and leave the longer amounts of NaN not interpolated.

I have interpolated my data by using:

nanp = isnan(data);
t = 1:numel(data);
data(nanp) = interp1(t(~nanp), data(~nanp), t(nanp), 'linear');

The issue is all of the data gets interpolated, including big chunks of data that I would rather leave empty. Is there a way to make guidelines for what data gets interpolated? Such as "If more than x NaN's in a row, then do not interpolate that chunk of data".

Answer 1

[kaan ilter]

As I understand you would like to remove some outliers. How about removing some values by using the following filter?

datanew=data(data>9999,:)=[];

Answer 2

[Matt J]

If more than x NaN's in a row, then do not interpolate that chunk of data

Using function from here,

https://www.mathworks.com/matlabcentral/fileexchange/78008-tools-for-processing-consecutive-repetitions-in-vectors

this can be done with,

G=groupTrue(nanp);

[~,~,runlengths]=groupLims(G,1);

doInterp=ismember( G,find(runlengths<=x));

data(dointerp) = interp1(t(~nanp), data(~nanp), t(doInterp), 'linear');

Answer 3

[Walter Roberson]

max_length_to_interpolate = 17;   %change as appropriate

nanp = isnan(data(:).');  %must have row vector
starts = strfind([false nanp], [false true]);
stops = strfind([nanp false], [true false]);
mask = (stops - starts + 1 <= max_length_to_interpolate);
starts = starts(mask); stops = stops(mask);
if ismember(1, starts) || ismember(numel(data), stops)
     error('asked to interpolate leading or trailing nan, which we cannot do');
end

nan_replacements = arrayfun(@(B,E) interp1(t([B-1,E+1]), data([B-1,E+1]), t(B:E), 'linear'), starts, stops, 'uniform', 0);

for K = 1 : length(starts)
   data(starts(K):stops(K)) = nan_replacements{K};
end

if really needed it would be possible to do the final assignments without a loop.

Answer 4

[Image Analyst]

This works. It uses the bwareaopen() function in the Image Processing Toolbox to throw out stretches of nans longer than you specify.

% Create sample data.
y = [10,20,30,nan, nan, 60,70,80, nan, nan, nan, nan, 90,100,110]
% Get the indexes we're supposed to interpolate over.
xOriginal = 1 : length(y);
% Find nan locations
nanMask = isnan(y)
% Define the longest stretch of nans you'll interpolate over.
nanLength = 3;
% Get mask of stretches over that length.
longNanMask = bwareaopen(nanMask, nanLength) % Needs Image Processing Toolbox.
% Get non-nan data:
x2 = xOriginal(~nanMask);
y2 = y(~nanMask)
% Interpolate over everything, both long and short nan runs.
v2 = interp1(x2, y2, xOriginal)
% Replace long nan stretches with nans
v2(longNanMask) = nan
fprintf('Done!\n');

y =

10 20 30 NaN NaN 60 70 80 NaN NaN NaN NaN 90 100 110

nanMask =

1×15 logical array

0 0 0 1 1 0 0 0 1 1 1 1 0 0 0

longNanMask =

1×15 logical array

0 0 0 0 0 0 0 0 1 1 1 1 0 0 0

y2 =

10 20 30 60 70 80 90 100 110

v2 =

10 20 30 40 50 60 70 80 82 84 86 88 90 100 110

v2 =

10 20 30 40 50 60 70 80 NaN NaN NaN NaN 90 100 110

Done!