MATLAB Central Newsreader - interpolate missing data

interpolate missing data

Hydroman S — Tue, 06 May 2008 16:17:02 -0400

My matrix

a = magic(4); a(2,2) = NaN ; a(3,3)=NaN

a =

    16 2 3 13
     5 NaN 10 8
     9 7 NaN 12
     4 14 15 1

is missing some data, how do I interpolate missing data
along the columns?

Re: interpolate missing data

John D'Errico — Tue, 06 May 2008 16:30:21 -0400

"Hydroman S" <amirgsalem@gmail.com> wrote in message
<fvq09u$59o$1@fred.mathworks.com>...
> My matrix
>
> a = magic(4); a(2,2) = NaN ; a(3,3)=NaN
>
> a =
>
> 16 2 3 13
> 5 NaN 10 8
> 9 7 NaN 12
> 4 14 15 1
>
> is missing some data, how do I interpolate missing data
> along the columns?

inpaint nans will interpolate in two
dimensions.

http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?
objectId=4551&objectType=file

If you only want to interpolate in one
dimension, then you can most simply loop
over each column, calling inpaint_nans
repeatedly.

Or you could use interp1, but you will
need to do more work, finding the nans
yourself.

John

Re: interpolate missing data

Hydroman S — Tue, 06 May 2008 18:39:03 -0400

Thanks John,

Here is what I did, and it seems to have worked, I just
wanted to get your blessing. Also, I was not sure how to
implement a ‘spline’ interpolation using your
inpaint_nans.m function.

a = magic(4); a(2,2) = NaN ; a(3,3)=NaN
[M,N] = size(a);
    for k = 1:length(M)
a=inpaint_nans(a)
    end

a =

    16 2 3 13
     5 NaN 10 8
     9 7 NaN 12
     4 14 15 1

a =

   16.0000 2.0000 3.0000 13.0000
    5.0000 6.0000 10.0000 8.0000
    9.0000 7.0000 11.0000 12.0000
    4.0000 14.0000 15.0000 1.0000

Re: interpolate missing data

John D'Errico — Tue, 06 May 2008 18:58:03 -0400

"Hydroman S" <amirgsalem@gmail.com> wrote in message
<fvq8k7$4h2$1@fred.mathworks.com>...
> Thanks John,
>
> Here is what I did, and it seems to have worked, I just
> wanted to get your blessing. Also, I was not sure how to
> implement a ‘spline’ interpolation using your
> inpaint_nans.m function.
>
>
>
>
> a = magic(4); a(2,2) = NaN ; a(3,3)=NaN
> [M,N] = size(a);
> for k = 1:length(M)
> a=inpaint_nans(a)
> end
>
> a =
>
> 16 2 3 13
> 5 NaN 10 8
> 9 7 NaN 12
> 4 14 15 1
>
>
> a =
>
> 16.0000 2.0000 3.0000 13.0000
> 5.0000 6.0000 10.0000 8.0000
> 9.0000 7.0000 11.0000 12.0000
> 4.0000 14.0000 15.0000 1.0000

What you did was equivalent to a single call
to inpaint_nans, working in 2 dimensions.
Note the difference below between b and c.

a = magic(4); a(2,2) = NaN ; a(3,3)=NaN
b=inpaint_nans(a)

[M,N] = size(a);
c = a;
for i = 1:M
  c(:,i) = inpaint_nans(a(:,i));
end
c

b =
           16 2 3 13
            5 6 10 8
            9 7 11 12
            4 14 15 1

c =
           16 2 3 13
            5 3.6 10 8
            9 7 13.4 12
            4 14 15 1

Of course, you need to decide whether
one is more appropriate than the other.
It depends on how the columns of a are
related to each other.

John

Re: interpolate missing data

Hydroman S — Tue, 06 May 2008 19:48:03 -0400

Q: Why do I get this error, even with the example
provided?

Warning: Matrix is singular to working precision.
> In inpaint_nans at 170
Warning: Matrix is singular to working precision.
> In inpaint_nans at 170

I don't see any singularities with my matrix. Both det(b)
& det(c) are +ve real numbers

Re: interpolate missing data

John D'Errico — Tue, 06 May 2008 21:23:04 -0400

"Hydroman S" <amirgsalem@gmail.com> wrote in message
<fvqclj$81r$1@fred.mathworks.com>...
> Q: Why do I get this error, even with the example
> provided?
>
>
> Warning: Matrix is singular to working precision.
> > In inpaint_nans at 170
> Warning: Matrix is singular to working precision.
> > In inpaint_nans at 170
>
>
> I don't see any singularities with my matrix. Both det(b)
> & det(c) are +ve real numbers

What example generates this error message?
Please show me or send it to me.

Even if all of the elements are NaNs, that
message does not result in my tests.

John

Re: interpolate missing data

Hydroman S — Tue, 06 May 2008 21:44:03 -0400

% the example

a = magic(4); a(2,2) = NaN ; a(3,3)=NaN
b=inpaint_nans(a)
[M,N] = size(a);
c = a;
for i = 1:M
  c(:,i) = inpaint_nans(a(:,i));
end
c

if I paste above example in matlab 7.0.0.19920 (R14), this
is the output I get:

a =

    16 2 3 13
     5 NaN 10 8
     9 7 NaN 12
     4 14 15 1

b =

           16 2 3 13
            5 6 10 8
            9 7 11 12
            4 14 15 1

Warning: Matrix is singular to working precision.
> In inpaint_nans at 170
Warning: Matrix is singular to working precision.
> In inpaint_nans at 170

c =

           16 2 3 13
            5 3.6 10 8
            9 7 13.4 12
            4 14 15 1

and if I paste it in matlab 6.1.0.450 (R12.1), this is
what I get:

a =

    16 2 3 13
     5 NaN 10 8
     9 7 NaN 12
     4 14 15 1

??? Error: File: C:\MATLAB6p1\work\inpaint_nans.m Line:
123 Column: 16
Expected a variable, function, or constant, found "|".

Re: interpolate missing data

John D'Errico — Wed, 07 May 2008 01:12:03 -0400

"Hydroman S" <amirgsalem@gmail.com> wrote in message
<fvqjf3$mp2$1@fred.mathworks.com>...
> % the example
>
> a = magic(4); a(2,2) = NaN ; a(3,3)=NaN
> b=inpaint_nans(a)
> [M,N] = size(a);
> c = a;
> for i = 1:M
> c(:,i) = inpaint_nans(a(:,i));
> end
> c
>
> if I paste above example in matlab 7.0.0.19920 (R14), this
> is the output I get:
>
> a =
>
> 16 2 3 13
> 5 NaN 10 8
> 9 7 NaN 12
> 4 14 15 1
>
>
> b =
>
> 16 2 3 13
> 5 6 10 8
> 9 7 11 12
> 4 14 15 1
>
> Warning: Matrix is singular to working precision.
> > In inpaint_nans at 170
> Warning: Matrix is singular to working precision.
> > In inpaint_nans at 170
>
> c =
>
> 16 2 3 13
> 5 3.6 10 8
> 9 7 13.4 12
> 4 14 15 1
>
>
> and if I paste it in matlab 6.1.0.450 (R12.1), this is
> what I get:
>
> a =
>
> 16 2 3 13
> 5 NaN 10 8
> 9 7 NaN 12
> 4 14 15 1
>
> ??? Error: File: C:\MATLAB6p1\work\inpaint_nans.m Line:
> 123 Column: 16
> Expected a variable, function, or constant, found "|".

The latter error happened in an older
release of Matlab. Inpaint_nans uses a
facility that was introduced after version
6.1. So it is not backwards compatible to
the older release.

The first message is a warning message,
not an error. I'm surprised that it happens
on that matrix, since I don't get that when
I run this test. You are using an older
release of Matlab than I have, so I must
presume that it is version specific, and
resolved with my newer release. The
result is at least correct despite the
warning message.

John

Re: interpolate missing data

NZTideMan — Wed, 07 May 2008 01:33:07 -0400

On May 7, 9:44=A0am, "Hydroman S" <amirgsa...@gmail.com> wrote:
> % the example
>
> a =3D magic(4); a(2,2) =3D NaN ; a(3,3)=3DNaN
> b=3Dinpaint_nans(a)
> [M,N] =3D size(a);
> c =3D a;
> for i =3D 1:M
> =A0 c(:,i) =3D inpaint_nans(a(:,i));
> end
> c
>
> if I paste above example in matlab 7.0.0.19920 (R14), this
> is the output I get:
>
> a =3D
>
> =A0 =A0 16 =A0 =A0 2 =A0 =A0 3 =A0 =A013
> =A0 =A0 =A05 =A0 NaN =A0 =A010 =A0 =A0 8
> =A0 =A0 =A09 =A0 =A0 7 =A0 NaN =A0 =A012
> =A0 =A0 =A04 =A0 =A014 =A0 =A015 =A0 =A0 1
>
> b =3D
>
> =A0 =A0 =A0 =A0 =A0 =A016 =A0 =A0 =A0 =A0 =A0 =A02 =A0 =A0 =A0 =A0 =A0 =A0=
3 =A0 =A0 =A0 =A0 =A0 13
> =A0 =A0 =A0 =A0 =A0 =A0 5 =A0 =A0 =A0 =A0 =A0 =A06 =A0 =A0 =A0 =A0 =A0 10 =
=A0 =A0 =A0 =A0 =A0 =A08
> =A0 =A0 =A0 =A0 =A0 =A0 9 =A0 =A0 =A0 =A0 =A0 =A07 =A0 =A0 =A0 =A0 =A0 11 =
=A0 =A0 =A0 =A0 =A0 12
> =A0 =A0 =A0 =A0 =A0 =A0 4 =A0 =A0 =A0 =A0 =A0 14 =A0 =A0 =A0 =A0 =A0 15 =
=A0 =A0 =A0 =A0 =A0 =A01
>
> Warning: Matrix is singular to working precision.> In inpaint_nans at 170
>
> Warning: Matrix is singular to working precision.
>
> > In inpaint_nans at 170
>
> c =3D
>
> =A0 =A0 =A0 =A0 =A0 =A016 =A0 =A0 =A0 =A0 =A0 =A02 =A0 =A0 =A0 =A0 =A0 =A0=
3 =A0 =A0 =A0 =A0 =A0 13
> =A0 =A0 =A0 =A0 =A0 =A0 5 =A0 =A0 =A0 =A0 =A03.6 =A0 =A0 =A0 =A0 =A0 10 =
=A0 =A0 =A0 =A0 =A0 =A08
> =A0 =A0 =A0 =A0 =A0 =A0 9 =A0 =A0 =A0 =A0 =A0 =A07 =A0 =A0 =A0 =A0 13.4 =
=A0 =A0 =A0 =A0 =A0 12
> =A0 =A0 =A0 =A0 =A0 =A0 4 =A0 =A0 =A0 =A0 =A0 14 =A0 =A0 =A0 =A0 =A0 15 =
=A0 =A0 =A0 =A0 =A0 =A01
>
> and if I paste it in matlab 6.1.0.450 (R12.1), this is
> what I get:
>
> a =3D
>
> =A0 =A0 16 =A0 =A0 2 =A0 =A0 3 =A0 =A013
> =A0 =A0 =A05 =A0 NaN =A0 =A010 =A0 =A0 8
> =A0 =A0 =A09 =A0 =A0 7 =A0 NaN =A0 =A012
> =A0 =A0 =A04 =A0 =A014 =A0 =A015 =A0 =A0 1
>
> ??? Error: File: C:\MATLAB6p1\work\inpaint_nans.m Line:
> 123 Column: 16
> Expected a variable, function, or constant, found "|".

Here is an alternative routine that is much simpler than
inpaint_nans. It works on vectors and simply does linear
interpolation across gaps. It comes from Rich Pawlowicz's t_tide
package:
function y=3Dfixgaps(x);
% FIXGAPS Linearly interpolates gaps in a time series
% YOUT=3DFIXGAPS(YIN) linearly interpolates over NaN
% in the input time series (may be complex), but ignores
% trailing and leading NaN.
%

% R. Pawlowicz 6/Nov/99

y=3Dx;

bd=3Disnan(x);
gd=3Dfind(~bd);

bd([1:(min(gd)-1) (max(gd)+1):end])=3D0;

y(bd)=3Dinterp1(gd,x(gd),find(bd));