Thread Subject: Strategies for avoiding NaNs when doing N dimensional e.g. griddatan

Hi all

I'm currently trying to grid a 3D data set to a
[100x100x100] grid using griddata3 and I've been having a
lot of strife with NaN values in the resulting array. I
believe part of the trouble is that my experimental data
only represents a tetrahedronal section of the cubic grid.

(see the picture
http://farm4.static.flickr.com/3283/2780566774_2ee53a9441_b.
jpg which shows my data, the grid, and the NaN values that
I'm currently trying to get rid of)

To start with, I wrote a little function called
extendconvexhull which puts 8 dummy points into the
scattered x,y,z, and f(x,y,z) vectors. The 8 points
represent the corners of a cube with side lengths +/- 10%
bigger than the cubic grid and I set the function value f
(x,y,z) equal to the nearest neighbors.

This seemed to work ok at first but now with a different
data set I'm getting the same problems occuring near the
inner boundaries of the tetrahedron as seeen in the above
linked to picture (it's kind of tricky to see from that
angle but trust me - the NaNs are near the inner
boundaries).

Using John D'Errico's function 'inhull' I found that all
the grid values are inside the convex hull of my data so I
am guessing that the NaNs are RIGHT on the boundary and are
a result of rounding errors or something similar.

So my question is - what are some strategies for avoiding
these NaNs? I'm relatively new to this type of stuff but it
must be a very common problem!

I currently have 2 ideas for how to get rid of my problem -
the first would be to modify my extendconvexhull function
so that instead of just adding 8 points at the corners of
the grid extend each section of my data so that I would
have a +number+ of points on the boundaries of the enlarged
cube rather than just 8 points.

The second idea would be to just let griddata3 do its thing
and then subsequently try and interpolate the NaNs (I
believe John's function 'inpaint_nans' does this in 2D) or
else just set the NaN values equal to the nearest neighbor
since interpolation would be difficult on the boundary.

Thanks in advance for any help!

 - Tim

"Timothy " <perceptualREMOVEMEchaos@gmail.com> wrote in
message <g8gbc5$1sh$1@fred.mathworks.com>...

>
> I currently have 2 ideas for how to get rid of my
problem -
> the first would be to modify my extendconvexhull function
> so that instead of just adding 8 points at the corners of
> the grid extend each section of my data so that I would
> have a +number+ of points on the boundaries of the
enlarged
> cube rather than just 8 points.
>


Due to the nature of my data this idea actually made the
problem worse.

All help still appreciated!

 - Tim

"Timothy " <perceptualREMOVEMEchaos@gmail.com> wrote in
message <g8gbc5$1sh$1@fred.mathworks.com>...
> Hi all
>
> I'm currently trying to grid a 3D data set to a
> [100x100x100] grid using griddata3 and I've been having a
> lot of strife with NaN values in the resulting array. I
> believe part of the trouble is that my experimental data
> only represents a tetrahedronal section of the cubic grid.
>
> (see the picture
> http://farm4.static.flickr.com/3283/2780566774_2ee53a9441_b.
> jpg which shows my data, the grid, and the NaN values that
> I'm currently trying to get rid of)
>
> To start with, I wrote a little function called
> extendconvexhull which puts 8 dummy points into the
> scattered x,y,z, and f(x,y,z) vectors. The 8 points
> represent the corners of a cube with side lengths +/- 10%
> bigger than the cubic grid and I set the function value f
> (x,y,z) equal to the nearest neighbors.
>
> This seemed to work ok at first but now with a different
> data set I'm getting the same problems occuring near the
> inner boundaries of the tetrahedron as seeen in the above
> linked to picture (it's kind of tricky to see from that
> angle but trust me - the NaNs are near the inner
> boundaries).
>
> Using John D'Errico's function 'inhull' I found that all
> the grid values are inside the convex hull of my data so I
> am guessing that the NaNs are RIGHT on the boundary and are
> a result of rounding errors or something similar.
>
> So my question is - what are some strategies for avoiding
> these NaNs? I'm relatively new to this type of stuff but it
> must be a very common problem!
>
> I currently have 2 ideas for how to get rid of my problem -
> the first would be to modify my extendconvexhull function
> so that instead of just adding 8 points at the corners of
> the grid extend each section of my data so that I would
> have a +number+ of points on the boundaries of the enlarged
> cube rather than just 8 points.
>
> The second idea would be to just let griddata3 do its thing
> and then subsequently try and interpolate the NaNs (I
> believe John's function 'inpaint_nans' does this in 2D) or
> else just set the NaN values equal to the nearest neighbor
> since interpolation would be difficult on the boundary.
>
> Thanks in advance for any help!
>
> - Tim


Hi, I made a routine once, called NewtFit which you could
find in the File exchange lib <a
href="http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=8970&objectType=file">here</a>
http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=8970&objectType=file

It will never give you NaN as a result and it will
extrapolate as well. But accuracy may be bad in some cases.
It might be good to try if griddata3 cannot do
triangularization of your mesh or if you get NaN.

There is a bug there, but I can send you the corrected
version if you want.

Nearest neighbors technique may be bad sometimes, since your
x,y,z grid doesn't give you enough variation in all 3 space
dimensions. If all neighbors are on a plane, say at fixed z,
then how could you determine behavior in the z-direction?

Have you tried to scale your grid to the [0,1] cube? Then
you get better chanse to succed:

X=(X-min(X(:)))/(max(X(:))-min(X(:)));
etc.

Hope this helps a little,
Per

"Timothy " <perceptualREMOVEMEchaos@gmail.com> wrote in message
<g8gbc5$1sh$1@fred.mathworks.com>...
> Hi all
>
> I'm currently trying to grid a 3D data set to a
> [100x100x100] grid using griddata3 and I've been having a
> lot of strife with NaN values in the resulting array. I
> believe part of the trouble is that my experimental data
> only represents a tetrahedronal section of the cubic grid.

A warning. Extrapolation out that far will be
nasty, regardless of how you pick the values
for those corner points of the cube.


 
> (see the picture
> http://farm4.static.flickr.com/3283/2780566774_2ee53a9441_b.
> jpg which shows my data, the grid, and the NaN values that
> I'm currently trying to get rid of)
>
> To start with, I wrote a little function called
> extendconvexhull which puts 8 dummy points into the
> scattered x,y,z, and f(x,y,z) vectors. The 8 points
> represent the corners of a cube with side lengths +/- 10%
> bigger than the cubic grid and I set the function value f
> (x,y,z) equal to the nearest neighbors.
>
> This seemed to work ok at first but now with a different
> data set I'm getting the same problems occuring near the
> inner boundaries of the tetrahedron as seeen in the above
> linked to picture (it's kind of tricky to see from that
> angle but trust me - the NaNs are near the inner
> boundaries).

Yes, it was tricky to see where those points
fell exactly.

What version of MATLAB are you running?
The delaunay/quickhull routines have
changed some over the years, and will
(I hope) continue to improve with time and
new releases.



> Using John D'Errico's function 'inhull' I found that all
> the grid values are inside the convex hull of my data so I
> am guessing that the NaNs are RIGHT on the boundary and are
> a result of rounding errors or something similar.
>
> So my question is - what are some strategies for avoiding
> these NaNs? I'm relatively new to this type of stuff but it
> must be a very common problem!

If you are still seeing NaNs internal to the
cube that you have grown, but are on a
facet of the original tetrahedron domain,
this may be a delaunay bug that I'd seen
some years ago. It is why I asked which
MATLAB version you are using.



> I currently have 2 ideas for how to get rid of my problem -
> the first would be to modify my extendconvexhull function
> so that instead of just adding 8 points at the corners of
> the grid extend each section of my data so that I would
> have a +number+ of points on the boundaries of the enlarged
> cube rather than just 8 points.

Adding extra points on the surface of the
cube will only potentially make life more
difficult for the quickhull codes. These
codes do not "like" when they have lots of
coplanar points. It makes the tessellations
they create ambiguous, and depending on
how the code is written, a singularity may
result. I've even seen internal "holes" formed
inside the convex hull of a tessellation (some
years ago.)


 
> The second idea would be to just let griddata3 do its thing
> and then subsequently try and interpolate the NaNs (I
> believe John's function 'inpaint_nans' does this in 2D) or
> else just set the NaN values equal to the nearest neighbor
> since interpolation would be difficult on the boundary.
>
> Thanks in advance for any help!

Another choice would be to use a code like
my gridfit, if it existed in 3-d (in the public
domain.) A 100x100x100 grid might be too
large for that code anyway.

Yes, inpaint_nans3 would work nicely, were
it written for 3-d. Ugh. One more thing to
add to the list.

John

As an update to this...

I'm using R2008a so I'm not sure about any delaunay bugs
but John posted a 3d version of inpaint_nans on the file
exchange under "Inpainting nan elements in 3-d" which has
solved my problems.

It still seems strange that I'm getting NaNs when all the
grid points are inside the convex hull of the data, or what
I would have done if I was trying to work in ND with N > 3
but for now John's function is working very well!


John posted

"John D'Errico" <woodchips@rochester.rr.com> wrote in
message <g8jgap$pt8$1@fred.mathworks.com>...
> "Timothy " <perceptualREMOVEMEchaos@gmail.com> wrote in
message
> <g8gbc5$1sh$1@fred.mathworks.com>...
> > Hi all
> >
> > I'm currently trying to grid a 3D data set to a
> > [100x100x100] grid using griddata3 and I've been having
a
> > lot of strife with NaN values in the resulting array. I
> > believe part of the trouble is that my experimental
data
> > only represents a tetrahedronal section of the cubic
grid.
>
> A warning. Extrapolation out that far will be
> nasty, regardless of how you pick the values
> for those corner points of the cube.
>
>
>
> > (see the picture
> >
http://farm4.static.flickr.com/3283/2780566774_2ee53a9441_b.
> > jpg which shows my data, the grid, and the NaN values
that
> > I'm currently trying to get rid of)
> >
> > To start with, I wrote a little function called
> > extendconvexhull which puts 8 dummy points into the
> > scattered x,y,z, and f(x,y,z) vectors. The 8 points
> > represent the corners of a cube with side lengths +/-
10%
> > bigger than the cubic grid and I set the function value
f
> > (x,y,z) equal to the nearest neighbors.
> >
> > This seemed to work ok at first but now with a
different
> > data set I'm getting the same problems occuring near
the
> > inner boundaries of the tetrahedron as seeen in the
above
> > linked to picture (it's kind of tricky to see from that
> > angle but trust me - the NaNs are near the inner
> > boundaries).
>
> Yes, it was tricky to see where those points
> fell exactly.
>
> What version of MATLAB are you running?
> The delaunay/quickhull routines have
> changed some over the years, and will
> (I hope) continue to improve with time and
> new releases.
>
>
>
> > Using John D'Errico's function 'inhull' I found that
all
> > the grid values are inside the convex hull of my data
so I
> > am guessing that the NaNs are RIGHT on the boundary and
are
> > a result of rounding errors or something similar.
> >
> > So my question is - what are some strategies for
avoiding
> > these NaNs? I'm relatively new to this type of stuff
but it
> > must be a very common problem!
>
> If you are still seeing NaNs internal to the
> cube that you have grown, but are on a
> facet of the original tetrahedron domain,
> this may be a delaunay bug that I'd seen
> some years ago. It is why I asked which
> MATLAB version you are using.
>
>
>
> > I currently have 2 ideas for how to get rid of my
problem -
> > the first would be to modify my extendconvexhull
function
> > so that instead of just adding 8 points at the corners
of
> > the grid extend each section of my data so that I would
> > have a +number+ of points on the boundaries of the
enlarged
> > cube rather than just 8 points.
>
> Adding extra points on the surface of the
> cube will only potentially make life more
> difficult for the quickhull codes. These
> codes do not "like" when they have lots of
> coplanar points. It makes the tessellations
> they create ambiguous, and depending on
> how the code is written, a singularity may
> result. I've even seen internal "holes" formed
> inside the convex hull of a tessellation (some
> years ago.)
>
>
>
> > The second idea would be to just let griddata3 do its
thing
> > and then subsequently try and interpolate the NaNs (I
> > believe John's function 'inpaint_nans' does this in 2D)
or
> > else just set the NaN values equal to the nearest
neighbor
> > since interpolation would be difficult on the boundary.
> >
> > Thanks in advance for any help!
>
> Another choice would be to use a code like
> my gridfit, if it existed in 3-d (in the public
> domain.) A 100x100x100 grid might be too
> large for that code anyway.
>
> Yes, inpaint_nans3 would work nicely, were
> it written for 3-d. Ugh. One more thing to
> add to the list.
>
> John

"Timothy " <perceptualREMOVEMEchaos@gmail.com> wrote in message
<g908e6$2cd$1@fred.mathworks.com>...
>
> As an update to this...
>
> I'm using R2008a so I'm not sure about any delaunay bugs
> but John posted a 3d version of inpaint_nans on the file
> exchange under "Inpainting nan elements in 3-d" which has
> solved my problems.
>
> It still seems strange that I'm getting NaNs when all the
> grid points are inside the convex hull of the data, or what
> I would have done if I was trying to work in ND with N > 3
> but for now John's function is working very well!


It was a moderately simple hack to move
inpaint_nans into 3-d. This was the easiest
way to resolve the problem.

I believe the internal nans were coming
from a quickhull problem I've seen before,
but I hope will disappear in some future
release.

John