How to fit 3D surface to datasets (excluding specific datapoints) without Curve Fitting Toolbox

Hi all,

I want to fit a 3D surface to my dataset using a gaussian function — however, some of my data is saturated and I would like to exclude DATA above a specific value in my fit without removing columns or rows from my data.

I do not have the Matlab Curve Fitting Toolbox. I understand the Curve Fitting Toolbox can exclude datapoints from the fit without having to remove full rows — this is what I'd like to do here. Rmmissing is not the solution I am looking for as it deletes full rows.

My current code works well when I leave the saturated data in:

[OUTPUT, ~, RESIDUALS, ~, ~, ~, ~] = lsqcurvefit(FIT_FUNCTION, GUESSES, XY, DATA, LWR_BND, UPR_BND, OPTS);

but when I change the saturated cells to NaN (i.e. DATA(DATA>1000) = NaN), I get the following error:

'Finite difference Jacobian at initial point contains Inf or NaN values. lsqcurvefit cannot continue.'

Is there a way to exclude data above a specific integer value from a 3D fit without deleting rows/columns from my data?

Is there another solution other than using lsqcurvefit that would work? I have been unable to resolve this so far.

Thank you in advance!

Below is what I am working with in terms of MATLAB version and toolboxes:

MATLAB                                                Version 9.9         (R2020b)
Image Processing Toolbox                              Version 11.2        (R2020b)
Optimization Toolbox                                  Version 9.0         (R2020b)
Signal Processing Toolbox                             Version 8.5         (R2020b)

EDIT: re-worded question for clarity based on two answers given up to now.

Answer 1

[Joe Vinciguerra]

I would recommend using either:

1. rmmissing to remove NaN values, but be sure to remove the matching data points from your XY array also.
2. fillmissing to replace NaN values with a predefined constant, method, or function.

Since you're trying to fit the data you probably don't want to use fillmissing, but that depends on your application and your dataset.

For more information: Missing Data in MATLAB - MATLAB & Simulink (mathworks.com)

[B.L.]

Thanks for your comment. Using rmmissing would mean removing full columns or rows from my dataset (rows/columns which have NaNs)? Please let me know if I misunderstood...

My data file is 1024 x 1280. Therefore, I fear that using rmmissing would involve removing a few rows/columns (many data points!) due to saturated data points. I would be removing a lot more points below saturation level than actual saturated data points.

I understand the Curve Fitting Toolbox can exclude datapoints from the fit without having to remove full rows/columns. Is there a way to do that here? If not with lsqcurvefit, could this be done with lsqnonlin?

Thanks again for your time.

[Joe Vinciguerra]

Without knowing what your data looks like, you could do something like this:

DATA = rand(3)

DATA = 3×3

0.1493 0.6278 0.2672 0.5048 0.3716 0.7166 0.0373 0.4043 0.8517

DATA(DATA>0.8) = NaN

DATA = 3×3

0.1493 0.6278 0.2672 0.5048 0.3716 0.7166 0.0373 0.4043 NaN

DATA = reshape(DATA, [1, numel(DATA)])

DATA = 1×9

0.1493 0.5048 0.0373 0.6278 0.3716 0.4043 0.2672 0.7166 NaN

[A, indx] = rmmissing(DATA)

A = 1×8

0.1493 0.5048 0.0373 0.6278 0.3716 0.4043 0.2672 0.7166

indx = 1×9 logical array
   0   0   0   0   0   0   0   0   1

Then use the indx variable to selectively remove the matching data points from XY

[B.L.]

Thanks for your clarification. How would the 3D function be fitted onto the 1D vector?

[Joe Vinciguerra]

Since it's a 3D dataset, so you need to remove the entire 3-tuple (x, y, z) for any point where any part of the tuple is NaN. In other words you can't plot a 3D point with only 2 known dimensions.

Making some assumptions about your variables, I would guess XY represents a set of ordered pairs of x and y values, while DATA is a 1024x1280 array of Z values that correspent with each set of X and Y values. So, even if DATA is a 2-D array it represents a single ordered list and can be represented by a vector. But now that I've typed out 1024x1280 I wonder if you're talking not about a 3D curve but a 3D surface.

Are you dealing with a curve (i.e. line) or a surface?

What is the size of XY and what does it represent?

What is the size of DATA and what does it represent?

It might be helpful if you post your data in a .mat file.

Note: I'm not familiar with lsqcurvefit and don't have that toolbox, so I can't experiment directly with the array shape of inputs for that funciton. I can only read the documentation for it.

[B.L.]

It is a 3D surface, not a line. Thank you for helping me narrow down the query here. I should have been clearer.

Considering it's a surface and not a line, would any other toolbox/function allow (other than lsqcurvefit) manage to exclude single data points from a surface fit without removing full rows or columns?

Thanks.

[Joe Vinciguerra]

The Curve Fitting Toolbox would be appropriate for fitting surfaces.

However, you would still need to remove the NaN values as I described above. You don't need to remove entire ros and columns, just the (x,y,z) tuple if one of the components is NaN as I described above.

[B.L.]

Ok, I don't have this Toolbox at the moment, but will get a free trial to see if it works. If so, I will return to accept your answer.