Interpolation of noisy data

Hi,

I have the following plot (can be plotted with plot(B,C) based on the attached mat file):

However, I would like to plot the "smoothed" trend, i.e something like:

What would be the best approach? I have seen the splines function, but I am not sure if this is what I am looking for.

[John D'Errico]

Note that interpolation is the act of passing a curve EXACTLY through your data. You have no desire to interpolate. You want to do smoothing, thus approximation, so find a curve that passes essentially smoothly through the middle of your data. That is NOT interpolation, so you are asking for the wrong thing.

[MiauMiau]

Thanks for the clarification!

Answer 1

[Image Analyst]

You can use a clever trick to get unique numbers, and avoid the problem Star brought up. You can add an insignificant amount of noise and then sort the values. Try this:

% Load data.
storedStructure = load('dataForPlot.mat')
B = storedStructure.B;
C = storedStructure.C;

% B data (x axis) is not unique yet.  
% Add noise and then sort to make it unique.
bPlusNoise = B + 0.00001 * rand(1, length(B));
% Now all B will be unique.
% Assume B is x and C is y
% Sort x in ascending order.
[x, sortOrder] = sort(bPlusNoise, 'ascend');
% Sort y in the same order.
y = C(sortOrder);
% Plot
plot(x, y, 'b-');
grid on;
drawnow;

% Filter the y data
windowWidth = 301; % Whatever - bigger odd number for more smoothing.
kernel = ones(1, windowWidth) / windowWidth;
% ySmoothed = conv(y, kernel, 'same');
ySmoothed = sgolayfilt(y, 1, windowWidth);
% Plot
hold on;
plot(x, ySmoothed, 'r-', 'LineWidth', 3);
legend('Original Data', 'Smoothed Data');
xlabel('B', 'FontSize', 20);
ylabel('C', 'FontSize', 20);
title('C and Smoothed C    vs.    B', 'FontSize', 20);
% Enlarge figure to full screen.
set(gcf, 'Units', 'Normalized', 'Outerposition', [0, 0.05, 1, 0.95]);

What do you think? Will this work for you?

[MiauMiau]

Many thanks for that! I also read the point made by Star and was just thinking about it. Just to give some background on the data:

- The underlying analysis is based on pixels from an image. Chosen is one particular pixel p*, and the x axis represents the distance of all other pixels of the image to pixel p*.

- The y axis represent classification accuracies corresponding to the pixel whose distance to pixel p* is calculated.

Hence the plot shows classification accuracies as a function of the distance to the specific pixel p* selected. There can be "duplicates" since several pixels can have the same distance to pixel p* (but not necessarily the same classification accuracy).

I feel the way you did it, none of the information is lost - you did not really remove any "duplicates", right?

[jonas]

The random noise is a really nice feature for interpolation, but I don't know if it's really needed when smoothing data.

[Image Analyst]

If you don't have unique values then built-in functions won't be able to smooth them. I did not remove any duplicates, I just used the noise trick and moved them sideways by a completely insignificant amount so they're not at exactly the same x value anymore.

By the way, that noise trick can also be used in other situations, like to produce perfectly flat histograms when doing histogram equalization, instead of the non-flat histograms traditional methods produce. It's a nice trick to add to your bag of tricks.

Just full disclosure - the smoothing is done only on the y values, so it's over elements. It does not take into account that the elements may have different spacing (non-uniform x values), though as you can see from the fit, it doesn't seem to hurt the fit at all.

[MiauMiau]

That is such a nice/useful "trick" - thank you! I have a related question: How would it be possible to plot a histogram based on this data - something like the average accuracy per bin - do you know if there is a standard approach to that?