Despiking

This function remove spike noise from data.

Description

This function remove spike noise from data. This function originally wrote for removing spike noise in time-series water velocity data but can be used for general purpose. The basic idea comes from Goring and Nikora (2002) which considers first and second derivatives of time series signal. See detail in the reference.

Referece
- Mori, N., T. Suzuki and S. Kakuno (2007) Noise of acoustic Doppler velocimeter data in bubbly flow, Journal of Engineering Mechanics, American Society of Civil Engineers, Volume 133, Issue 1, pp.122-125.

• MATLAB release: MATLAB 7.2 (R2006a)

[Tanja]

I'm trying to use this to only remove (or interpolate over) spikes that are, lets say, >5 above the surrounding data points. Any ideas?

[Martin Wilkes]

Ignore previous post. I have now got this to work by installing the statistics toolbox. Results are rather worrying, however, as it is filtering out 100% of my data points even though WinADV rates were only ~5% using the same filter!

[Martin Wilkes]

I created my own nanmean function as suggested by Georg Stillfried above. I then tried running func_despike_phasespace3d and found I needed a nanstd function. I tried this:

function m = nanstd (x,dim)
if nargin<2, dim=1; end
nans = isnan(x);
x(nans) = 0;
sumx = sum(x,dim);
m = sqrt((sumx./sum(~nans))/sum(~nans));

...but now getting back:

Error using *
Inner matrix dimensions must agree.

Error in func_excludeoutlier_ellipsoid3d (line 97)
x2 = a*b*c*x1/sqrt((a*c*y1)^2+b^2*(c^2*x1^2+a^2*z1^2));

Error in func_despike_phasespace3d (line 101)
[xp,yp,zp,ip,coef] = func_excludeoutlier_ellipsoid3d(f,f_t,f_tt,theta);

...what have I done wrong?

[Frank]

Using this function set for a custom MatLab based ADV signal processing toolset I made. Exactly what I needed. So glad I didn't have to write this myself. Works great!
Thx!

[Christian Stranne]

Great for despiking ADCP data!

[santa teo]

doesnt work

[Shan]

So far it works fairly well for my seismic data, even though it might take some of the valid signals as spikes as well. Still, I am amazed by the quite accurate targeting this this function. Excellent work!

[Georg Stillfried]

P.S. I just noticed that this works only if x is a vector. If x is an array, the "official" nanmean will calculate the mean columnwise or along a specified dimension. What they do there is to set the NaNs to zero, sum up the columns and divide by the number of no-NaNs:

function m = nanmean (x,dim)
if nargin<2, dim=1; end
nans = isnan(x);
x(nans) = 0;
sumx = sum(x,dim);
m = sumx./sum(~nans);

Save the file as nanmin.m in a directory on the Matlab path.

hope this helps,
GS

[Georg Stillfried]

Jac Billington

write your own nanmean function, e.g.:

function m = nanmean (x)
x = x(~isnan(x))
m = mean(x)

[Jac Billington]

Is there anyway I can get round the use of nanmean for this? I don't have the statistical toolbox in order to use this.

It would be great if you could offer some advice, still pretty new to Matlab.

thanks in advance. Jaclyn.

[Evaggelos Karvounis]

Dear Mr. Nobuhito,

Could you please send me the paper again. There were errors when i was
trying to open the pdf gile.

Send it as a pdf and in a zip file if its possible.

Thanks in advance...

Evaggelos Karvounis, PhD

[landulfo Silveira]

The algorithm leaves NAN symbols in the place of removed spikes. I coudn't use it.

[D. Januar]

Works realy fine with 3D face data, but some interpolation algorithm for filling the holes would be fine!