Highlights from moving_average v3.1 (Mar 2008)

4.28571
4.3 | 22 ratings Rate this file 81 Downloads (last 30 days) File Size: 97.8 KB File ID: #12276

moving_average v3.1 (Mar 2008)

Carlos Adrian Vargas Aguilera (view profile)

14 Sep 2006 (Updated )

Smooths a matrix (with/without NaN's) via recursive moving average method and eliminates data gaps.

File Information
Description

MOVING_AVERAGE(X,F) smooths the vector data X with a boxcar window of size 2F+1, i.e., by means of averaging each element with the F elements at his right and the F elements at his left. The extreme elements are also averaged but with less data, obviously. Leaving the edges intact. The method is really fast.

MOVING_AVERAGE2(X,M,N) smooths the matrix X with a boxcar window of size (2M+1)x(2N+1), i.e., by means of averaging each element with its surrounding elements that fits in the mentioned box centered on it. This one is also really fast. The elements at the edges are averaged too, but the corners are left intact.

NANMOVING_AVERAGE(X,F) or NANMOVING_AVERAGE(X,F,1) accept NaN's elements in the vector X; the latter interpolates also those NaN's elements surrounded by numeric elements.

NANMOVING_AVERAGE2(X,M,N) or NANMOVING_AVERAGE2(X,M,N,1) accept elements NaN's in the matrix X; the latter interpolates also those NaN elements surrounded by numeric elements.

[ New simple GAP filling ]:
SMOOTH_MAVERAGE(X,M,N,IND) this one smooths only the X(IND) elements. ignoring NaNs. This can be used to elimante GAPS on your data.

Each M-files has an example (see the screenshot).

Check below to see the CHANGES on v3.1.

Note: Looking the 2 dimensional code from MOVING_AVERAGE2.M (and RUNMEAN for some hints) somebody can easily make an N-dimensional MA. Would you?

Acknowledgements

This file inspired Mvaverage and Ndnanfilter.M.

MATLAB release MATLAB 7.5 (R2007b)
20 Oct 2014 sixie yu

sixie yu (view profile)

04 May 2012 AJIl Kottayil

AJIl Kottayil (view profile)

I would like to get a moving average function dealing with nan values

Comment only
04 Nov 2009 Eric

Eric (view profile)

Carlos,
I like your function moving_average, very easy to use. I have data which has small to large time gaps and I don't want to filter across the gaps. I could break the vector up at each gap but that would mean work. Any suggestion on how to handle something like this?

17 Sep 2009 almog shalom

almog shalom (view profile)

Very useful foe me (mainly for plotting)

05 May 2009 Peter

Peter (view profile)

27 Feb 2009 Kuo-Hsien

Kuo-Hsien (view profile)

Dear all, I'm dealing with gap filling on weather measurements which the NaN should be filled based on the time window of several days.(i.e., neighborhood hour of several days).

For example, one NaN at 5pm will be replaced by the mean value in the neighborhood hour of neighborhood several days. (let's say 4, 5 and 6pm of neighborhood 5 days)

Here is the bone of question I like to deal with:

values = rand(1,1000)';
fake_NaN = floor(rand(1,300)'*1000);
values(fake_NaN) = NaN;
for i = 1:length(values)
n = 24 * i * (1:5)
having_nan_index = find(isnan(values))
new_values = nanmean(values(having_nan_index * n-1:having_nan_index*n+1))
:
:
Something like that
:
:

If you have any solutions or advices, please feel free to let me know. Thanks, Michael

15 Jan 2009 hajer gharsallaoui

hajer gharsallaoui (view profile)

07 Jul 2008 Edgar Guevara Codina

It's adequate por postprocessing of spectroscopic signals, quite useful indeed.

01 Jul 2008 Carlos Adrian Vargas Aguilera

Common Aslak, you are searching for FLEAS instead of BUGS... But, ok, USERS: BEWARE of OUTLIERS and LARGE MEANS when using CUMSUM runmean method! Rather use: NDNANFILTER :)

Carlos Vargas

Comment only
01 Jul 2008 Aslak Grinsted

Hi Carlos. I meant it as a constructive criticism. The point is that the error is unnecessary, easily avoidable, and there is no speed advantage. You are right that in the specific example the error is not very big (although you make the mistake of comparing it to the mean rather than the standard deviation). However, just because the error is small in that case does not mean it is for all series. Try for example this:

m=3;
n=100000;
x=randn(n,1); x(1)=1e100;

The problem is that the outlier gives rise to huge errors in the whole smoothed series, and not just within the window.

Comment only
30 Jun 2008 Carlos Adrian Vargas Aguilera

I forget the rating for Aslak's example (1 star): he is comparing errors of precisions 1e-9=0.000000001 and 1e-13~eps. Yes 10,000 times greater, but for values with 1,000 mean which is 1'000,000'000,000 greater than the greater error. That is a great little error isn't it?

Comment only
30 Jun 2008 Carlos Adrian Vargas Aguilera

Yes, Aslak, you are right, for that reason I left commented the normal recursive way ('s' in your example).

The problem with cumsum is with the last values which became incredibly large and 'rounding' problems, I guess, appear (eliminantig the large mean, 1000, the error is reduced). But, this is only a method and the user can choose which he likes more.

For example, now I prefer to use my new function NDNANFILTER here at the FEx.

Cheers!

Comment only
27 Jun 2008 Aslak Grinsted

Nice code but ... The cumsum trick to calculate moving averages can result in unneccarily large errors under certain conditions:
* The mean is very different from zero and the series is very long.

Here's a small test that illustrates the problem by using 3 different approaches of calculating the moving average. It shows that the 'cumsum' method has errors that are more than 10000 times greater than the errors from the filter method. There is no real speed difference.

m=10;
n=300000;
x=randn(n,1)+1000; %think for example atmospheric pressure.

tic
s=nan(length(x)-m+1,1);
for ii=1:length(s)
s(ii)=mean(x(ii+(0:m-1)));
end
slowtime=toc

tic;
c=[0;cumsum(x)];
c=(c(m+1:end)-c(1:end-m))/m;
cumsumtime=toc
cumsumerror=sqrt(mean((c-s).^2))

tic;
flt=ones(m,1)/m;
f=filter2(flt,x,'valid');
filtertime=toc
filtererror=sqrt(mean((f-s).^2))

slowtime =
9.4732
cumsumtime =
0.041549
cumsumerror =
2.6456e-009
filtertime =
0.033685
filtererror =
1.4151e-013

22 Apr 2008 T V

Well commented
Check I/O numbers and errors
vectorized
quick

so: nice

06 Mar 2008 Carlos Adrián Vargas Aguilera

Found a harmless bug on MOVING_AVERAGE, line 75: extra comma in the warning! ja!

Comment only
06 Mar 2008 K R

R K: In fact, those are limitations of this moving average, but the problem with the edges are common in filtering theory. The author provides us an idea and you?

BTW: that GAP filling is what i was looking for. Thanks!

28 Jan 2008 R K

There are some problems with this routine. It doesnt work for matrixes smaller than the desired window size and it doesnt work where in areas where a full window cannot be applied.

23 Aug 2007 Gomes Glenn

Used moving_average.

It works wonderfully. Thanks.

10 Aug 2007 Carlos Adrián Vargas Aguilera

It does it Ian. moving_average2(x,0,n) smooths the rows, and moving_average2(x,m,0) the columns.

The author.

Comment only
04 Aug 2007 Ian Howat

Thanks. I would recommend adding array-operation capabilities (ie. smooth each column or row recursively).

03 Aug 2007 Kyoung ho Chong

Simple, optimizing code, thank you!!

29 Apr 2007 Roger Lew

Works exactly as described. Great find.

14 Feb 2007 Ron Meyer

Thanks for this nice set of functions. Works excellent.

14 Dec 2006 Igor Milicevic

Thank you very much, it works excellent.
Greetings
Igor

28 Nov 2006 Carlos Adrián Vargas Aguilera

Hello, Igor. Yes, I hadn't noticed NANMEAN was a function of stat toolbox. I already changed it for NAN_MEAN. It is a vector average ignoring NaN. In about 1 day it would be updated.

Comment only
28 Nov 2006 Igor Milicevic

I am having trouble with the function nanmean , which is not defined. Can somebody help me, please?
Igor Milicevic

16 Oct 2006 Carlos Adrián Vargas Aguilera

Hello Sam! Thank you for your comments. I had no problem with the example, you should get the screenshot above (without the holes), maybe it's your matlab release but the code is really simple and should work on others.

About the F, to get a centered average around an element, the number of elements to average should be odd, so 2F+1, and in this way F is the half-width of the window (check the Description above). Similar for the m,n on 2D.

Comment only
16 Oct 2006 Sam Khanjar

Hi Carlos, thank you so much for your interesting matlab code. I have one doubt about the window size in 1D moving average. The size of the window is '2F+1' can you please tell me what 'F' stand for?

Also I have tried to get the 2D moving average to work with your supplied example, i couldn't get it to work.

04 Oct 2006 R. Castro

Very usefull. Siyi: I think the loops on this programs are faster than any box FILTER, it's easier to use and also work with NaN's!

16 Sep 2006 Siyi Deng

It is functional, but I'd still prefer FILTER over for loops.

15 Sep 2006 Jonathan Zena

I cant believe it was that easy, thanks!

04 Oct 2006

1. Fixed error in the sums through the rows!

2. New files accepting elements with NaN's!

3. New screenshot

13 Nov 2006

English translation from spanish. Little changes on the submit and screenshot.

28 Nov 2006

1. STATISTICS toolbox independent (nanmean replaced by nan_mean) 2. Change category from Graphics to DSP. 3. Little changes on the submit.

05 Mar 2008

1. Fixed bugs on the 2D averaging.
2. Rewritten for clarity.
3. NAN programs are totally recursive, so, faster.
4. New added program to SMOOTH only NaN's or specific elements.
5. Now 1D works columnwise.