Thread Subject: Moving averages over a vector in MatLAB?

Hi everyone, recently I have been doing some analysis on
some FFTS trying to find and analyze regions with peaks.
One thing we have been doing to make this process easier
is to take a moving average of the FFT before analyzing
the peaks to ensure that rogue spikes in our FFTs do not
mislead our signal processing. However, one problem I am
forseeing is just how slow our current moving average is.
Currently we are simply looking at x points on the left
side and x points on the right side of our current point
and taking a mean of that data and then setting the same
index in our new smoothed out FFT vector to be that mean
value. I would like to see if anyone knows a slick way to
accomplish this with either a built in toolbox function or
if there is a way to apply a custom written function to
each index in a vector without a for loop.

Below is some code that is similar to what we are
currently using to accomplish our FFT smoothing process.
FFTVect represents our intensity values at a given
frequency. We are simply trying to build a new "smoothed
out" FFT that does not show any thin but tall spikes that
our original FFT shows. This averaging process eliminates
this issue previous to our peak analysis but the way we
are currently doing it seems very slow and clunky. Any
suggestions?

avgWin = 10;

avgFFT = FFTvect;

for index = avgWin+1:length(avgFFT)-avgWin
    avgFFT(index) = mean(FFTVect(index-
avgWin:index+avgWin));
end

Erik Carlsten wrote:
> Hi everyone, recently I have been doing some analysis on
> some FFTS trying to find and analyze regions with peaks.
> One thing we have been doing to make this process easier
> is to take a moving average of the FFT before analyzing ...

lookfor filter

Depending on which toolbox(es) you have, results can vary widely, of
course...

--

On Nov 1, 12:21 pm, "Erik Carlsten" <ecarl...@yahoo.com> wrote:
> Hi everyone, recently I have been doing some analysis on
> some FFTS trying to find and analyze regions with peaks.
> One thing we have been doing to make this process easier
> is to take a moving average of the FFT before analyzing
> the peaks to ensure that rogue spikes in our FFTs do not
> mislead our signal processing. However, one problem I am
> forseeing is just how slow our current moving average is.
> Currently we are simply looking at x points on the left
> side and x points on the right side of our current point
> and taking a mean of that data and then setting the same
> index in our new smoothed out FFT vector to be that mean
> value. I would like to see if anyone knows a slick way to
> accomplish this with either a built in toolbox function or
> if there is a way to apply a custom written function to
> each index in a vector without a for loop.
>
> Below is some code that is similar to what we are
> currently using to accomplish our FFT smoothing process.
> FFTVect represents our intensity values at a given
> frequency. We are simply trying to build a new "smoothed
> out" FFT that does not show any thin but tall spikes that
> our original FFT shows. This averaging process eliminates
> this issue previous to our peak analysis but the way we
> are currently doing it seems very slow and clunky. Any
> suggestions?
>
> avgWin = 10;
>
> avgFFT = FFTvect;
>
> for index = avgWin+1:length(avgFFT)-avgWin
> avgFFT(index) = mean(FFTVect(index-
> avgWin:index+avgWin));
> end

This is a convolution with the rectangular
window function [1/N 1/N .... 1/N]. Look up
CONV or FILTER.

            - Randy

 Randy Poe <poespam-trap@yahoo.com> wrote in message
<1193935442.969959.94480@50g2000hsm.googlegroups.com>...
> On Nov 1, 12:21 pm, "Erik Carlsten" <ecarl...@yahoo.com>
wrote:
> > Hi everyone, recently I have been doing some analysis on
> > some FFTS trying to find and analyze regions with peaks.
> > One thing we have been doing to make this process easier
> > is to take a moving average of the FFT before analyzing
> > the peaks to ensure that rogue spikes in our FFTs do not
> > mislead our signal processing. However, one problem I am
> > forseeing is just how slow our current moving average
is.
> > Currently we are simply looking at x points on the left
> > side and x points on the right side of our current point
> > and taking a mean of that data and then setting the same
> > index in our new smoothed out FFT vector to be that mean
> > value. I would like to see if anyone knows a slick way
to
> > accomplish this with either a built in toolbox function
or
> > if there is a way to apply a custom written function to
> > each index in a vector without a for loop.
> >
> > Below is some code that is similar to what we are
> > currently using to accomplish our FFT smoothing process.
> > FFTVect represents our intensity values at a given
> > frequency. We are simply trying to build a new "smoothed
> > out" FFT that does not show any thin but tall spikes
that
> > our original FFT shows. This averaging process
eliminates
> > this issue previous to our peak analysis but the way we
> > are currently doing it seems very slow and clunky. Any
> > suggestions?
> >
> > avgWin = 10;
> >
> > avgFFT = FFTvect;
> >
> > for index = avgWin+1:length(avgFFT)-avgWin
> > avgFFT(index) = mean(FFTVect(index-
> > avgWin:index+avgWin));
> > end
>
> This is a convolution with the rectangular
> window function [1/N 1/N .... 1/N]. Look up
> CONV or FILTER.
>
> - Randy
>
Just a thought here, if you are convolving with a
rectangular window in the frequency domain, isn't that
simply a multiplication with a sync function in the time
domain, which must be a lot faster.

Regards

Dave Robinson

Why not just shorten the length of the fft so the resulting
bins are wider? You would then also have more sections to
average if you are forming a periodogram.

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in
message <fgd391$kds$1@fred.mathworks.com>...
> Why not just shorten the length of the fft so the
resulting
> bins are wider? You would then also have more sections
to
> average if you are forming a periodogram.

When I am taking the average of my FFT, I am not actually
taking an average of the entire FFT. The entire FFT shows
a frequency domain of 0-3600Hz and I am only interested in
the shape of the FFT for frequencies between 100-400Hz. I
will of course only take a moving average of the 100-400Hz
range for speed considerations but I was just throwing
this general question out there to see if anyone had any
ideas on how to go about taking the moving average of a
vector.

"Malcolm Lidierth" <ku.ca.lck@htreidil.mloclam> wrote in
message <fgd391$kds$1@fred.mathworks.com>...
> Why not just shorten the length of the fft so the
resulting
> bins are wider? You would then also have more sections
to
> average if you are forming a periodogram.

I think I misunderstood what you meant during my first
reply. Are you referring to simply changing the overall
resolution when I convert my data over the time domain
into an FFT? So that way my FFT returns a vector over the
frequency domain of say 2048 bins rather than 4096 so the
FFT appears to be smoother?

On Nov 1, 1:15 pm, "Dave Robinson" <dave.robin...@somewhere.biz>
wrote:
> Randy Poe <poespam-t...@yahoo.com> wrote in message
> <1193935442.969959.94...@50g2000hsm.googlegroups.com>...
>
> > On Nov 1, 12:21 pm, "Erik Carlsten" <ecarl...@yahoo.com>
> wrote:
> > > Hi everyone, recently I have been doing some analysis on
> > > some FFTS trying to find and analyze regions with peaks.
> > > One thing we have been doing to make this process easier
> > > is to take a moving average of the FFT before analyzing
> > > the peaks to ensure that rogue spikes in our FFTs do not
> > > mislead our signal processing. However, one problem I am
> > > forseeing is just how slow our current moving average
> is.
> > > Currently we are simply looking at x points on the left
> > > side and x points on the right side of our current point
> > > and taking a mean of that data and then setting the same
> > > index in our new smoothed out FFT vector to be that mean
> > > value. I would like to see if anyone knows a slick way
> to
> > > accomplish this with either a built in toolbox function
> or
> > > if there is a way to apply a custom written function to
> > > each index in a vector without a for loop.
>
> > > Below is some code that is similar to what we are
> > > currently using to accomplish our FFT smoothing process.
> > > FFTVect represents our intensity values at a given
> > > frequency. We are simply trying to build a new "smoothed
> > > out" FFT that does not show any thin but tall spikes
> that
> > > our original FFT shows. This averaging process
> eliminates
> > > this issue previous to our peak analysis but the way we
> > > are currently doing it seems very slow and clunky. Any
> > > suggestions?
>
> > > avgWin = 10;
>
> > > avgFFT = FFTvect;
>
> > > for index = avgWin+1:length(avgFFT)-avgWin
> > > avgFFT(index) = mean(FFTVect(index-
> > > avgWin:index+avgWin));
> > > end
>
> > This is a convolution with the rectangular
> > window function [1/N 1/N .... 1/N]. Look up
> > CONV or FILTER.
>
> Just a thought here, if you are convolving with a
> rectangular window in the frequency domain, isn't that
> simply a multiplication with a sync function in the time
> domain, which must be a lot faster.

I believe that CONV and FILTER both use that approach,
but do all the index bookkeeping for you. That's why
I suggested them. I may be wrong on CONV however,
because it does append those extra elements at the
beginning and end.

                    - Randy

Erik
Yes. Reducing the length of the FFT allows you to divide
the data into more sections. This does not reduce noise of
the individual FFTs- fewer points out, but also fewer in,
so the coefficient of variation stays at ~100%. However,
averaging the FFTs decreases the noise by the square root
of the number of data sections (i.e. FFTs). This is the
basis of the PWELCH function in the Signal Processing
Toolbox. See
http://www.nrbook.com/a/bookcpdf.php
Chapter 13.

Filtering the data as you suggest might be OK but it
depends on how you have corrected for the window: if you
have a spectral density OK, but if it is a spectrum then
only the peaks in your result are meaningful.
See
http://www.rssd.esa.int/SP/LISAPATHFINDER/docs/Data_Analysis
/GH_FFT.pdf

Thanks very much to everyone that responded. I ended up
using filter() for my solution and it decreased the time
in my running average by over a factor of 200. Needless to
say this is exactly what I was looking for!

windowSize = avgWin*2+1;
avgFFTNoise = filter(ones(1,windowSize)/windowSize, 1, ...
        rangeMinusNoiseY(cutIndexRange+avgWin));

Thanks Randy for the suggestion to use filter() to take
the running average it worked great!

On Nov 2, 3:38 pm, "Erik Carlsten" <ecarl...@gmail.com> wrote:
> Thanks very much to everyone that responded. I ended up
> using filter() for my solution and it decreased the time
> in my running average by over a factor of 200. Needless to
> say this is exactly what I was looking for!
>
> windowSize = avgWin*2+1;
> avgFFTNoise = filter(ones(1,windowSize)/windowSize, 1, ...
> rangeMinusNoiseY(cutIndexRange+avgWin));
>
> Thanks Randy for the suggestion to use filter() to take
> the running average it worked great!

If I understand correctly this can be implemented very efficiently.
Think of it this way, after you've computed the mean of all the
samples in your window you then move it one point forward. All that
really does is add a new sample to your mean and subtract the last
one. That means you've got an integrating filter and a differencing
filter, with a delay, and no multiplies. This is the basic idea behind
CIC filters, just without the decimation. Look up CIC filters and you
should get the idea, I found part of Fred Harris' Multirate book
online had the best explanation, but I can't find it now.