Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
moving average code taking too long

Subject: moving average code taking too long

From: Sean Douglas

Date: 1 Jun, 2010 18:59:22

Message: 1 of 11

I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.

       a= 20041126 7.11
                  20041127 2.33
                  20041128 5.88
                  20041129 4.67
                  20041130 4.66
                  20041201 3.01
                  20041203 2.58
                  20041204 1.35


MA=3
n=length(a)
k=n - MA
m=1

    for q=1:n+1 - MA
    j=m:MA+m - 1
    Mavg(q)=sum(a(j,2))/MA
    m=m+1
    end

so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.

thank you sean

Subject: moving average code taking too long

From: Walter Roberson

Date: 1 Jun, 2010 19:14:27

Message: 2 of 11

Sean Douglas wrote:
> I just wrote a moving average program, but it is taking about 10 minutes
> to run and this is too long. Here is the code I am using on a small
> example.
>
> a= 20041126 7.11
> 20041127 2.33
> 20041128 5.88
> 20041129 4.67
> 20041130 4.66
> 20041201 3.01
> 20041203 2.58
> 20041204 1.35

If I understand correctly, the fastest way to do that is to use conv().

blkproc() could also be used, but that would likely be slower than conv(), and
there have been reports that blockproc(), the replacement for blkproc(), is
sometimes considerably slower than blkproc().

Another approach is to use cumsum, as you can then calculate all the averages
over the range in one go by using

(TheCumSum(MA+1:end) - TheCumSum(1:end-MA)) ./ MA

Note this is limited by the precision of the cumulative summation rather than
by the precision of each segment.

Subject: moving average code taking too long

From: Roger Stafford

Date: 1 Jun, 2010 19:16:04

Message: 3 of 11

"Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3la9$4hp$1@fred.mathworks.com>...
> I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.
>
> a= 20041126 7.11
> 20041127 2.33
> 20041128 5.88
> 20041129 4.67
> 20041130 4.66
> 20041201 3.01
> 20041203 2.58
> 20041204 1.35
>
>
> MA=3
> n=length(a)
> k=n - MA
> m=1
>
> for q=1:n+1 - MA
> j=m:MA+m - 1
> Mavg(q)=sum(a(j,2))/MA
> m=m+1
> end
>
> so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.
>
> thank you sean

  Do a cumsum on the numbers and then take differences among these MA apart. That can be done without a for-loop. It also cuts down on the number of additions required.

Roger Stafford

Subject: moving average code taking too long

From: us

Date: 1 Jun, 2010 19:19:05

Message: 4 of 11

"Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3la9$4hp$1@fred.mathworks.com>...
> I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.
>
> a= 20041126 7.11
> 20041127 2.33
> 20041128 5.88
> 20041129 4.67
> 20041130 4.66
> 20041201 3.01
> 20041203 2.58
> 20041204 1.35
>
>
> MA=3
> n=length(a)
> k=n - MA
> m=1
>
> for q=1:n+1 - MA
> j=m:MA+m - 1
> Mavg(q)=sum(a(j,2))/MA
> m=m+1
> end
>
> so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.
>
> thank you sean

a hint:

     help filter;

us

Subject: moving average code taking too long

From: John D'Errico

Date: 1 Jun, 2010 19:34:04

Message: 5 of 11

"Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3la9$4hp$1@fred.mathworks.com>...
> I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.
>
> a= 20041126 7.11
> 20041127 2.33
> 20041128 5.88
> 20041129 4.67
> 20041130 4.66
> 20041201 3.01
> 20041203 2.58
> 20041204 1.35
>
>
> MA=3
> n=length(a)
> k=n - MA
> m=1
>
> for q=1:n+1 - MA
> j=m:MA+m - 1
> Mavg(q)=sum(a(j,2))/MA
> m=m+1
> end
>
> so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.
>

I would point out that the correct way to do this is with
conv or filter. These solutions are far better choices than
the cumsum trick, which will have precision issues.

However, nobody has pointed out why your code is so
miserably slow. Learn to preallocate that array! It is
because you have not preallocated it to the final size
that your loop is slow.

John

Subject: moving average code taking too long

From: Sean Douglas

Date: 1 Jun, 2010 20:48:04

Message: 6 of 11

"John D'Errico" <woodchips@rochester.rr.com> wrote in message <hu3nbc$o7m$1@fred.mathworks.com>...
> "Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3la9$4hp$1@fred.mathworks.com>...
> > I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.
> >
> > a= 20041126 7.11
> > 20041127 2.33
> > 20041128 5.88
> > 20041129 4.67
> > 20041130 4.66
> > 20041201 3.01
> > 20041203 2.58
> > 20041204 1.35
> >
> >
> > MA=3
> > n=length(a)
> > k=n - MA
> > m=1
> >
> > for q=1:n+1 - MA
> > j=m:MA+m - 1
> > Mavg(q)=sum(a(j,2))/MA
> > m=m+1
> > end
> >
> > so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.
> >
>
> I would point out that the correct way to do this is with
> conv or filter. These solutions are far better choices than
> the cumsum trick, which will have precision issues.
>
> However, nobody has pointed out why your code is so
> miserably slow. Learn to preallocate that array! It is
> because you have not preallocated it to the final size
> that your loop is slow.
>
> John

thank you John and everyone else I am looking up "conv "and "preallocate array" so I can get this program working faster.

thanks

Subject: moving average code taking too long

From: Sean Douglas

Date: 1 Jun, 2010 20:55:21

Message: 7 of 11

"John D'Errico" <woodchips@rochester.rr.com> wrote in message <hu3nbc$o7m$1@fred.mathworks.com>...
> "Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3la9$4hp$1@fred.mathworks.com>...
> > I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.
> >
> > a= 20041126 7.11
> > 20041127 2.33
> > 20041128 5.88
> > 20041129 4.67
> > 20041130 4.66
> > 20041201 3.01
> > 20041203 2.58
> > 20041204 1.35
> >
> >
> > MA=3
> > n=length(a)
> > k=n - MA
> > m=1
> >
> > for q=1:n+1 - MA
> > j=m:MA+m - 1
> > Mavg(q)=sum(a(j,2))/MA
> > m=m+1
> > end
> >
> > so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.
> >
>
> I would point out that the correct way to do this is with
> conv or filter. These solutions are far better choices than
> the cumsum trick, which will have precision issues.
>
> However, nobody has pointed out why your code is so
> miserably slow. Learn to preallocate that array! It is
> because you have not preallocated it to the final size
> that your loop is slow.
>
> John

Hey John, Should I be using both conv and preallocating the arry, or would I just do one or the other?

thanks

Subject: moving average code taking too long

From: Walter Roberson

Date: 1 Jun, 2010 21:05:14

Message: 8 of 11

Sean Douglas wrote:

> Hey John, Should I be using both conv and preallocating the arry, or
> would I just do one or the other?

If you use conv, you will likely produce the entire answer in one command and
so will not need preallocation for this task (but might need it for whatever
you are going to do with the averages.)

Subject: moving average code taking too long

From: John D'Errico

Date: 1 Jun, 2010 21:36:04

Message: 9 of 11

"Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3s3p$igi$1@fred.mathworks.com>...
> "John D'Errico" <woodchips@rochester.rr.com> wrote in message <hu3nbc$o7m$1@fred.mathworks.com>...
> > "Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3la9$4hp$1@fred.mathworks.com>...
> > > I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.
> > >
> > > a= 20041126 7.11
> > > 20041127 2.33
> > > 20041128 5.88
> > > 20041129 4.67
> > > 20041130 4.66
> > > 20041201 3.01
> > > 20041203 2.58
> > > 20041204 1.35
> > >
> > >
> > > MA=3
> > > n=length(a)
> > > k=n - MA
> > > m=1
> > >
> > > for q=1:n+1 - MA
> > > j=m:MA+m - 1
> > > Mavg(q)=sum(a(j,2))/MA
> > > m=m+1
> > > end
> > >
> > > so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.
> > >
> >
> > I would point out that the correct way to do this is with
> > conv or filter. These solutions are far better choices than
> > the cumsum trick, which will have precision issues.
> >
> > However, nobody has pointed out why your code is so
> > miserably slow. Learn to preallocate that array! It is
> > because you have not preallocated it to the final size
> > that your loop is slow.
> >
> > John
>
> Hey John, Should I be using both conv and preallocating the arry, or would I just do one or the other?

When you append new elements to an array in a loop,
increasing the size of the array in every pass, matlab
must reallocate new (more) memory for that array every
time. This is what takes time.

When you create the result using conv, there is no
need to preallocate, since conv creates the entire array
in one step.

So loops that grow arrays should preallocate.

John

Subject: moving average code taking too long

From: Steve Amphlett

Date: 1 Jun, 2010 21:52:03

Message: 10 of 11

"John D'Errico" <woodchips@rochester.rr.com> wrote in message <hu3ug4$2e9$1@fred.mathworks.com>...
> "Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3s3p$igi$1@fred.mathworks.com>...
> > "John D'Errico" <woodchips@rochester.rr.com> wrote in message <hu3nbc$o7m$1@fred.mathworks.com>...
> > > "Sean Douglas" <seanjdouglas@hotmail.com> wrote in message <hu3la9$4hp$1@fred.mathworks.com>...
> > > > I just wrote a moving average program, but it is taking about 10 minutes to run and this is too long. Here is the code I am using on a small example.
> > > >
> > > > a= 20041126 7.11
> > > > 20041127 2.33
> > > > 20041128 5.88
> > > > 20041129 4.67
> > > > 20041130 4.66
> > > > 20041201 3.01
> > > > 20041203 2.58
> > > > 20041204 1.35
> > > >
> > > >
> > > > MA=3
> > > > n=length(a)
> > > > k=n - MA
> > > > m=1
> > > >
> > > > for q=1:n+1 - MA
> > > > j=m:MA+m - 1
> > > > Mavg(q)=sum(a(j,2))/MA
> > > > m=m+1
> > > > end
> > > >
> > > > so i think this for statement is taking way way too long, expecially when using 3000 columns of data and figuring a Moving Average of 150 rather then the MA=3 I am using in this example to show you guys what I am up to.
> > > >
> > >
> > > I would point out that the correct way to do this is with
> > > conv or filter. These solutions are far better choices than
> > > the cumsum trick, which will have precision issues.
> > >
> > > However, nobody has pointed out why your code is so
> > > miserably slow. Learn to preallocate that array! It is
> > > because you have not preallocated it to the final size
> > > that your loop is slow.
> > >
> > > John
> >
> > Hey John, Should I be using both conv and preallocating the arry, or would I just do one or the other?
>
> When you append new elements to an array in a loop,
> increasing the size of the array in every pass, matlab
> must reallocate new (more) memory for that array every
> time. This is what takes time.
>
> When you create the result using conv, there is no
> need to preallocate, since conv creates the entire array
> in one step.
>
> So loops that grow arrays should preallocate.
>
> John

If the filter has equal coefficients, consider writing a MEX function that uses a moving buffer. The full convolution will do many more floating point ops than required. It's a good learning exercise for newcomers to MEX.

Subject: moving average code taking too long

From: ImageAnalyst

Date: 2 Jun, 2010 01:05:13

Message: 11 of 11

On Jun 1, 5:52 pm, "Steve Amphlett" <Firstname.Lastn...@Where-I-
Work.com> wrote:
> If the filter has equal coefficients, consider writing a MEX function that uses a moving buffer.  The full convolution will do many more floating point ops than required.  It's a good learning exercise for newcomers to MEX.
-----------------------------------------------------------------------
Not sure why you say "The full convolution will do many more floating
point ops than required." There is a smart way to do convolutions
(actually 2 smart ways) and a dumb brainless slow way. One of the
smart ways is not even that clever - it's rather obvious. I would bet
that the MATLAB conv() method uses the smart way that does not do any
more computations than is necessary. I know in the last release they
spent a lot of time in speeding up routines, though I bet conv() has
been optimized for quite some time now. Most novice users probably
program up convolution the dumb way because that is the most obvious.
But I agree that doing it any way could be a simple way to get used to
creating MEX files.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us