Summarizes data using a sliding window without loops

This function is extremely fast calculating a sum, mean, etc, using no loops.

Description

This function will summarize a 1-D vector of data (i.e. 1 column) based on what the user specifies as the statistic (e.g. mean, sum, etc.). It uses no loops, so it is extremely fast. However, it will use up memory as a result. I've included memory checks such that if you do not have enough memory, it will perform the moving summary using loops. Thus what would take a few seconds, will take minute(s).

The function will summarize data over a moving window incrementing one value at a time. For example, if data are hourly and the user wants a 7-day average (i.e. 168 hours), the average will be computed for each 1-hour increment, sliding the 7-day window 1-hour at a time.

UPDATED: 10/14/2010 - streamlined function, uses less memory.
UPDATED: 10/14/2010 - no more guessing on memory usage. Exact solution is used.

Example syntax:
    dataout = movingstat(datain,20,@sum);

• MATLAB release: MATLAB 7.10 (2010a)

[Kevin Sheppard]

You might want to handle some cases like @sum or @mean separately since these can both be easily computed without loops using cumsum and one subtraction. Almost all estimators which involve sums of functions of data (e.g. mean, var, std) can be computed without either loops or substantial additional memory allocation.

[Jeff Burkey]

Thanks Kevin. Cumsum would work, except the trick is how would one perform an action like that on a sliding window where one may want to sum up a 30 day total on a sliding one hour window? If there is a more intrinsic method using canned functions, I open to advice. Thanks for the feedback!!

[Rob Campbell]

You're using a lot of RAM because of the repmat, obviously. Why not reshape instead? No loops and no memory hogging. Toy example:

t=[1,1,1,2,2,2,3,3,3];
r=reshape(t,3,3)

ans =

     1 2 3
     1 2 3
     1 2 3

>> mean(r)

ans =

     1 2 3

[Jeff Burkey]

After Rob's comment, it got me thinking about Kevin's comment. I greatly simplified the function and memory usage. Below is the entire code necessary to perform the action. I inserted SUM as an example. I've updated the file, should show up soon.

sum(data(repmat(0:N-1,L-N+1,1)+cumsum(ones(L-N+1,N))),2)

[Louis]

Hi Jeff,

I haven't looked at the function yet but that's exactly what i am looking for but for an image (i.e. 2D).

Do you think i can generalize it? or will you be able to help me do that please.

Thank you

[Jeff Burkey]

Lous,

Anything is possible. :) Assuming the image is BW it seems conceptually doable. Are you wanting to summarize in a particular direction or a rectangle/square? Either way, I would think that to summarize your 2D, you would process it in 3D. The third deminsion would give you the window in 2D space. Like in this function the second dimension gives the window in 1D space. However, you'll have an edge effect that will reduce your image N pixels in both directions. I'm not sure if that would work for you. If you need the full image after the processing, you'll have to come up with a scheme to allow for a shrinking window or reflect back as it goes over the edge. I hope this makes sense. I don't really do much image processing except in ArcGIS, which has those types of tools/functions. Which makes me think that MATLAB image processing toolbox may have something like this already?

[Cris Luengo]

   conv(data,ones(N,1),'valid');
does the same as
   sum(data(repmat(0:N-1,L-N+1,1)+cumsum(ones(L-N+1,N))),2)
but is 2 orders of magnitude faster (tested on a 1x1e6 data array).

Sure, that doesn't generalize to things other than mean or sum (but note that std and var can be built using the mean filter). Your function is still useful for the median, for example.

[Jeff Burkey]

Thanks Cris. I've never even heard of conv before. Going online at mathworks, it's not redialy apparant what toolbox that function is in...maybe Signal Processing? I can find the function documentation but I can't tell what toolbox it's in. Thus, while I would like to use your solution, it's not available to me. I only have Statistical Toolbox and conv is not in that toolbox. At least I still have some utility to people with conv! Thanks for the input. I'm sure these dialogues are very helpful for all interested.

[John D'Errico]

The function conv is part of basic matlab. EVERYBODY has it. You can find it by typing

which conv

at the command line.

[Jeff Burkey]

Well, I swear when I typed in conv I got an undefined function, but I do it again and it's there. Magic!! Thanks John.

Interestingly, when I type conv in the help search, it doesn't come up but a whole bunch of convert 's nothing convolution which lead to conv. Once I found conv in the help it shows up now.