MATLAB Central Newsreader - Nonuniform time moving window average

Nonuniform time moving window average

Ben — Tue, 12 May 2009 19:20:02 -0400

I need to create a 1-second moving window average on a data array with a nonuniform time series. The data and time vectors have dropouts (i.e no data recorded) and there are slight variations in the sampling rate (~100Hz). So the number of points in each window should vary. Below is a simplified example of what I am trying to do. However this does not scale well and takes entirely to long when the data sets get large. On average this would need to work on arrays ~200,000 samples in length.

time = 0:0.1:10;
time([5 25:28 45 60:62 90]) = []; %Simulated data drops
data = rand(size(time));
%Calc 1 second moving window average of the data.
windowAve = zeros(size(data));
for i = 1:length(time)
windowAve(i) = mean(data(find(time >=time(i) & time< time(i)+1)));
end

Can anyone help me with a more efficient method.

Thanks

Re: Nonuniform time moving window average

Bruno Luong — Tue, 12 May 2009 19:50:18 -0400

Please try this:

% Data
time = 0:0.1:10;
time([5 25:28 45 60:62 90]) = []; %Simulated data drops
data = rand(size(time));
dt =1;

% Engine
n=length(time);
lastfirst = n; % alternatively: find(time<=time(end)-dt,1,'last');
first=1:lastfirst;
tlast = time(first)+dt;
last = interp1(time,1:n,tlast,'linear','extrap');
last = max(min(floor(last),n),first);
%
csdata = cumsum([0 data]);
windowAve = (csdata(last+1)-csdata(first))./(last-first+1)

% Bruno

Re: Nonuniform time moving window average

Chaos — Wed, 13 May 2009 00:35:02 -0400

"Ben" <basburywvu.nospam@hotmail.com> wrote in message <guci52$fku$1@fred.mathworks.com>...
> I need to create a 1-second moving window average on a data array with a nonuniform time series. The data and time vectors have dropouts (i.e no data recorded) and there are slight variations in the sampling rate (~100Hz). So the number of points in each window should vary. Below is a simplified example of what I am trying to do. However this does not scale well and takes entirely to long when the data sets get large. On average this would need to work on arrays ~200,000 samples in length.
>
> time = 0:0.1:10;
> time([5 25:28 45 60:62 90]) = []; %Simulated data drops
> data = rand(size(time));
> %Calc 1 second moving window average of the data.
> windowAve = zeros(size(data));
> for i = 1:length(time)
> windowAve(i) = mean(data(find(time >=time(i) & time< time(i)+1)));
> end
>
> Can anyone help me with a more efficient method.
>
> Thanks

you need to be more specific on data that "drops out". what is output? seldom does a device output "nothing", either blank spaces, Inf, etc, etc. you are essentially doing a poor man's integration but have not accounted for the situation where the data is unavailable for more than a timeframe. i would directly access the hardware data stream and hex/string parse it out to find the correct codes the machine gives for non-existant data.

Re: Nonuniform time moving window average

Ben — Wed, 13 May 2009 12:08:01 -0400

"Chaos" <rothko.fan@gmail.com> wrote in message
> you need to be more specific on data that "drops out". what is output? seldom does
> a device output "nothing", either blank spaces, Inf, etc, etc. you are essentially
> doing a poor man's integration but have not accounted for the situation where the
> data is unavailable for more than a timeframe. i would directly access the hardware
> data stream and hex/string parse it out to find the correct codes the machine gives
> for non-existant data.

No there really is no data in a "drop out". The data is from an RF telemetry acquisition system so when there is interference the frame sync pattern is lost and the data recorders have a gap in the time tagged (IRIG-B) output. Also there are times when a individual minor frame of telemetry becomes corrupt. We have certain methods to determine when this happens and discard that minor frame which in general removes a single sample in a data item creating another gap in time. (More than 1 sample if super-commutated)

This particular data item has a mandatory requirement that a moving window average of a particular time interval (lets say 1 second) be less than a certain value. This has been interpreted as all data samples within that time window (not 100 samples if it is 100Hz) because of the data dropout issue described above.

To the other poster I will try what you described and post back here later today. Plus I will try to mex the code to see if it speeds it up to a reasonable runtime.