Averaging data series every 10 seconds

I have 3 instruments on an aircraft with different Hz sampling rates measuring radar reflections, gravity and magnetics. I need to average each of these data over the same 10 second sampling interval. Is there a more efficient way to code this in Matlab than using loops (e.g. matrix arrays)? These are huge datasets that I am attempting to concatenate once averaged to the same sampling interval.

I appreciate any help! I was just getting started using a simple little test array:

t_grav=(1:1:50)'; t_rad=(0.1:0.1:5)'; t_mag=(0.2:0.2:10)';

x=(10:10:500)'; y=(100:100:5000)'; radar=(1:2:100)'; grav=(1:4:200)'; mag=(1:8:400)';

array=[t,x,y,radar,grav,mag];

So the results for this sample set should look like =

1 55 10 11 155 30 et cetera 21 255 50 et cetera

Answer 1

[Star Strider]

There is a core MATLAB function synchronize and the Signal Processing Toolbox function resample that can put all your data on the same sampling interval (if you don’t mind interpolated data). That could be automated in a loop so that averaging them in the resulting matrices (possibly using reshape in the process to produce 10-second records in some dimension of the matrices) would be much easier. Archive your original data of course, and use the resampled data knowing that it was resampled.

That’s how I would do it. There may be better and more efficient ways.

[Bruce frederick]

I think this method will work. If I have NaN values in the data, is that going to be a problem?

[Star Strider]

Yes, they will. You can’t have NaN in your data. It’s not in the documentation for resample, so I did this experiment to see:

x1 = [1:6  NaN  7:10];
y1 = resample(x1, 5, 2)

x2 = x1(~isnan(x1));
y2 = resample(x2, 5, 2)

Turns out that y1 fails but y2 succeeds. Note that this snippet eliminates the NaN values but does not replace them.

Before you do the resample, you can interpolate the NaN values if you want. This came up in a different context a while back, so I wrote the following code to do just that:

% INTERPOLATING ‘NaN’ VALUES IN DATA
% Create data —
x = 0:0.1:2*pi;
y = sin(x);

% Create NaNs in data — 
yn = y;
xn = x;
xn(5:5:end) = NaN;
yn(5:5:end) = NaN;

% Convert NaNs to missing values —
xm = xn;
ym = yn;
ym(isnan(ym)) = [];
xm(isnan(xm)) = [];

% Find indices of NaNs and interpolate for missing ‘x’ values —
ixn = find(isnan(xn));
for k1 = 1:size(ixn,2)
    ixr = [xn(ixn(k1)-1)  xn(ixn(k1)+1)];
    xi(k1) = mean(ixr);
end

% Use interpolated ‘x’ values to interpolate missing ‘y’ values —
yi = interp1(xm, ym, xi);

figure(1)
plot(xn, yn, 'ob')
hold on
plot(xi, yi, '+r')
hold off
grid

The code is its own demonstration, but note that it assumes NaN values are present in both the x (independent) and y (dependent) data. For your data, start after the

% Convert NaNs to missing values —

comment. If you only have missing y data, comment out the lines that interploate the x data. (You might want to make a function out of it since you have such voluminous data.)

With the NaN values removed, resample should be happy.

Answer 2

[Kelly Kearney]

Do you mean you want to average data that falls within certain intervals, for all datasets? If so, accumarray paired with histc should do the trick:

t_grav=(1:1:50)'; 
t_rad=(0.1:0.1:5)'; 
t_mag=(0.2:0.2:10)';

x=(10:10:500)'; 
y=(100:100:5000)'; 
radar=(1:2:100)'; 
grav=(1:4:200)'; 
mag=(1:8:400)';

tbin = 0:50;
[n, idx{1}] = histc(t_grav, tbin);
[n, idx{2}] = histc(t_rad,  tbin);
[n, idx{3}] = histc(t_mag,  tbin);

sz = [max(cat(1, idx{:})) 1];

data(:,1) = accumarray(idx{1}, grav,  sz, @mean, NaN);
data(:,2) = accumarray(idx{2}, radar, sz, @mean, NaN);
data(:,3) = accumarray(idx{3}, mag,   sz, @mean, NaN);

[Bruce frederick]

No. I essentially need to resample the data such that each instrument has the same sampling interval. In this sample case, say 0.2s, so that all data would have to be interpolated over the new time interval using existing datapoints that fall within the new interval. In this sample case, the new time range would be 1 to 5s since that is all the 3 datasets have in common.

[Kelly Kearney]

In that case, a simple interp1 might do the trick.

tnew = 1:0.2:5;
data(:,1) = interp1(t_grav, grav, tnew);
data(:,2) = interp1(t_rad, radar, tnew);
data(:,3) = interp1(t_mag, mag, tnew);

If some of your data is much higher-frequency than others, and noisy, it might make sense to run a moving average or similar across the data before interpolating, to prevent the interpolation from hitting on an anomalously low or high value.

Answer 3

[Jaeyoung]

Hi all,

Sorry, I guess my problem is very similar to what you are looking for. In my case, I want to sample one output (not averaging) in every 1us(micro second) when the current data is not acquired by 1us.

I have measurement data form Logic Analyzer. I measured my chip using 1 Mhz clock, and saved data to a csv file. I want to have one output in every 1us since my clock is 1Mhz. However, the data was not measured in every 1ns. I do understand a Logic Analyzer is smart so it reduces a step size in a transition, and the logic analyzer increase a step size when outputs do not change in long time. However, it makes me crazy to collect and analyze data. How do I solve this issue?

The first column shows time, and the second column shows value. When it transits from 0 to 1. Time step is smaller than 1us since a logic analyzer tried to measure it precisely. When there are sequential ones, it increases a step size to 2us or more. I just want to sample one data in every 1us. I would appreciate you help me. Thank you in advance.

Time (sec) Value Time step (sec)

-1.048596968 0

-1.048596072 1 0.000000896

-1.048595072 1 0.000001

-1.048594072 1 0.000001

-1.048592072 0 0.000002

-1.048588072 1 0.000004

-1.048587072 1 0.000001

-1.048586072 1 0.000001

-1.048585072 1 0.000001

-1.048584072 1 0.000001

-1.048583072 1 0.000001

-1.048580586 1 0.000002486

-1.048580072 1 0.000000514

-1.048577072 1 0.000003

-1.048576072 1 0.000001

-1.048575072 0 0.000001

-1.048565072 1 0.00001

-1.048564072 0 0.000001

-1.048563072 0 0.000001

Best,

Jaeyoung