Filtering white noise signal from acceleration data

I have white noise that I'd like to filter out of a signal. The data is from an accelerometer that is recording milling data. At the start of the data collection I have 1-2 seconds of just the white noise and I'd like to use a filter that removes it but keeps the rest of the signal when the milling is occuring. The noise and the good data both occupy the same frequency range so I don't want to use a bandpass filter. Here is the white noise of the data.

Here is the noisy data that I want to filter.

I don't know if there is a feasible filtering method I could use but I thought I'd ask here first.

[dpb]

You forgot to attach the input data trace...a .mat file would be good way.

[dpb]

If it is truly noise, then the most effective solution is to average; the noise will average out leaving the deterministic signal. This requires a stationary signal which at least this trace appears to be pretty good...

Alternatively, what's the source of the noise and can the signal quality be improved to reduce it? The scale on the acceleration plot is so large, can't really judge what the relative magnitudes are.

It the process is stationary, another thing that sometimes works is to just subtract the reference psd from the measured...

Answer 1

[Star Strider]

The best way to deal with broadband noise is either to use wavelet denoiising or the Savitzky-Golay filter (the sgolayfilt function). I usually use a third-order polynomial, and experiment with the frame length (‘framelen’ argument) until I get the result I want. (It implements a sort of FIR comb filter, as can be seen in the Fourier transform of the result.)

As you mentioned, frequency-selective fiilters won’t work. Unfortunately, median filters or moviing-average approaches are both in a sense lowpass filters.

There is no ‘best’ way to deal with broadband noise, however some are a bit better than others.

[dpb]

From the FFT example page, the simplest of examples of averaging...

Fs = 1000;            % Sampling frequency
T = 1/Fs;             % Sampling period
L = 1500;             % Length of signal
t = (0:L-1)*T;        % Time vector
S = 0.7*sin(2*pi*50*t) + sin(2*pi*120*t);
X = S + 2*randn(size(t));
subplot(2,1,1)
plot(t,S)
title('Clean time signal')
subplot(2,1,2)
plot(t,X)
title('Noisy time signal')

Y = fft(X);
P2 = abs(Y/L);
P1 = P2(1:L/2+1);
P1(2:end-1) = 2*P1(2:end-1);
f = Fs*(0:(L/2))/L;
figure
subplot(2,1,1)
plot(f,P1)
title('Single Trace One-Sided Amplitude Spectrum')
% take a longer data sample -- use 10X just for example
L=10*L;
t = (0:L-1)*T;        % Time vector
S = 0.7*sin(2*pi*50*t) + sin(2*pi*120*t);
X = S + 2*randn(size(t));
X=reshape(X,[],10);                 % fft works by column
Y = fft(X);
L=L/10;                             % set length back to original
P2 = abs(Y/L);
P2=mean(P2,2);                      % now average the 10 one-sided PSDs...
P1 = P2(1:L/2+1);
P1(2:end-1) = 2*P1(2:end-1);
subplot(2,1,2)
plot(f,P1)
title('Average 10 Traces One-Sided Amplitude Spectrum')

illustrates the random noise is averaged out leaving the deterministic signals much cleaner. If the magnitudes are not as different as in the sample, the effectiveness is even more pronounced.

It does require being able to take sufficient data to be able to do the averaging although overlapping while not as effective will also help...

[Otto Randolph]

Okay, thanks! I'm playing around with those but I ran into another question. What I'd like to do is make a unique filter for each data set based on the 1-2 seconds of white noise present in each sample. I'd then apply this custom filter to the full data set to remove the background noise. For the filters you mentioned, do they adapt themselves to the input data or are they fixed based on the input arguments you use? If its the former, how do I extract the newly made filter to use for the full data set?

[Star Strider]

@Otto Randolph — You could certainly do that wiith sgolayfilt. If all the noise in the signals has about the same characeristic, choose a polynomial order and ‘framelen’ value that work best, and use those for all the signals. All filters work best of the signals have constant sampling intervals, so use the resample function for that.

I generally use this syntax with it:

Fs = 1/mean(diff(t));

[sr,tr] = resample(s, t, Fs);

where ‘s’ is the original signal,‘t’ the corresponding time vector, and ‘sr’ and ‘tr’ their resampled versions. There is nothing specific about whatever ‘Fs’ is. You can use whatever value for it that makes sense in the context of your signals. Ideally, it should be close to the value I calculated in this example, in order that the resampled signal closely resemble the original.

If your data are in a table, another option is to use table2timetable and then retime. It should produce the same result, although the syntax is slightly different.