How to filter noise from a time series without losing important informations?

Hello everyone,

as described in the topic I have a time series like this:

Zoomed in you can see there is a lot of noise.

I need to filter the noise in the green rectangle without losing informations like the spike in the green circle.

As can be seen in the frequency domain there is a lot going on.

I have tried smoothing and filtering endlessly with Data Analyser Toobox and the Signal Processing Toolbox. I have tried the Filter Designer just like the filter funktion with various settings like Savitzky-Golay Filter or Exponential Moving Average Filter or Bandpass. But everytime I get rid of the noise I also lose to much important informations.

Can anybody tell if and how I can achive this?

I have attached the data.mat with the time series in case there is someone who likes to try.

Best regards

Fabian

[Sharmin Kibria]

Did you try denoising it using the Signal analyzer app? I tried to denoise the signal with symlet 4 wavelet (default setting) and got the attached denoised version. I was able to preserve 98% of the original signal energy in the denoising process. Do you think it is good enough?

Thanks

Sharmin

[Fabian Lürßen]

Yes, indeed I tried. But a rest of the nois remains and although 98% of the original signal energy is preserved the form of the curves is changed.

Answer 1

[Mathieu NOE]

hello

this is my suggestion and the result : as you can see , the large amplitude spikes are not changed by the filtering (perfect overlay with raw data)

clc
clearvars

load('data.mat');

x = data(:,1);
y = data(:,2);
t = data(:,3);
dt = mean(diff(t));
Fs = 1/dt;

ind = find(t>3.9 & t < 4);
t = t(ind);
x = x(ind); 
y = y(ind); 
samples = length(y);

figure(1)
plot(t,x);

%% "smart" data smoothing 

N = 9;
xs = medfilt1(x, N,'truncate');
xs = medfilt1(xs, N,'truncate');

% replace noisy data by smoothed (only low amplitude signal is affected)
ind = find(abs(xs)<1);
xx = x;
xx(ind) = xs(ind);

figure(1)
% plot(t,x,'b',t,xs,'r',t,xx,'g');legend('Raw','Smoothed');
plot(t,x,'b',t,xx,'r');legend('Raw','"smart" Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with medfilt1' ]);
grid on

[Mathieu NOE]

hello

even simpler code , only one stage of filtering , and noisy low amplitude signal simply replaced by zeros

clc
clearvars

load('data.mat');

x = data(:,1);
y = data(:,2);
t = data(:,3);
dt = mean(diff(t));
Fs = 1/dt;

ind = find(t>3.9 & t < 4);
t = t(ind);
x = x(ind); 
y = y(ind); 
samples = length(y);

figure(1)
plot(t,x);

%% "smart" data smoothing 

N = 9;
xs = medfilt1(x, N,'truncate');
% replace noisy data by smoothed (only low amplitude signal is affected)
ind = find(abs(xs)>3);
xx = zeros(size(x));
xx(ind) = x(ind);
figure(1)
% plot(t,x,'b',t,xs,'r',t,xx,'g');legend('Raw','Smoothed');
plot(t,x,'b',t,xx,'r');legend('Raw','"smart" Smoothed');
title(['Data samples at Fs = ' num2str(round(Fs)) ' Hz / Smoothed with medfilt1' ]);
grid on

[Mathieu NOE]

hello again

does my suggestion answer your request ?

all the best