Wishing to remove invalid noisy data from a graph.

I have plotted a graph of some Tempertaure vs Time data.

I have used 'hampel' and 'medfilt1' to minimise and reduce noise that occurs, while most have been cleaned up there is a period of continious noise I am wanting to remove or mitigate so it doesn't make other data hard to read.

Looking for suggestions to remove said continuous noise besides manually adding in an x-axis cut off point as this experiment will be further continued with multiple results and could become tedious.

Answer 1

[Mathieu NOE]

My first idea would be to analyse the ratio of variance vs mean value of a buffer of samples (repeat if until the end of the data) , so the noisy sections should show clearly a higher ratio vs a clean one

then replace this secton of data with a smoothed / filtered version of it (or replace with NaN's)

after some trials , maybe this code is a good starting point :

% dummy data
n=1000;
x=(0:n-1)/n;
ynf = 0.3+ max(0,sin(12*x+5*x.^2)).*exp(-3*x); % signal without noise 
% add local noise 
y = ynf;
l = 300; 
y(l:n) = y(l:n) + 0.05*randn(1,1000-l+1); % add some noise in the trailling portion


%% main code
y_out = y;

% create the low pass filtered / smoothed version of y 
k = 60;
ys = smoothdata(y,'movmean',k); % smoothed version of y 

% let's detect when we have too much noise content and replace y with ys in
% those sections
[B,A] = butter(2,0.03,'high');
yhp = filtfilt(B,A,y);
% logic signal 
ls = abs(yhp);
k = 50;
ls = movmean(ls.^2,k);
ls = ls./max(ls);
idx = (ls>0.1);
y_out(idx) = ys(idx);

figure(1)
plot(x,y,'b',x,y_out,'r',x,ynf,'g--')
legend('noisy signal','denoised signal','noise free signal');