Correcting Analogue signal due temperature and humidity variations.?

I have attached my dataset as a reference, which I believe can be downloaded. I was hoping to have my data displayed as a result of my code on here, but I could not get it working, so I have attached images instead.

I have an analogue signal from the electrochemical sensor over a period of a few months. There is also temperature and humidity in the same Time Table.

I also have reference data. My sensor data does not need to be identical but uses the reference data to ensure that the general trend is met. This is more as confirmation that the sensor is working correctly.

The first plot shows Temperature(Blue) and Humidity(Red). The second plot my sensor(blue) and ref data(red).

The issue I am facing is that my sensor data needs to be corrected due to the environmental effects of temperature and humidity. Temperature correction is fairly simple, as you need to create some form of a lookup table for each temperature step.

But when it comes to humidity or change in humidity, that is where I have some challenges. As you can see, it creates some type of transients, negative and positive spikes. The negative spike is more often as humidity levels drop.

I have performed a simple process of creating NaN when the sensor is producing negative values and then using fillmissing() with a moving mean, but that does not solve the issue with the positive spikes or accurate results when spikes are much shorter and not negative.

When you look at the spikes, you can identify the area at the bottom to have some characteristic of what the signal should be, but the sudden drop or rise in amplitude needs to be reduced or corrected.

Has anyone else had similar issues, and what method did you use?

Answer 1

[Mathieu NOE]

hello

I have not experienced a similar situation, but maybe my idea is worth a look

I wanted to find the local positive peaks we can find on both hum and sensor data

then I created a new signal based on mixing the raw and a smoothed version of it.

So when you are in the vicinity of the identied peaks , the new signal is mostly the smoothed version of the raw signal (and vice versa)

see the result in yellow : most of the time it's the same signal as the raw data , only at some locations we take instead the smoothed signal

now your local dips and peaks are smoothed out

load('mydata.mat');

t1 = refdata.Time;
Ref_data = refdata.Var1;


t2 = exampleData.Time;
Sensor_data = exampleData.Sensor;
Temp_data = exampleData.Temp;
Hum_data = exampleData.Hum;


%% zoom data on first 80 samples 
ind = 1:80;
t2 = t2(ind);
Sensor_data = Sensor_data(ind);
Hum_data = Hum_data(ind);

% find common x indexes where Sensor and Hum data has local maxima
tf1 = islocalmax(Hum_data,'MinProminence',1,'ProminenceWindow',4);
tf2 = islocalmax(Sensor_data,'MinProminence',1,'ProminenceWindow',4);
locs = find(tf1&tf2);

Sensor_data_s = smoothdata(Sensor_data,'gaussian',13);


%% create mix ratio profile
% ratio is close to one at vicinity of locs x indexes , zero otherwise
% gaussian peak profile
x = (1:numel(t2))';
ratio = zeros(size(x));
a = 0.07;
for ck = 1:numel(locs)
    ratio = ratio + exp(-a*(x - locs(ck)).^2);
end
% plot(x,ratio);

%% apply mix formula 
Sensor_data_mix = (1-ratio).*Sensor_data + ratio.*Sensor_data_s;

subplot(2,1,1);
plot(t2,Sensor_data,t2,Sensor_data_s,'--',t2,Sensor_data_mix);
legend('Sensor data (raw)','Sensor data (smoothed)','Sensor data (mix)');
subplot(2,1,2);
plot(t2,Hum_data,t2(locs),Hum_data(locs),'dr');
legend('Hum data','Hum peaks selected');

[Dharmesh Joshi]

Hi

Thank your message.

It seems that smoothing might not work for this, due to humdity changes the signal drops by 30, where it almost reached -20.

The blue signal needs to be similar to the red signal.

The smoothing would work for small spikes, but these changes are droping in large values due to humdity so we need work our some correction factor.

[Mathieu NOE]

hello again

well ok, I thought you were only looking for small corrections inside this region - according to your post above :

now if you say the blue curve must be globally corrected so it matches the orange curve (bellow image),

I don't know what kind of (massive) correction we could "scientifically" derive from the humidity data . it seem the difference between the two curvesis not a simple linear transformation ... what do you expect from this correction ? a sketch of the expected result maybe usefull here

[Dharmesh Joshi]

Hi

Thank You for your reply.

The differnce between the two plot to be closer, So this data neets to be shift upwards, without effecting the other data.

[Mathieu NOE]

hello again

I tried a very simple approach, assuming we would need a shift amount which is proportionnal to the difference of Hum vs 100. So more shift is created as Hum decreases. You can further tweak it with the "gain" factor

the balck curve is the data afetr variable shift is applied . It get's closer to the ref signal , but still there are differences ...

not 100% sure it fullfils your dreams ....

load('mydata.mat');

t = exampleData.Time;
Ref_data = refdata.Var1;
Sensor_data = exampleData.Sensor;
Temp_data = exampleData.Temp;
Hum_data = exampleData.Hum;
%% zoom data on first samples 
ind = 1:100;
t = t(ind);
Ref_data = Ref_data(ind);
Sensor_data = Sensor_data(ind);
Hum_data = Hum_data(ind);

% correction formula
gain = 1; % drives the amount of shift correction, according to humidity drop vs level 100
shift = gain*(100 - Hum_data);
Sensor_data_corr = Sensor_data + shift;
%% apply mix formula 
plot(t,Ref_data,t,Sensor_data,t,Sensor_data_corr,'k');
legend('Ref data','Sensor data (raw)','Sensor data (shifted)');

[Mathieu NOE]

hello

did it solve your problem ?

[Dharmesh Joshi]

Sorry for the delay,

It worked to a cetain degree. I am also expirmenting with Regresion Application.

[Mathieu NOE]

No problem

let me know if I can still help you