Zeroeing baseline & measuring noise in noisy signal

I am analysing voltage signal acquired at 40KHz sampling frequency, my aim is to detect an event at an exact time, I attach example. I have 2 questions regarding signal processing.

1. What is the most appropriate way to zero the signal?

I tried subtracting the mean measured in a window of variable lengths - anything from 40 to 4000 points. But as you can see the signal is quite noisy with lots of spontaneous events, so at times if the window happens to include lots of events the mean will be affected and the zeroing will not work as well.

2. How to best measure the standard deviation of baseline noise?

To establish if anything occurred at my timepoint of interest, I would like to measure baseline noise standard deviation, and set a criterion that if the peak of the event of interest is larger than 4*standard deviation of the noise then the event has occurred. However, again, because the signal is so noisy, I am not sure how to approach measuring baseline noise.

I am new to signal processing, and I want to make sure I am not making any silly mistakes, any input will be much appreciated!

[Image Analyst]

What does this signal represent? A Raman spectroscopy signal or something?

s = load('example.mat')
plot(s.example, 'b-');
grid on;

Why do you say that the baseline signal has noise? Wouldn't the baseline be defined as having no noise and then you subtract that from the signal and then all the noise is in the resulting signal?

[bull113]

The signal is a recording of voltage changes in a nerve cell. Sorry, as I said I am quite new to this analysis, so I am getting terms mixed up. This trace consists of true signal due to neuronal activity which are basically all the triangual-ish events, and also electrical signal generated by equipment, that electrical signal is what I consider baseline noise, but now I see how that's not right. In my analysis, I set a time window where I detect max value of the signal, and then I need to have a way to set a criterion as to whether this max value is actually neuronal activity or background noise, I thought the best way would be to use standard deviation of the noise, ie max values < 3*std of background noise would not be considered to be true neuronal signal. I was wondering if you thought that is a reasonable approach? Sorry if I am not explaining this too well!

Answer 1

[Image Analyst]

There is noise everywhere - on the low valleys and even on the high peaks. If you want you can just set a threshold and zero out the signal if it falls below some threshold:

signal(signal < someValue) = 0;

or you can try to smooth the whole thing with sgolayfilt(). See attached demo.

[bull113]

That works really well, thank you! Sorry to bother you with another question, but I was wondering if you would have any suggestions what's the most appropriate way to measure the mean and standard deviation of the baseline post filtering? Measuring it in one time window near region of interest often doesn't work as there are a lot of spontaneous neuronal events, and then the mean and standard deviation come out biased. Sorry if this is not really the right approach, I am trying to figure out how to set a detection limit.

[Image Analyst]

I'm still not sure what you consider to be valid signal, signal below the noise threshold, baseline, or just pure noise. What's wrong with just dealing with the signal as it is? Why do you need to filter it, and if it were ideally filtered, what would it look like and what would you then do with the resulting signal? I can't really answer until I know these things.

[Image Analyst]

Still not sure how to help you but maybe if you subtract the smoothed signal from the original one you get noise only and maybe that will help:

% Initialization steps.
clc;    % Clear the command window.
close all;  % Close all figures (except those of imtool.)
clear;  % Erase all existing variables. Or clearvars if you want.
workspace;  % Make sure the workspace panel is showing.
format long g;
format compact;
fontSize = 20;

s = load('example.mat')
signal = s.example;
subplot(3, 1, 1);
plot(signal, 'b-');
grid on;
title('Original Signal', 'FontSize', fontSize);

smoothedSignal = sgolayfilt(signal, 2, 1001);
subplot(3, 1, 2);
plot(smoothedSignal, 'b-');
grid on;
title('Smoothed Signal', 'FontSize', fontSize);

noiseOnly = signal - smoothedSignal;
subplot(3, 1, 3);
plot(noiseOnly, 'b-');
grid on;
title('Noise-only Signal', 'FontSize', fontSize);

[bull113]

Sorry for being confusing! My ultimate aim is to analyse multiple repetitions of recordings like the example below. Then I would like to check whether at a certain timepoint there was any signal. So in the attached picture which shows region of interest from 2 experimental measurements, I would like to have an algorithm that will pick up the peak and something that will classify the other trace as a lack of signal. So I thought the best way to do it would be to measure standard deviation (or maybe height) of background noise and then have an if statement that rejects that flat line as signal if the max peak of that recording is less than 3*standard deviation of the noise. But then I ran into problems measuring background noise as there are a lot of spontaneous neuronal events in the recordings.

[Image Analyst]

You could get a list of indexes that contained good signal like this:

goodIndexes = signal > 0.1;

Then do whatever you want with the good signal elements. That code is "an algorithm that will pick up the peak", meaning good, valid signal. For example, you could zero out the bad ones and then use the whole signal.

signal(~goodIndexes) = 0;

Or you could extract just the good ones and take their mean,

meanGoodSignal = mean(signal(goodIndexes));

or whatever you want to do. Again goodIndexes indicates what elements of the signal are good parts of the peak and which are noise or not part of the peak.