Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Find rising point of peak in noisy data

Subject: Find rising point of peak in noisy data

From: Steve Sachay

Date: 26 Jan, 2011 19:43:04

Message: 1 of 11

Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.

Subject: Find rising point of peak in noisy data

From: TideMan

Date: 26 Jan, 2011 21:49:50

Message: 2 of 11

On Jan 27, 8:43 am, "Steve Sachay" <core...@gmail.com> wrote:
> Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through.  Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.  

The human eye is a very accurate peak detector. It automatically
discards the noise and concentrates on the peaks behind the noise.
To emulate that with a computer algorithm is not always easy.
One way is to low pass filter to remove the noise first.
Then derivative-based solutions have a better chance of success.
I use orthogonal wavelet decomposition for low-pass filtering, but
there are other methods like Butterworth.

Subject: Find rising point of peak in noisy data

From: Paulo Silva

Date: 26 Jan, 2011 22:12:03

Message: 3 of 11

"Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.

Can you share the data with us or just part of it?

Subject: Find rising point of peak in noisy data

From: Steve Sachay

Date: 27 Jan, 2011 05:05:04

Message: 4 of 11

"Paulo Silva" wrote in message <ihq67j$joa$1@fred.mathworks.com>...
> "Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> > Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.
>
> Can you share the data with us or just part of it?

Sure. This is a 4124x2 matrix with seconds (starting at 1177) in column 1 and data in column 2. There are 'rises' in there, the first one is easy, and the next three are a bit trickier. I want to get the points where the 'rise' begins to be at 1257, 2155, 3953 and 4854s respectively.

http://www.filedropper.com/examplerisefind

Subject: Find rising point of peak in noisy data

From: Paulo Silva

Date: 27 Jan, 2011 06:53:03

Message: 5 of 11

"Steve Sachay" wrote in message <ihque0$gra$1@fred.mathworks.com>...
> "Paulo Silva" wrote in message <ihq67j$joa$1@fred.mathworks.com>...
> > "Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> > > Hi. I have some noisy data with peaks that I wish to find, however it's not the maximum of the peak I am after. I'd like to be able to determine where the peak begins to rise from the baseline. Of course it doesn't have to be perfect, a good estimation is fine. The peaks are fairly clear and relatively narrow when they do rise- I can pick them out easily by hand but I have a lot of data to go through. Simple derivative based solutions are not working due to the noise. Is there a function that could help? Thanks.
> >
> > Can you share the data with us or just part of it?
>
> Sure. This is a 4124x2 matrix with seconds (starting at 1177) in column 1 and data in column 2. There are 'rises' in there, the first one is easy, and the next three are a bit trickier. I want to get the points where the 'rise' begins to be at 1257, 2155, 3953 and 4854s respectively.
>
> http://www.filedropper.com/examplerisefind

I hope the values are close enough

function Points=findstartpeaks(examp)
windowSize = 50;
data=filter(ones(1,windowSize)/windowSize,5,examp(:,2));
md=max(data);
mm=find(data>=(0.20*md));
mfil=[mm(1)];
for a=2:numel(mm)
if((mm(a)-mm(a-1)) > 10)
mfil=[mfil mm(a)];
end
end
Points=examp(mfil,1);

Subject: Find rising point of peak in noisy data

From: Steve Sachay

Date: 27 Jan, 2011 16:33:04

Message: 6 of 11

"Paulo Silva" wrote in message <ihr4of$41u$1@fred.mathworks.com>...

> I hope the values are close enough
>
> function Points=findstartpeaks(examp)
> windowSize = 50;
> data=filter(ones(1,windowSize)/windowSize,5,examp(:,2));
> md=max(data);
> mm=find(data>=(0.20*md));
> mfil=[mm(1)];
> for a=2:numel(mm)
> if((mm(a)-mm(a-1)) > 10)
> mfil=[mfil mm(a)];
> end
> end
> Points=examp(mfil,1);

Thanks very much for the effort, but this seem to find values closer to the peaks than the beginning of the rise. I'll play with the parameters and see what I can do, but it was very helpful to see your code.

Subject: Find rising point of peak in noisy data

From: Paulo Silva

Date: 27 Jan, 2011 19:34:03

Message: 7 of 11

"Steve Sachay" wrote in message <ihptg8$h6g$1@fred.mathworks.com>...
> Of course it doesn't have to be perfect, a good estimation is fine.

let's see...

"Steve Sachay" wrote in message <ihs6o0$arn$1@fred.mathworks.com>...
 
> Thanks very much for the effort, but this seem to find values closer to the peaks than the beginning of the rise. I'll play with the parameters and see what I can do, but it was very helpful to see your code.

So this simple and free code isn't good enough for your needs, that's great, maybe you can do much better or just wait for another fool like me to give you a miracle solution, with the sample data my code detected the points with a maximum error of 22 points from your "eye" values but still not good enough, I must remember to stop wasting my time trying to help solving other peoples problems.

Subject: Find rising point of peak in noisy data

From: fburton@nyx.net (Francis Burton)

Date: 27 Jan, 2011 20:14:56

Message: 8 of 11

In article <ihs6o0$arn$1@fred.mathworks.com>,
Steve Sachay <coregis@gmail.com> wrote:
>Thanks very much for the effort, but this seem to find values closer to
>the peaks than the beginning of the rise. I'll play with the parameters
>and see what I can do, but it was very helpful to see your code.

If your example is truly representative of your data, my approach
would be to do the following:

1) find the global data maximum, at time tpeak
2) find the mean value and standard deviation (or range) of the
   baseline preceding this peak within a time window tpeak-d1..tpeak-d2
3) find the first point (if any) that occurs above this baseline mean
   plus some threshold derived from the amplitude of the baseline noise
   (the time of this point is what you are seeking)
4) blank off the data between tpeak-b1 and tpeak+b2
5) repeat steps 1-4 until all the data is blanked and/or no more
   'first points' are found

This algorithm/heuristic should be fairly straightforward to implement
in MATLAB code.

Francis

Subject: Find rising point of peak in noisy data

From: Steve Sachay

Date: 27 Jan, 2011 22:11:03

Message: 9 of 11

"Paulo Silva" wrote in message <ihshbb$m2o$1@fred.mathworks.com>...

> So this simple and free code isn't good enough for your needs, that's great, maybe you can do much better or just wait for another fool like me to give you a miracle solution, with the sample data my code detected the points with a maximum error of 22 points from your "eye" values but still not good enough, I must remember to stop wasting my time trying to help solving other peoples problems.

I am very sorry if you are offended, I meant no ill will. I did not ask anyone to write code for me, just make suggestions of how to proceed beyond my simple initial pass that was not working. My entire rise and fall lasts about 100 points, so within 22 points was not the resolution I am after. I'm sorry if that was unclear.

I have incorporated some of Francis' ideas and now I can obtain the beginning of the rise within 3 points, which works well for my application. Thank you all for your help.

Subject: Find rising point of peak in noisy data

From: fburton@nyx.net (Francis Burton)

Date: 28 Jan, 2011 15:18:24

Message: 10 of 11

In article <ihsqhn$m68$1@fred.mathworks.com>,
Steve Sachay <coregis@gmail.com> wrote:
>I have incorporated some of Francis' ideas and now I can obtain the
>beginning of the rise within 3 points, which works well for my
>application. Thank you all for your help.

Below is some code I wrote to implement some of the ideas I
outlined in my other post (actually simplified). Both data
and difference data are filtered by running medians of window
size 5 and 3 which doesn't smooth the data in the same way as
a linear filter would but has the useful effect of getting rid
of brief transients. It's a way of reducing noise without
introducing a significant 'lag' into the main transients.

The function takes two threshold arguments which may be tweaked
as necessary:

>> rt=FindPeakRiseTimes(examp, 10, 4)

rt =

        2158 4854 3952 1257

which is fairly close to what you required. The times are
listed in order of peak 'energy', but you can of course do a
sort(rt).

Note that I haven't tested this code at all other than to run
it on your data. Also, I am sure that some of it (e.g. the
cummag function) could be expressed much more concisely with
'vector' code.

Uncomment the plot code to watch the algorithm in action.

Hope this is useful.

function risetimes = FindPeakRiseTimes(data, pthresh, rthresh)
t = data(:,1); y = data(:,2); n = numel(y);
yf = medfilt1(y,5); yf = medfilt1(yf,3);
yd = diff(yf);
ydf = medfilt1(yd,5); ydf = medfilt1(ydf,3);
yc = cummag(ydf); % see below
npeak = 0;
risetimes = [];
[ypeak, ipeak] = max(yc);
while (ypeak > pthresh)
    i = ipeak; % found peak, now look for rise time
    while (i>=1) && (yc(i)>rthresh)
        i = i-1;
    end
    if i>0
        npeak = npeak+1;
        risetimes(npeak)=t(i); %#ok<AGROW>
    end
% plot(t(1:end-1), yc);
% hold on;
% plot(t(ipeak),yc(ipeak),'r+');
% plot(t(i),yc(i),'g+');
% hold off;
% pause;
    i = ipeak;
    while (i>=1) && (yc(i)>0) % blank before peak
        yc(i) = 0;
        i = i-1;
    end
    i=ipeak;
    while (i<=n) && (yc(i)>0) % blank after peak
        yc(i) = 0;
        i = i+1;
    end
    [ypeak, ipeak] = max(yc);
end

function yc = cummag(y)
yc = zeros(numel(y), 1);
s = 0;
for i = 1:numel(y)
    if y(i)>0
        s = s+y(i);
    else
        s = 0;
    end
    yc(i) = s;
end

Francis

Subject: Find rising point of peak in noisy data

From: Steve Sachay

Date: 28 Jan, 2011 16:31:19

Message: 11 of 11

fburton@nyx.net (Francis Burton) wrote in message <1296227906.852290@irys.nyx.net>...

> Below is some code I wrote to implement some of the ideas I
> outlined in my other post (actually simplified). Both data
> and difference data are filtered by running medians of window
> size 5 and 3 which doesn't smooth the data in the same way as
> a linear filter would but has the useful effect of getting rid
> of brief transients. It's a way of reducing noise without
> introducing a significant 'lag' into the main transients.
>
> The function takes two threshold arguments which may be tweaked
> as necessary:

Thank you again Franicis. The mean filter is a good idea for this, I did find that lowpass filtering was moving the peaks slightly so it was not ideal. I took a similar approach at attacking the derivative but your solution is more elegant (and my code much sloppier :) Thank you for posting this so others may find it in the future.

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us