Fundamental question about extracting a portion of repeating data over a large data set.

So I am going to do my best to explain this... but I have a sensor that is pusling on and off at a certain time interval Pt. After each pulse the signal decays for a period of time. I would like to extract and average the final 5 seconds for each pulse. The data is in the format of a 2 set of vectors. One being the signal, and the secod being time. I am a electrochemist, and dont have much formal training with programming, so must of the last week has been simply reading these forums trying to get a basic understanding. A brief example of the data format is below:

Signal Time

100 1

50 2

25 3

12.5 4

6.25 5

3.125 6

ect...

This goes for a few 100 seconds and i would like to only extract the average of the final 5 seconds. This then repeats for a few hundred cycles.

[David Probst]

Sorry to add, I want to extract the average of the final 5 seconds, and export that into a new array. Also the time does not repeat, it simply continues over each cycle.

[William Rose]

@David Probst, if you can add a comment in which you attach a sample file, using the paper clip icon, that could be helpful. The file can be just a text file with 2 columns of numbers and any separator. Column headers not required, but tolerated.

[William Rose]

Is it, like, 300 seconds of data, and each cycle is T seconds long (like 20 s), and you want the last 5 sec of each cycle? Are the cycles all the same duration, and you know the duration? Or each cycle is different, and you want get the last 5 sec before each new peak? Or do the cycles all have the same duration, but you do not know the duration?

[J. Alex Lee]

Is the cycle period fixed for each pulse, i.e., can you make a rule a priori about what time stamps correspond to the last 5 seconds of a pulse, or is it not fixed, i.e., do you additionally need to identify when each pulse ends? These would be different problems...

[David Probst]

Yes so cycles are same duration see the clip below for an of what the figure looks like. The X axis is time, and the Y is the sigal, it decays and i want to try and extract the average of the last 5 seconds for each "spike" persea. Yes i know the duration for each, and they are identical everytime (unless my system has some error but i have yet to run into that).

I appreciate the support.

[David Probst]

Each pulse is 60 seocnds, with a scan rate of 0.1sec.

[David Probst]

Here is another exmaple of the raw data, but in seconds as the x axis. I mentioned the time inbetween each pulse, and the time of the pusle is known and does not change over the entire test.

[William Rose]

@David Probst,

The first plot appears to have minutes on the horizontal axis, since one cycle has duration ~= 1. But actually the plotted duraiton is just a hair longer than 1. The second plot appears to have "sample number", not seconds, on the horizontal axis, since one cycle has duration ~=600.

I rquested a text file with numbers, but these plots are informative. I still would like a text file, to be more elpful to you.

The baseline, but not th top of the spikes, is drifting down over time. Do you really want to average the last 5 seconds of ALL the cycles, when the last 5 sec of cycle 1 is very different from the last 5 sec of the final cycle?

Maybe that is the problem you want to address: you want to find out how the basline is drifting down with time?

[David Probst]

@William Rose,

I dont need to see the drift, I would like the average of each individual spikes or decyas last 5 seocnds. Then to take those points export all those individually into a new array against time. Here is an exmaple of a text file, sorry for the lack of description.

I attahced a text file, and the hope (ideally) would be to have a new text file of the last 5 second avverage for each decay curve.

[David Probst]

I guess another way to say this would be the "drift" but its not a drift but a change in analyte concentration over time for a sensor. So we want to quantify this by averaging the "end point" of each decay. I normally do this manually on excel but we want to run this for extneded periods and once the files get to large this obvisouly becomes very very challneing to not crash excel haha.

Answer 1

[William Rose]

@David Probst,

This version reads data from the file you posted.

The columns are time, sensor reading.

The first data point is at time t=3600 (seconds?) (i.e. 1 hour).

dt=0.100000 for segment 1, so I assume dt=0.100000 for all segments. But there is a large time gap between segments.

The first eight segments have 601 samples, so I assume every segment has 601 samples.

data=importdata('OCPSen1.txt');
N=length(data.data);        %number of data points
t=data.data(:,1);           %time values (unevenly sampled)
t=t-t(1);                   %remove the initial time offset
y=data.data(:,2);           %sensor readings
t1=t(t<61);                 %times in segment 1 (times<61)
dt=(t1(end)-t1(1))/(length(t1)-1);      %seg.1 sampling interval
%fprintf('Segment 1: dt=%.6f sec',dt);  %display seg.1 sampling interval
Tdur=max(t);                %total duration
Npc=601;                    %number of point per cycle
Nc=floor(N/Npc);            %number of complete cycles
tr=reshape(t(1:Npc*Nc),[Npc,Nc]);       %reshape t into columns for segments
yr=reshape(y(1:Npc*Nc),[Npc,Nc]);       %reshape y into columns for segments
tend=tr(end-round(5/dt)+1:end,:);       %times: last 5 s of each segment
yend=yr(end-round(5/dt)+1:end,:);       %y values: last 5 s of each segment
tendmean=mean(tend);                    %mean time of ends
yendmean=mean(yend);                    %mean y value of ends

%% Plot results
subplot(211), plot(t,y,'-b')                %plot all the data
hold on; plot(tendmean,yendmean,'r*')       %plot means of ends of segments
grid on; xlabel('Time (s)'); ylabel('Y')
subplot(234), plot(tr(:,1),yr(:,1),'b.')    %plot data segment 1
hold on; plot(tendmean(1),yendmean(1),'r*') %plot mean of end of seg.1
grid on; xlabel('T'); title('Segment 1')
subplot(235), plot(tr(:,2),yr(:,2),'b.')    %plot data segment 2
hold on; plot(tendmean(2),yendmean(2),'r*') %plot mean of end of seg.2
grid on; xlabel('T'); title('Segment 2')
subplot(236), plot(tr(:,Nc),yr(:,Nc),'b.')  %plot last data segment
hold on; plot(tendmean(Nc),yendmean(Nc),'r*')   %plot mean of end of last seg.
grid on; xlabel('T'); title('Last Segment')

Try it.

[William Rose]

@David Probst, you can write the end values to a text file with

tendmean=[1:116];           %example data; use data from previous code 
yendmean=10*tendmean;       %example data; use data from previous code
writematrix([tendmean',yendmean'],'meansOfEnds.txt')

I use prime after tendmean to transpose the row vector into a column vector. Same for yendmean. Therefore the file will have 2 columns and many rows.

Good luck.

[David Probst]

Thank you for the support! Once im bak in lab ill try this out. I may shoot you some questions to help me understadn this if thats okay!

I appreiate the support fomr this communitiy!

[David Probst]

@William Rose

Thank you for this help, i think i understand the general process. The key peice i was missing was the "floor" function, whenever i tried using the reshape i was being told that the arrays were uneven whihc i think was due to having not even cycles exaclty near the end. the floor isolates this to apply for only complete cycles which solves the last array being uneven.

[William Rose]

@David Probst, you're welcome. You may email with followup quesitons by clicking on my WR icon in one of these message. You will notice that the pop-up which appears has a clickable envelope in the top right. One can choose in the profile whther to accept secure email. The envelope is visible for users who say yes.

You are right that floor() is useful for rounding down. Once I had Nc from floor(), I used it to reshape only the first Nc*Npc elements of y(). When I tried to reshape all of y(), reshape() had an error, due to the leftover bits.

Answer 2

[William Rose]

@David Probst

[Edit: fix typo in my comments: change "No" to "Now". Code is OK, I think.]

I will make some data, assuming Pt is constant, and assuming the cycles start at time 0:

dt=0.5; Tmax=200;   %adjust as needed
Pt=20.9;            %adjust as needed
t=0:dt:Tmax;        %elapsed time
tcycle=mod(t,Pt);   %time after each reset
tau=4;              %decay time
y=exp(-tcycle/tau)+randn(size(t))/5;  %decay, with noise

Let's plot the data, versus time and versus cycle time.

subplot(211); plot(t,y,'-b.')
grid on; xlabel('Elapsed Time')
subplot(212); plot(tcycle,y,'b.')
grid on; xlabel('Cycle Time')

Now let's average all the values that occur in the last 5 seconds of each cycle.

y5mean=mean(y(tcycle>=Pt-5))

y5mean = -0.0027

The line above uses the indexing ability of Matlab. It finds the mean of all the y values whose tcycle value is >= Pt-5.

[David Probst]

okay, so im trying to ensure I understand this correct (again i aplogize if im way off, im a grad student in electrochem and just started getting into this this week for some work).

The number of cycles is t, the total time divided by the pulse time (pt). you are taking this and usnig the y5mean to find the mean of the function y (the decay curve) when the final Pt -5 being the 20.9-5 , so 15.9 through the end of the pulse?

Is this correct in how im interpretting this?

Answer 3

[William Rose]

@David Probst,

Adjusting my answer in light of your comments and in light of the figures you posted.

In the second figure you posted, the spikes at the end of the recording have slightly higher tops and lower baseline than at the start. The simulated data below reproduces those features.

dt=0.1; Tmax=1000;  %sampling interval, total duration
Pt=60;              %cycle time
t=0:dt:Tmax;        %elapsed time
tau=20;             %decay time
y=(0.28+t/2e4).*exp(-tcycle/tau)-t/3e4-.2;  %decay, with noise

Plot the data, versus time.

plot(t,y,'-b')
grid on; xlabel('Time (s)')

Make an array with a separate column for each cycle. Discard any partial cycle at the end.

Nc=floor(Tmax/Pt);                  %number of complete cycles
Npc=Pt/dt;                          %number of points per cycle
yr=reshape(y(1:Nc*Npc),[Npc,Nc]);   %reshape the data into columns for cycles
yend=yr(end-round(5/dt)+1:end,:);   %array with last 5 seconds of each column

Now we can find the mean of the last 5 seconds of data, for every cycle.

yendmn=mean(yend);      %vector: mean of each column
tendmn=(1:Nc)*Pt-2.5;   %vector: mid-time of each mean

Plot the mean values.

hold on;
plot(tendmn,yendmn,'r+');

In the code above, I find the mean of the last 5 seconds separately for every cycle, since I assume you want to understand and perhaps compensate for the baseline drift which is evident in the second figure you posted.