Use retime to sum over a timeinterval, but skip time intervals where data doesn't exist

Hello,

I have been using retime sum some of my data in 6 hour chunks. I have multiple locations [locationGroups] that each have data, and so I use a for loop to retime each location separtely.

DataSum= [];
for ggg = 1:height(locationGroups)
    %Extract data from the TT that is just in this location
    station = char(locationGroups.Station(ggg));
    locData=OriginalDataTT(OriginalDataTT.station == station,:);
    %Retime data
    DataSum.(station) = retime(locData(:,'Data'),'regular','sum','timestep',hours(6));
end

This works very well, except that retime inserts a zero where there is no data present in the original time table.

For example, in the OriginalDataTT, the data could look like:

Time Data On Dep station

'08-Sep-2020 23:56:00' 16 1 DepA 2200

'08-Sep-2020 23:57:00' 457 1 DepA 2200

'08-Sep-2020 23:58:00' 86 1 DepA 2200

'08-Sep-2020 23:59:00' 1 1 DepA 2200

'22-Sep-2020 00:00:00' 106 1 DepC 2200

'22-Sep-2020 00:01:00' 0 1 DepC 2200

'22-Sep-2020 00:02:00' 270 1 DepC 2200

Between deployments A and C, there is gap between the 8th and 22nd of Sept. This is intentional, as there was no data collected between 9th and 21st of Sept. When I retime this data, to sum all values in the 'Data' column every 6 hours, it generates a line for every 6 hour period, even if that time period wasn't included in the OriginalDataTT. It lists the sum as 0. I can not use this, as 0 is a valid count when the station is active. I can't filter out 0s because I risk removing actual data.

Is there an easy way to do this? Can I tell retime to skip time intervals that have no data, or to at least enter NaN rather than 0? I thought by removing those time intervals in the original TT would be enough, but that doesn't seem to be the case. Do I have to add another nest to my loop to only sum each deployment (Dep) individually?

Cheers!

Answer 1

[Cris LaPierre]

To me, it sounds like you may want to use groupsummary instead of retime. Retime is doing exactly what it was designed to do - convert unevenly sampled data into evenly sampled data. That means filling gaps and spaces in the original data.

Groupsummary, on the other hand, can still be used to group your data, but by default it ignores empty groups. You can use the name-value pairs 'IncludeMissingGroups' (default is True) and 'IncludeEmptyGroups' (default is False) to tell it how to handle these situations.

For your data, I think the following would work (untested).

grpdData = groupsummary(locData,'Time',hours(6),'sum','Data');

[Emily T. Griffiths]

Fantastic! That worked exactly as I needed it to. Thank you very much!