Problem using retime for timetables

Hello,

i have a timetable of measured data with rowtimes every minute and i need to increase the resolution to seconds using linear interpolation. I use retime for that. The problem is,some points of the real data are not include in the interpolation. Why is this happening and how could i solve this problem? I attacht the real data below. Thanks a lot.

[Jiri Hajek]

Hi, you forgot to show us how you use the retime function on your data. It has several options which could explain the observed behaviour.

[Daniel]

Hi,

sorry yes i forgot that. The code i use is

NewTT = retime(TT,'regular','linear','Timestep',seconds(1));

[Jiri Hajek]

OK, this is sufficient to explain the observed behaviour. Your retiming scheme uses a regular timestep, which is generated first, then the function finds appropriate values for the new times by linear interpolation. The times of points on the interpolated data explain this nicely:

Now, this is not a problem per se, it could be quite acceptable for many purposes. If you see this behaviour as a problem, then you need to define how it should be done - and only then look for a way to achieve the desired behaviour. The retime function has several options, so perhaps the first step would be to test them to see, which would suit you best.

[Daniel]

Thanks! i have already tried other options. But any works. I need obligatory the real data to be inside the interpolation data

Answer 1

[dpb]

Your issue is the input time sampling isn't actually on precisely a one-minute interval; the recorded timestamps include fractional seconds. Taking your timetable and increasing the time resolution displayed, the beginning of the TT looks like

> TT.Time.Format='dd-MMM-uuuu HH:mm:ss.SS';
>> head(TT)
ans =
  8×1 timetable
             Time              Var1 
    _______________________    _____
    01-Jun-2022 05:37:11.25        0
    01-Jun-2022 05:38:10.78        0
    01-Jun-2022 05:39:10.31     3000
    01-Jun-2022 05:40:09.84     3000
    01-Jun-2022 05:41:09.37     3000
    01-Jun-2022 05:44:07.96    17000
    01-Jun-2022 05:45:07.49    17000
    01-Jun-2022 05:46:07.02    17000
>> 

The actual time interval looks like

>> seconds(diff(ans.Time))
ans =
   59.5310
   59.5300
   59.5300
   59.5310
  178.5890
   59.5300
   59.5310
>> 

so it's operating on almost a minute sampling rate but is just about a half-second fast with an occasional lapse. Down inside the file, there are places where are missing data for several days, even...

>> find(dt>1E5,1)
ans =
        3330
>> TT(ans-3:ans+3,:)
ans =
  7×1 timetable
             Time              Var1 
    _______________________    _____
    03-Jun-2022 18:33:58.42    18000
    03-Jun-2022 18:34:58.42        0
    03-Jun-2022 18:35:58.43        0
    03-Jun-2022 18:36:58.43        0
    08-Jun-2022 01:17:07.23        0
    08-Jun-2022 01:18:07.70        0
    08-Jun-2022 01:19:07.74        0
>> 

skips from late afternoon on June 3 to early hours of June 8. There is an even dozen of those events of at least

>> sum(dt>1E5)
ans =
    12
>> 

that long.

In order for your resultant timeseries to have every actual sampled point, you would need to duplicate the input time vector and then synchronize that with the regularly-sampled resampled series; that is probably not really what you need/want.

Looking at the timestamps; since the data collection runs asynchronously but is triggered on rougly the one minute frequency note that the series begins at roughly 11 seconds after the minute, but is moving by about 50 msec faster than the 1-minute sample rate so that eventually the sample time will cover the entire spectrum of from 0:59 seconds after the minute.

Consequently, to create an output time vector that contains the actual sampled values, you first need to adjust the sample times to the nearest minute, retaining the sampled data for that minute, and then resample that data to the one-second resolution...

TT=retime(TT.'minutely','nearest');         % adjust the sample times to minute; keep nearest reading
TT=retime(TT,'secondly','linear');          % now interpolate to seconds

You may want to do something different to handle the longer missing periods; the above will also end up filling in those with an interpolated value across those several-day missing periods; that may not be the best way to handle those periods but you'll know much better than we what would make sense.