Calculate daily standard deviation from timetable
Show older comments
Hello, I want to calculate the mean max and mean min standard deviation of temperature data from a daily timetable.
How can I calculate the standard deviation for each day?
Accepted Answer
Try something like this —
DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)
figure
plot(TT1.DT, TT1.Temperature)
grid
DailyMax = retime(TT1, 'daily', 'max');
DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
TT2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})
The standard deviation of a single value is zero by definition, so it is only possible to calculate the standard deviation of all temperatures for a single day for that day.
It is possible to calculate the standard deviation for the maximum and minimum temperatures for several days (at least more than 1) and of course for the entire month.
.
27 Comments
Ancalagon8
on 21 Dec 2022
Thank you!
Yes!
Example —
DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)
figure
plot(TT1.DT, TT1.Temperature)
grid
DailyMax = retime(TT1, 'daily', 'max');
DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
T2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})
DaysMeetingThreshold = nnz(T2.Max >= 18) % Days Equal To Or Higher Than 18°C
Adjust the conditions to select either greater than (>) or greater than or equal to (>=) and the desired temperature threshold (here 18°C). I did not include that in the timetable because it is for the entire time period and so would have to have a column length equal to the entire timetable. This could also be done for calendar months or calendar weeks or for the entire table. That would require extra code and probably an additional timetable as well.
An example of doing that using retime wouild be:
Threshold_1 = retime(TT1, 'weekly', @(x)nnz(x>=18))
TT2 = table2timetable(T2);
Threshold2 = retime(TT2, 'weekly', @(x)nnz(x>=18))
The ‘Threshold1_1’ timetable counts the number of times each week any ‘Temperature’ equalled or exceeded 18°C.
The ‘Threshold_2’ timetable counts the number of times the maximum ‘Temperature’ equalled or exceeded 18°C.
It could be added to the existing timetable, however that would require additional code and that code would not be trivial, since the lengths of the constant values would have to match the entries of the corresponding dates in position and length.
.
Ancalagon8
on 23 Dec 2022
As always, my pleasure!
I am not certain that I understand what you want to do.
You would likely need a loop for the various thresholds. After that, putting them into a table is straightforward —
DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)
figure
plot(TT1.DT, TT1.Temperature)
grid
DailyMax = retime(TT1, 'daily', 'max');
DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
T2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})
Thresholdv = [16; 17; 18; 19; 20]; % Column Vector
for k = 1:numel(Thresholdv)
DaysMeetingThreshold(k,:) = nnz(T2.Max >= Thresholdv(k)); % Days Equal To Or Higher Than Thresholdv (°C)
end
ThresholdTable = table(Thresholdv, DaysMeetingThreshold)
.
Ancalagon8
on 23 Dec 2022
Edited: Ancalagon8
on 30 Dec 2022
Star Strider
on 23 Dec 2022
Edited: Star Strider
on 30 Dec 2022
As always, my pleasure!
EDIT — (30 Dec 2022 at 16:56)
Not easily.
DT = datetime('01-Dec-2022 00:00:00') + hours(0:720).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)
figure
plot(TT1.DT, TT1.Temperature)
grid
DailyMax = retime(TT1, 'daily', 'max');
HourlyMax = retime(TT1, 'hourly', 'max')
DailyMin = retime(TT1, 'daily', 'min');
DailySTD = retime(TT1, 'daily', @std);
T2 = table(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailySTD.Temperature, 'VariableNames',{'DT','Max','Min','StDv'})
Thresholdv = [16; 17; 18; 19; 20]; % Column Vector
for k = 1:numel(Thresholdv)
DaysMeetingThreshold(k,:) = nnz(T2.Max >= Thresholdv(k)); % Days Equal To Or Higher Than Thresholdv (°C)
end
ThresholdTable = table(Thresholdv, DaysMeetingThreshold)
[yh,mh,dh] = ymd(HourlyMax.DT); % Return Hour At Which Maximum Temperature Occurred
for k = 1:numel(DailyMax)
[yd,md,dd] = ymd(DailyMax.DT(k));
Lv = ismember([yh,mh,dh], [yd,md,dd], 'rows');
idx = HourlyMax.Temperature(Lv) == DailyMax.Temperature(k);
HM = HourlyMax(Lv,:);
DayHourMax(k,:) = timetable2table(HM(idx,:));
end
DayHourMax % Table Of Hour In Each Day At Which Maximum Temperature Occurred
This should be easier!
.
Ancalagon8
on 30 Dec 2022
Star Strider
on 30 Dec 2022
My pleasure!
My code worked when I ran it, or I would not have posted it.
I have no idea what the problem could be.
Ancalagon8
on 28 Jan 2023
Probably something like this —
MonthlyMax = retime(TT1, 'monthly', @(x)mean(max(x)));
MonthlyMin = retime(TT1, 'monthly', @(x)mean(min(x)));
although I am not certain what you want.
Testing that to see what it returns —
DT = datetime('01-Dec-2022 00:00:00') + caldays(1:364).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)
figure
plot(TT1.DT, TT1.Temperature)
grid
hold on
MonthlyMax = retime(TT1, 'monthly', @(x)mean(max(x)))
MonthlyMin = retime(TT1, 'monthly', @(x)mean(min(x)))
MthV = DT(1) + calmonths(0:11).';
hold on
plot(MthV, [MonthlyMax.Temperature, MonthlyMin.Temperature], '-r')
hold off
.
Ancalagon8
on 28 Jan 2023
Star Strider
on 28 Jan 2023
As always, my pleasure!
MonthlyMaxSD = retime(TT1, 'monthly', @(x)std(max(x)));
MonthlyMinSD = retime(TT1, 'monthly', @(x)std(min(x)));
That should work without further modification.
Ancalagon8
on 28 Jan 2023
I forgot about that. It’s taking the standard deviation of a scalar, and that’s zero by definition. The maximum and minimum of a standard deviation are going to be the same (since it’s also a scalar), and the standard deviation of the maxima or minima for the entire year is going to be scalars representing the entire year, plotting a straight line over all the months. so a different sort of metric may be necessary.
There may be other ways, for example the standard error of the mean for each month, presented as confidence intervals —
DT = datetime('01-Dec-2022 00:00:00') + caldays(1:364).';
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5;
TT1 = timetable(DT, Temperature)
figure
plot(TT1.DT, TT1.Temperature)
grid
hold on
MonthlyMax = retime(TT1, 'monthly', @(x)mean(x)+std(x)/sqrt(numel(x))*1.96)
MonthlyMin = retime(TT1, 'monthly', @(x)mean(x)-std(x)/sqrt(numel(x))*1.96)
MthV = DT(1) + days(15) + calmonths(0:11).';
hold on
plot(MthV, [MonthlyMax.Temperature, MonthlyMin.Temperature], '-r')
hold off
Other options could be calculating the monthly standard deviation of the maxima and minima for each day during the month. That would initially require one aggregation to calculate the daily maxima and minima, and a second aggretation to calculate the standard deviation of the monthly values of the daily data. That may be the only way to get a standard deviation aggregated over a month.
The aggregation depends on what you want, and what you want to do with the aggregated data after calculating it.
.
Ancalagon8
on 28 Jan 2023
Star Strider
on 28 Jan 2023
As always, my pleasure!
I would create ‘DailyMin’ and ‘DailyMax’ as additional temporary variables, probably in a separate timetable, and then aggregate over them as 'monthly' to create the monthly standard deviations. That may be the only way to do it.
Ancalagon8
on 28 Jan 2023
I was thinking of something like this —
DT = datetime('01-Dec-2022 00:00:00') + hours(1:364*24).'; % Hours
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5; % Temperatures
TT1 = timetable(DT, Temperature)
DailyMax = retime(TT1, 'daily', 'max') % Daily Aggregation
DailyMin = retime(TT1, 'daily', 'min') % Daily Aggregation
DailyTT = DailyMax;
DailyTT = addvars(DailyTT, DailyMin.Temperature)
MonthlyTT = retime(DailyTT, 'monthly', @(x)std(x)) % Monthly Aggregation Of Daily Data
MthV = DT(1) + calmonths(0:11).';
figure
hold on
plot(MthV, MonthlyTT{:,1}, '-r')
plot(MthV, MonthlyTT{:,2}, '-b')
hold off
legend('Max Monthly SD','Min Monthly SD', 'Location','best')
This plots the standard deviations, not the actual temperature values.
.
Ancalagon8
on 28 Jan 2023
Star Strider
on 28 Jan 2023
The ‘DailyTT’ timetable is defined initially as having the daily max temperatures. The daily min temperatures were added to it as a new variable, so now ‘DailyTT’ has both as different variables. It’s just not possible to change their variable names easily to reflect that.
Ancalagon8
on 28 Jan 2023
Edited: Ancalagon8
on 28 Jan 2023
Star Strider
on 28 Jan 2023
I’m not certain that I understand about ‘mean,min and max range for each month’. The mean for the month should be only one value, unless you intend the range of daily temperatures (max - min for each day) and the statistics on those values.
The coding depends on how you want to calculate the daily values and then aggregate them.
Ancalagon8
on 29 Jan 2023
Edited: Ancalagon8
on 29 Jan 2023
Star Strider
on 29 Jan 2023
Edited: Star Strider
on 29 Jan 2023
Thank you!
I just wanted to clarify my understanding of this.
EDIT — I’m kind of lost at this point. Isn’t that the result you’ree getting? That’s the result I got in the code in my earlier Comment. That dealt with the standard deviations, however calculating different parameters (min, max, and mean) should yield essentially the same sort of result. The only difference should be the last (function) argument in various aggregations:
MonthlyTT = retime(DailyTT, 'monthly', @(x)std(x)) % Monthly Aggregation Of Daily Data
to produce different results.
I’ll follow up on this tomorrow if there’s anything else to be coded.
.
Ancalagon8
on 30 Jan 2023
Edited: Ancalagon8
on 30 Jan 2023
Try something like this —
DT = datetime('01-Dec-2022 00:00:00') + hours(1:364*24).'; % Hours
Temperature = sin(2*pi*(0:numel(DT)-1)/24-0.5).'*12+5 + randn(size(DT))*1.5; % Temperatures
TT1 = timetable(DT, Temperature)
DailyMax = retime(TT1, 'daily', 'max'); % Daily Aggregation
DailyMin = retime(TT1, 'daily', 'min'); % Daily Aggregation
DailyMean = retime(TT1, 'daily', 'mean'); % Daily Aggregation
DailyRange = DailyMax.Temperature - DailyMin.Temperature; % Daily Aggregation
DailyTT = timetable(DailyMax.DT, DailyMax.Temperature, DailyMin.Temperature, DailyRange, DailyMean.Temperature, 'VariableNames',{'Max_Temperature','Min_Temperature','Range_Temperature','Mean_Temperature'})
MonthlyTTMax = retime(DailyTT, 'monthly', 'max')
MonthlyTTMin = retime(DailyTT, 'monthly', 'min')
MonthlyTTMean = retime(DailyTT, 'monthly', 'mean')
MonthlyTTSD = retime(DailyTT, 'monthly', @(x)std(x)) % Monthly Aggregation Of Daily Data
MthV = DT(1) + calmonths(0:11).';
figure
hold on
plot(MthV, MonthlyTTSD{:,1}, '-r')
plot(MthV, MonthlyTTSD{:,2}, '-b')
hold off
legend('Max Monthly SD','Min Monthly SD', 'Location','best')
The ‘MonthlyTT’ set of timetable arrays should have everything you need. If necessary, it should be straightforward to abstract the various variables from them and import them to a new summary timetable, similar to what I did to create ‘DailyTT’.
Creating numbered variables such as:
MeanDailyRange_January=mean(DailyRange_January);
is generally discouraged. You can get the various monthly values from the timetable arrays that calculated them.
I am doing my best to provide what you want, however that is not always clear to me. The result is that we need to iterate until we converge on the correct arrays.
.
Star Strider
on 30 Jan 2023
As always, my pleasure!
I thought we already covered that problem at some point. By now I’ve lost track of where it is (although searching for it, I believe it was here).
It would be essentially the same approach.
.
More Answers (0)
Categories
Find more on Dates and Time in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
