How do I remove leap year data?
Show older comments
The code works. However, I need to run eliminate all leap year data (February 29th) because March 1st and February 29th are getting mixed up in the outgoing text file. How can I do this? I have tried using braces around idxMn,2 and etc but received an error stating it's unable to use them. I also can't use leapyear for I do not have aerospace toolbox. Any help will be grateful.
close all;
clear all;
clc;
Datafiles = fileDatastore("temp_summary*.txt","ReadFcn",@readMonth,"UniformRead",true);
dataAll = readall(Datafiles)
dataAll.Year = year(dataAll.Day);
dataAll.Month = month(dataAll.Day);
dataAll.DD = day(dataAll.Day)
% Unstack variables
minT_tbl = unstack(dataAll,"MinT","Year","GroupingVariables", ["Month","DD"],"VariableNamingRule","preserve")
maxT_tbl = unstack(dataAll,"MaxT","Year","GroupingVariables", ["Month","DD"],"VariableNamingRule","preserve")
yrs =str2double(minT_tbl.Properties.VariableNames(3:end))';
% find min
[Tmin,idxMn] = min(minT_tbl{:,3:end},[],2,'omitnan');
Tmin_yr = yrs(eomday(idxMn,2),[]);
% find max
[Tmax,idxMx] = max(maxT_tbl{:,3:end},[],2,'omitnan');
Tmax_yr = yrs(eomday(idxMx,2),[]);
% find low high
[lowTMax,idxMx] = min(maxT_tbl{:,3:end},[],2,'omitnan');
LowTMax_yr = yrs(eomday(idxMx,2),[]);
% find high low
[highlowTMn,idxMn] = max(minT_tbl{:,3:end},[],2,'omitnan');
HighLowT_yr = yrs(eomday(idxMn,2),[]);
% find avg high
AvgTMx = round(mean(table2array(maxT_tbl(:,3:end)),2,'omitnan'));
% find avg low
AvgTMn = round(mean(table2array(minT_tbl(:,3:end)),2,'omitnan'));
% Results
tempTbl = [maxT_tbl(:,["Month","DD"]), table(Tmax,Tmax_yr,AvgTMx,lowTMax,LowTMax_yr,Tmin,Tmin_yr,AvgTMn,highlowTMn,HighLowT_yr)]
tempTbl2 = splitvars(tempTbl)
FID = fopen('Meda 05 Temperature Climatology.txt','w');
report_date = datetime('now','format','yyyy-MM-dd HH:MM');
fprintf(FID,'Meda 05 Temperature Climatology at %s \n', report_date);
fprintf(FID,"Month DD Temp Max (°F) Tmax_yr AvgTMax (°F) lowTMax (°F) LowTMax_yr TempMin (°F) TMin_yr AvgTMin (°F) HighlowTMin (°F) HighlowT_yr \n");
fprintf(FID,'%3d %6d %7d %14d %11d %11d %15d %11d %13d %10d %13d %17d \n', tempTbl2{:,1:end}');
fclose(FID);
winopen('Meda 05 Temperature Climatology.txt')
function Tbl = readMonth(filename)
opts = detectImportOptions(filename)
opts.ConsecutiveDelimitersRule = 'join';
opts.MissingRule = 'omitvar';
opts = setvartype(opts,'double');
opts.VariableNames = ["Day","MaxT","MinT","AvgT"];
Tbl = readtable(filename,opts);
Tbl = standardizeMissing(Tbl,{999,'N/A'},"DataVariables",{'MaxT','MinT','AvgT'})
Tbl = standardizeMissing(Tbl,{-99,'N/A'},"DataVariables",{'MaxT','MinT','AvgT'})
[~,basename] = fileparts(filename);
nameparts = regexp(basename, '\.', 'split');
dateparts = regexp(nameparts{end}, '_','split');
year_str = dateparts{end}
d = str2double(extract(filename,digitsPattern));
Tbl.Day = datetime(d(3),d(2),Tbl.Day)
end
1 Comment
This question is getting spread across several questions, so bringing my comments here.
You need to check if the month ends in 29 before you convert Day into a datetime. That conversion happens in readMonth. Since 2/29 gets convereted to 3/1 in non-leap year years, you need to identify this date before the converstion to datetime.
Normally this wouldn't be an issue, but for some reason your files contain data for Feb 29 even in non-leap year years.
dataAll = readtable('temp_summary.05.02_1981.txt',...
"ConsecutiveDelimitersRule","join","ReadVariableNames",false,...
"Delimiter",{' ','\t','*','#'},"LeadingDelimitersRule",'ignore',...
'EmptyLineRule','skip');
dataAll.Properties.VariableNames = {'Day' 'MaxT' 'MinT' 'AvgT'};
dataAll.Day = datetime(1981,2,dataAll.Day); % 1981, which is a non-leap year
dataAll.Month = month(dataAll.Day);
dataAll.DD = day(dataAll.Day);
tail(dataAll)
% Remove all Feb 29 dates from the table
LY = (dataAll.Month(:)== 2 & dataAll.DD(:) == 29);
dataAll(LY,:) = [ ];
tail(dataAll)
As you can see, the current LY code did not remove the data. This is because 2/29 is converted to 3/1 by datetime. You need to perform your LY correction before you convert to datetime.
dataAll2 = readtable('temp_summary.05.02_1981.txt',...
"ConsecutiveDelimitersRule","join","ReadVariableNames",false,...
"Delimiter",{' ','\t','*','#'},"LeadingDelimitersRule",'ignore',...
'EmptyLineRule','skip');
dataAll2.Properties.VariableNames = {'Day' 'MaxT' 'MinT' 'AvgT'};
tail(dataAll2)
% If the last date of the imported month is 29, delete
if dataAll2.Day(end==29)
dataAll2(end,:)=[];
end
% Now you can convert Day to datetime
dataAll2.Day = datetime(1981,2,dataAll2.Day); % 1981, which is a non-leap year
dataAll2.Month = month(dataAll2.Day);
dataAll2.DD = day(dataAll2.Day);
tail(dataAll2)
Now there is no 3/1 data in the Feb 1981 table
Since you are using a datastore to load you files, you must make this edit in your read function.
Accepted Answer
I ran your code with the provided data (here, online), however I do not see any leap year days.
That aside, something like this could work —
DateTime = datetime(2024,02,25) + caldays(0:11).'
DateTime(month(DateTime) == 2 & day(DateTime) == 29) = []
This year happens to be a leap year, however this approach is not sensitive to the year, since it just searches for and eliminates every February 29 entry.
.
16 Comments
Jonathon Klepatzki
on 5 Feb 2024
I had serious problems understanding the data your code produces.
Perhaps this instead:
dataAll(dataAll.Month == 2 & dataAll.Day == 29)=[];
If you have already isolated the month numbers and day numbers in other variables, using the month and day functions as I do in my example is probably not going to add anything, and in this instance, throws errors.
Testing that variation here, and changing my code to match your data —
dataAll = datetime(2024,02,25) + caldays(0:11).'
dataAll.Month = month(dataAll);
dataAll.Day = day(dataAll);
dataAll(dataAll.Month == 2 & dataAll.Day == 29)=[]
That seems to work, although it differs from your posted code, so consider checking that. (I removed some of the semicolons to display the results. Add them back in your code.)
.
Jonathon Klepatzki
on 5 Feb 2024
Star Strider
on 5 Feb 2024
I’m running R2023b, however that shouldn’t make a difference with datetime arrays, since I don’t believe there were any significant changes from R2020a to R2023b releases.
What error are you getting? Please copy the entire error message (all the red text from your Command Window) and paste it to a new Comment or an edited previous Comment here.
Jonathon Klepatzki
on 5 Feb 2024
That does not make sense to me, because the code extracts numbers (the month number and the day number) and then compares them. to double constants (2 and 29 respectively). There is no comparison between datetime variables and double variables.
One problem is that none of those years are leap years, if I am parsing the file names correctly, so datetime is assigning the 29-Feb as 1-Mar. The datetime function checks for that and assigns the dates appropriately to the months. (This is one of its strengths, in my opinion.)
If I assign the year as this year (a leap year), then my code works —
files = dir('*.txt');
for k = 1:numel(files)
filename = files(k).name
T{k} = readtable(files(k).name);
nrows = size(T{k},1);
Nr = regexp(filename,'\d*','match');
Check = eomday(cellfun(@str2double,Nr(3)),2) % Check 'filename' Year For Leap Year
Nr{3} = '2024'; % Artificially Assign Leap Year To Test Code
Yr = cellfun(@str2double,repmat(Nr(3),nrows,1));
Mo = cellfun(@str2double,repmat(Nr(2),nrows,1));
T{k}.DateTime = datetime(Yr,Mo,T{k}{:,1});
end
T{:}
for k = 1:numel(T) % Loop To Remove 29-Feb
Lv = (month(T{k}.DateTime) == 2) & (day(T{k}.DateTime) == 29);
T{k}(Lv,:) = [];
end
T{:}
The problem may be that the years in the file names are wrong. There is no way I can correct for that.
.
Jonathon Klepatzki
on 5 Feb 2024
Jonathon Klepatzki
on 5 Feb 2024
Star Strider
on 5 Feb 2024
As always, my pleasure!
Jonathon Klepatzki
on 7 Feb 2024
‘Any suggestions?’
Yes. It is difficult for me to follow what you are doing. See if you can incorporate elements of my code here into yours.
This works —
files = dir('*.txt');
for k = 1:numel(files)
filename = files(k).name
T{k} = readtable(filename);
fn{k} = filename;
nrows = size(T{k},1);
Nr = regexp(filename,'\d*','match');
% Check = eomday(cellfun(@str2double,Nr(3)),2) % Check 'filename' Year For Leap Year
% Nr{3} = '2024'; % Artificially Assign Leap Year To Test Code
Yr = cellfun(@str2double,repmat(Nr(3),nrows,1));
Mo = cellfun(@str2double,repmat(Nr(2),nrows,1));
T{k}.DateTime = datetime(Yr,Mo,T{k}{:,1});
end
T{:}
for k = 1:numel(T) % Loop To Remove 29-Feb
Lv = (month(T{k}.DateTime) == 2) & (day(T{k}.DateTime) == 29);
T{k}(Lv,:) = [];
if nnz(Lv) % Optional 'if' Block
fprintf('Feb 29 removed from %s\n', fn{k})
end
end
T{:}
Alternatively, it may be necessary for you to use my code example here to pre-process your data. Then save the resulting tables (each with a slightly different name so you will not overwrite the original files) and use them instead. That is the easiest approach I can suggest.
It is likely necessary to create a datetime variable in order to do this. (The first for loop in my code does this. It is likely straightforward for you to adapt it to use in your code.) That may seem superfluous, however using the datetime function to create a relevant date array actually makes this easier.
.
Jonathon Klepatzki
on 7 Feb 2024
We may need to be something different, then, if all Februaries have a 29th. That is not normal, and datetime will only work correctly for Februaries in leap years in that instance.
This version creates a new ‘YMD’ variable to replace the datetime call, and then scans ‘YMD’ for all instances of ‘Feb 29’ and removes them, leaving intact all the other months. I used all of your uploaded files to test this.
Try this —
files = dir('*.txt');
for k = 1:numel(files)
filename = files(k).name
T{k,:} = readtable(filename);
fn{k,:} = filename;
nrows = size(T{k},1);
Nr = regexp(filename,'\d*','match');
% Check = eomday(cellfun(@str2double,Nr(3)),2) % Check 'filename' Year For Leap Year
% Nr{3} = '2024'; % Artificially Assign Leap Year To Test Code
Yr = cellfun(@str2double,repmat(Nr(3),nrows,1));
Mo = cellfun(@str2double,repmat(Nr(2),nrows,1));
% T{k}.DateTime = datetime(Yr,Mo,T{k}{:,1});
T{k}.YMD = [Yr Mo T{k}{:,1}]; % Create Variable To Replace ''datetime'
end
T{:}
for k = 1:numel(T) % Loop To Remove 29-Feb
Lv = (T{k}.YMD(:,2) == 2) & (T{k}.YMD(:,3) == 29);
T{k}(Lv,:) = [];
if nnz(Lv) % Optional 'if' Block
fprintf('Feb 29 removed from %s\n', fn{k})
end
end
T{:}
I would still suggest using this as a preprocessing step, saving all the processed files by slightly different names from the originals, and then working with them instead of the originals.
.
Jonathon Klepatzki
on 8 Feb 2024
Star Strider
on 8 Feb 2024
As always, my pleasure!
I saw your other question and decided to see if I could incorporate an answer to it in my code.
Without changing my original code too much, I was able to recover the variable names and elimiinate the '*' and '#' characters. I also changed it to delete 29 Feb from only the non-leap years.
The code —
files = dir('*.txt');
for k = 1:numel(files)
filename = files(k).name
Tx = fileread(files(k).name);
VN = regexp(Tx(1,:), '(\s\s)|(\s\s\s)', 'split');
VN = cellfun(@strtrim,VN(2:5), 'Unif',0);
opts = detectImportOptions(files(1).name);
opts = setvartype(opts,{'Var2','Var3','Var4'},'char');
T{k,:} = readtable(filename, opts);
LLv1 = any(ismember(T{k}{:,4},{'*'}),2);
T{k}{LLv1,4} = {num2str(T{k}{LLv1,5},'%.1f')};
LLv2 = any(ismember(T{k}{:,3},{'#'}),2);
T{k}{LLv2,3:4} = [T{k}{LLv2,4},{num2str(T{k}{LLv2,5},'%.1f')}];
T{k} = T{k}(:,1:end-1);
T{k}.Properties.VariableNames = VN;
T{k}{:,2:4} = cellfun(@str2double,T{k}{:,2:4}, 'Unif',0);
Te = array2table([T{k}{:,1} cell2mat(T{k}{:,2:4})], 'VariableNames',VN);
T{k} = Te;
% T{k}
fn{k,:} = filename;
nrows = size(T{k},1);
Nr = regexp(filename,'\d*','match');
% Check = eomday(cellfun(@str2double,Nr(3)),2) % Check 'filename' Year For Leap Year
% Nr{3} = '2024'; % Artificially Assign Leap Year To Test Code
Yr = cellfun(@str2double,repmat(Nr(3),nrows,1));
Mo = cellfun(@str2double,repmat(Nr(2),nrows,1));
% T{k}.DateTime = datetime(Yr,Mo,T{k}{:,1});
T{k}.YMD = [Yr Mo T{k}{:,1}]; % Create Variable To Replace ''datetime'
end
T{:}
for k = 1:numel(T) % Loop To Remove 29-Feb
if eomday(T{k}.YMD(:,1), 2) ~= 29
Lv = (T{k}.YMD(:,2) == 2) & (T{k}.YMD(:,3) == 29);
T{k}(Lv,:) = [];
if nnz(Lv) % Optional 'if' Block
fprintf('Feb 29 removed from %s\n', fn{k})
end
end
end
T{:}
I just did this for the challenge! This should now produce the desired results.
.
More Answers (0)
Categories
Find more on Dates and Time in Help Center and File Exchange
on 5 Feb 2024
on 8 Feb 2024
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
