Monthly Average for Large Dataset
5 views (last 30 days)
Show older comments
Hello I am looking to calculate the monthly mean, max and min of a very large dataset. The data set includes 8 readings every day for 15 years so the dataset is huge.
I then need to calculate the monthly mean, min and max for every January, February March etc month over the 15 year period. How would I do this? Each month obviously has a different number of days so how would I take this into consideration? Attached is a picture of the layout of the data to give you an idea of what the dataset looks like. The highlighted column is what I am looking at.
0 Comments
Answers (7)
Andrei Bobrov
on 29 Jul 2015
f = fopen('NG1_all_wav');
c = textscan(f,['%s',repmat(' %f',1,15)],'CollectOutput',1);
fclose(f);
[a,~,c1] = unique(c{2}(:,1:2),'rows');
c2 = accumarray(c1,c{2}(:,5),[],@(ii){[min(ii),max(ii),mean(ii)]});
out = [a,cat(1,c2{:})];
0 Comments
Azzi Abdelmalek
on 29 Jul 2015
If your text file looks like
2000 02 01 1001 25 25
2000 02 01 2310 45 46
2000 03 02 1121 33 36
2000 03 02 1141 33 36
2000 03 02 1151 33 36
2001 02 05 1452 47 85
2001 02 05 1442 470 805
2001 02 05 1422 407 8005
fid=fopen('file.txt')
out=textscan(fid,'%s')
fclose(fid)
a=reshape(out{:},6,[])'
b=str2double(a(:,1:2))
data=str2double(a(:,5:6))
[ii,jj,kk]=unique(b,'rows')
c=accumarray(kk,(1:numel(kk))',[],@(x) {mean(data(x,:))})
out=[a(jj,1:2) num2cell(cell2mat(c))]
1 Comment
Muhammad Usman Saleem
on 29 Jul 2015
Hint: make a function that read text file and calculate statistics.
0 Comments
Peter Perkins
on 30 Jul 2015
Hard to tell if that file is space- or tab-delimited, and if that's the top of the file, or what. It helps to provide more precise information, preferable a small example of your data file. So you may need to adjust the followng for another delimiter, or whatever. In any case, read your data into a table, and do a grouped calculation using varfun:
t = readtable('yourdata.txt','delimiter','\t','ReadVariableNames',false);
t.Properties.VariableNames(2:5) = {'Year' 'Month' 'Day' 'TimeOfDay'};
monthlyMeans = varfun(@mean,t,'GroupingVariables',{'Year' 'Month'},'InputVariables',6);
Hope this helps.
0 Comments
Walter Roberson
on 6 Aug 2015
dv = datevec(YourMatrix(:,1));
monidx = dv(:,1) * 12 + dv(:,2);
[unique_monidx, ~, rel_monidx] = unique(monid);
mean_by_month = accumarray(rel_monidx, YourMatrix(:,3), [], @mean);
uyear = floor(unique_monidx / 12);
umon = mod(unique_monidx, 12);
output_table = [uyear, umon, mean_by_month];
5 Comments
Walter Roberson
on 12 Aug 2015
Leave out the dv = datevec and use
monidx = YourMatrix(:,3); %if that is the column with the month number
Walter Roberson
on 12 Aug 2015
Wait, the dv = datevec(YourMatrix(:,1)); solution was in response to you saying that you had a matrix with the serial date number in the first column. Is that not the case?
Tiffany de klerk
on 8 Aug 2015
3 Comments
Azzi Abdelmalek
on 12 Aug 2015
Ok, then delete this answer, and post a comment by clicking on comment on this answer
See Also
Categories
Find more on Calendar in Help Center and File Exchange
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!