data loop for matlab

for i= 1977:2012
    for j=1:12
        for k=1:31
          mx=min(q1,[],'omitnan');
        end
    end
end

[Adam Danz]

Assuming you have a vector of datetime values that correspond to the data in q1, you don't need and shouldn't use a loop. In this case, a simple index of datetime values that are after 1976 and before 2013 can be used. For example,

idx = year(datetimeValues)>=1977 & year(datetimeValues)<=2012;
max(q1(idx))

[zakary Surprenant]

I forgot to mention that I want to find the max values for each month!

[Adam Danz]

How are your data arranged? Could you show us a sample?

[zakary Surprenant]

Answer 1

[Adam Danz]

1. Read data in as a table (see readtable)
2. Convert the year-month-day columns to a single datetime vector and convert the table to a timetable
3. Use retime to get the monthly max

Demo:

% Create demo table
T = table([1977;1977;1977;1977;1978;1978],[11;11;12;12;1;1],[1;2;1;2;1;2],(2:7)','VariableNames',{'Year','Month','Day','MaxTemp'})

T = 6x4 table
    Year    Month    Day    MaxTemp
    ____    _____    ___    _______

    1977     11       1        2   
    1977     11       2        3   
    1977     12       1        4   
    1977     12       2        5   
    1978      1       1        6   
    1978      1       2        7   

% Convert to datetime values
TT = timetable(datetime(T.Year,T.Month,T.Day),T.MaxTemp,'VariableNames',{'MaxTemp'})

TT = 6x1 timetable
       Time        MaxTemp
    ___________    _______

    01-Nov-1977       2   
    02-Nov-1977       3   
    01-Dec-1977       4   
    02-Dec-1977       5   
    01-Jan-1978       6   
    02-Jan-1978       7   

% Get monthly max
TT_MonthlyMax = retime(TT,'Monthly','max')

TT_MonthlyMax = 3x1 timetable
       Time        MaxTemp
    ___________    _______

    01-Nov-1977       3   
    01-Dec-1977       5   
    01-Jan-1978       7   

If any month is missing from the data, the MaxTemp for that month will be NaN. Those rows can be removed with,

TT_MonthlyMax(isnan(TT_MonthlyMax.MaxTemp),:) = [];

[Peter Perkins]

+1 to timetable and retime. rmmissing is another option for removing those rows from the result.

[Adam Danz]

Thanks, Peter Perkins. I didn't know about rmmissing, quite useful.

I wonder why it doesn't recognize empty numeric-cells while it does recognize empty char-cells.

T = table([num2cell([1;2]);{[]};num2cell([4;5])],[1;nan;3;4;5],{'A';'B';'C';'';'E'})

T = 5x3 table
        Var1        Var2       Var3   
    ____________    ____    __________

    {[       1]}      1     {'A'     }
    {[       2]}    NaN     {'B'     }
    {0×0 double}      3     {'C'     }
    {[       4]}      4     {0×0 char}
    {[       5]}      5     {'E'     }

rmmissing(T)

ans = 3x3 table
        Var1        Var2    Var3 
    ____________    ____    _____

    {[       1]}     1      {'A'}
    {0×0 double}     3      {'C'}
    {[       5]}     5      {'E'}

The missing function doesn't cover all classes and I suspect that this has somethign to do with it. For example,

T = table((1:3)', ["A";"B";"C"], num2cell((1:3)')) 

T = 3x3 table
    Var1    Var2    Var3 
    ____    ____    _____

     1      "A"     {[1]}
     2      "B"     {[2]}
     3      "C"     {[3]}

T(end+1,:) = {missing}

The following error occurred converting from missing to cell:
Conversion to cell from missing is not possible.

I put in a feature request back in May to expland the functionality of missing to cover these classes and it will be very useful if that gets implemented.

[zakary Surprenant]

would there also be a way to divide them by months also ? like have all my feb, oct etc together ? like they all have their own sets of data like max temp, min temp etc

[Adam Danz]

Most of the time it's a better idea to keep the data together and just use indexing when you want to refer to specific rows.

If you really want to split up the data you could write a loop that extracts all rows for each month and stores each month as an independent table within a cell array.

[zakary Surprenant]

If i have like 450 rows to adress which one is the best to use ? and also right now i have a table for each like mean,max,min because when i tried putting it together it wouldn't work.

[Adam Danz]

> If i have like 450 rows to adress which one is the best to use

Indexing. It's as easy as this:

T = table([1977;1977;1977;1977;1978;1978],[11;11;12;12;1;1],[1;2;1;2;1;2],(2:7)','VariableNames',{'Year','Month','Day','MaxTemp'});
TT = timetable(datetime(T.Year,T.Month,T.Day),T.MaxTemp,'VariableNames',{'MaxTemp'})

TT = 6x1 timetable
       Time        MaxTemp
    ___________    _______

    01-Nov-1977       2   
    02-Nov-1977       3   
    01-Dec-1977       4   
    02-Dec-1977       5   
    01-Jan-1978       6   
    02-Jan-1978       7   

% Create anonymous function:     TOUT = monthYearIdx(Timetable, Month, Year) 
monthYearIdx = @(TT,M,Y)TT(TT.Time>=datetime(Y,M,1) & TT.Time<=dateshift(datetime(Y,M,1),'end','month'),:); 

% Return table for all dates in December 1977
monthYearIdx(TT, 12, 1977)

ans = 2x1 timetable
       Time        MaxTemp
    ___________    _______

    01-Dec-1977       4   
    02-Dec-1977       5   

% Return table for all dates in January 1978
monthYearIdx(TT, 1, 1978)

ans = 2x1 timetable
       Time        MaxTemp
    ___________    _______

    01-Jan-1978       6   
    02-Jan-1978       7   

> right now i have a table for each like mean,max,min because when i tried putting it together it wouldn't work.

If you're using different methods for retime you can combine the tables using synchronize and then rename the variables.

summaryTT = synchronize(retime(TT,'Monthly','max'),retime(TT,'Monthly','min'),retime(TT,'Monthly','mean'));
summaryTT.Properties.VariableNames = strcat('MaxTemp_',{'max','min','mean'})

summaryTT = 3x3 timetable
       Time        MaxTemp_max    MaxTemp_min    MaxTemp_mean
    ___________    ___________    ___________    ____________

    01-Nov-1977         3              2             2.5     
    01-Dec-1977         5              4             4.5     
    01-Jan-1978         7              6             6.5     

[zakary Surprenant]

I tried both of what you sent me, and the second one you sent me, the data that it gives me is not accurate like the max, min,and mean are not the right one.

but if i look into my other tables like mean,max,min i can see that i have values that are better depending if its the max,min or mean.

[Adam Danz]

Can you attach the table in a mat file?

[Adam Danz]

I see several files each with several variables. Make it very very easy for me and share the code I need to run to load the right files and access the right table(s) to produce the output you are having problems with.

[Adam Danz]

Where did the mat files go?

I can help more if I can do these two steps without any other steps:

1. Load mat file
2. copy-paste the code as-is without chaning anything

This is called a minimal working example.

[zakary Surprenant]

I was able to fix it, the only thing that i am not able and i dont know why is when i want to loop all the data, i get my data but it suppresses my data from the other one before. For example when I loop from 1977:2012 I will only get a table that is 1x1 which is 1-jan-2012 25degrees, I will see my other data points in my command window but they wont go into my table! It has to do with my loop for sure,

this is my loop I have

monthYearIdx = @(TT_MonthlyMax,M,Y)TT_MonthlyMax(TT_MonthlyMax.Time>=datetime(Y,M,1) & TT_MonthlyMax.Time<=dateshift(datetime(Y,M,1),'end','month'),:);

for i=1977:2012

monthYearIdx(TT_MonthlyMax, 1,i)

Thanks for helping really appreciate it!

[Adam Danz]

It looks like you're not saving the outputs anywhere.

Do you plan on saving them in a cell array or as rows in table or some of method?

[zakary Surprenant]

i am trying to save them in rows in a table

[Adam Danz]

I'm completely lost.

Your original question was how to find the max values for each month.

My answer showed you how to convert your table into a timetable and use retime to achive that.

You then asked how to separate the timetable by months. I suggested you use indexing instead and I shared with you the anonymous function monthYearIdx which clearly returns another table with several rows.

Now you're asking how to loop through years and save something (???) into rows. I'm lost and it's really hard to follow the thread and what you're trying to do.

[Adam Danz]

zakary Surprenant's answer moved here as a comment

Hi Adam, its just I had another question because i've been trying to loop for three hours and I thought maybe I should asked before i make another question! Ill make another post about it, thank you!

[Adam Danz]

Ok, sounds like a good idea.

[Peter Perkins]

Adam, the answer to "I wonder why it doesn't recognize empty numeric-cells while it does recognize empty char-cells." is that those are not missing. You have this cell array:

{1 2 [] 4 5}

That's not a double vector with one value missing, it's a cell array of double arrays, four of which happen to be 1x1 and one of which is 0x0. In the context of an array of double arrays, [] is a perfectly valid value. It's not missing, it's just empty. It does however use the convention that for cell arrays of char row vectors ("cellstrs"), '' is considered "missing", because there's no other way to represent missing char data. For that reason (among many others), use string for text data.

The more pragmatic answer is that rmmissing, ismissing and so on don't support cell arrays other than cellstrs.

[Adam Danz]

Thanks for the explanation, Peter Perkins.

I appreciate that there's a difference between missing and empty values where missing numeric values are indicated with NaNs while empty numeric values are indicated with a 0x0 double array. An example that makes this distinction clear to other readers is data from a detector that samples at a fixed rate and unexpectedly skips a cycle (missing, NaN) vs a detector that works perfectly well but does not pick up on a signal (empty array). The missing and empty distinction is a necessity.

But there are two remaining problems IMO.

First is the inconsistency between this distinction in other data types. An empty cellstr is represented by {0x0 char} whereas a missing cellstr could be represented by {''} or perhaps {NaC} (Not-a-char, when the entire char array is 'missing') which would be consistent with cell arrays of doubles (NaN) or datetimes (NaT), etc. Outside of cell arrays, empty values aren't supported at all in double or char vectors:

x = [1 2 [] 4]

x = 1×3

1 2 4

●

s = ['a' 'b' '' 'd']

s = 'abd'

Second, it's not uncommon to treat empty values the same as missing values. For example, preallocation often uses cell(n,m) which creates a cell array of 0x0 doubles and some of those cells may remain empty due to missing data within the loop. It would be helpful if there were an option in the ismissing/rmmissing/fillmissing family of functions to include empties as missing which is currently the case only for cellstr data, to my knowledge. For example, F=fillmissing(A,method,emptyflag) similar to 'nanflag' in mean(),sum(),etc.

The problem is particularly common (and annoying) in tabular data with mixed classes.

[Steven Lord]

An empty cellstr is by {0x0 char} whereas a missing cellstr could be represented by {''} or perhaps {NaC} (Not-a-char, when the entire char array is 'missing') which would be consistent with cell arrays of doubles (NaN) or datetimes (NaT), etc.

Instead of trying to find some way to represent missing data in a cellstr, I probably would store my text data (including missing) in a string array and use the string array instead of a cellstr.

A = ["apple"; "banana"; "cherry"; missing; "eggplant"]

A = 5×1 string array
    "apple"
    "banana"
    "cherry"
    <missing>
    "eggplant"

[Adam Danz]

Thanks, Steven Lord, strings do solve that problem and they distinguish between empties and missing.

A = ["apple"; ""; "cherry"; missing; "eggplant"]

A = 5×1 string array
    "apple"
    ""
    "cherry"
    <missing>
    "eggplant"

But the bigger picture I'd like to emphasize is the option to treat empties as missing in the fillmissing(), rmmissing() etc family of functions. The current workaround is to find the empty arrays and replace them with missing values and for most data types, the missing() function comes in handy to fill with <missing> but missing() does not support cell arrays. So I'm suggesting that the _missing() family of functions optionally treat empty arrays as missing and to do this for empty cells as well.