Help with findgroups/splitapply, and general deep learning help.

Hi All,

I am trying to use findgroups, and splitapply to organise a table into a cellarray in a certain way, but I am getting some issues. One of the issues is that the number of groups returned isn't what I am expecting, i.e. the number of uniques in group 1, and the number of uniques in group 2. The file is attached. What the end goal is, is having a cellarray, with the trade dates going down, and the EFAs going across. When I combine, just originally, it isn't dividable by either the trade dates, or the number of unique EFA. The code I used originally is this:

G=findgroups( dateshift(FCTable.tradeDate,'start','day'), categorical(FCTable.EFA));
xAll = splitapply(@(x) {x} , FCTable{:,[2:7,10:13]}  ,  G);

The end goal is for use in Deep Learning, as I originally have a model that does predictions for sequence to label classification for trade dates, however, I want to have the individual blocks included as well. So if there's any hints regarding this as well - please let me know. I'm assuming I will have to combine LSTM's, but I'm not sure. i.e. for tomorrow, there will be a prediction made for each EFA (time period basically) of the trade date.

Answer 1

[Voss]

FCTable = load('FCTable.mat').FCTable

FCTable = 17590x14 table
     EFA     Variable1    Variable2    Variable3    Variable4    Variable5    Variable6    Variable7    Variable8    Variable9    Variable10    Variable11    Variable12    tradeDate 
    _____    _________    _________    _________    _________    _________    _________    _________    _________    _________    __________    __________    __________    __________

    {'1'}      2572         2149         2996         4305         2604         3118            55            0        1255         67.705         31.71        20.106      03/02/2024
    {'1'}      2507         2068         2947         4186         2523         3036            55            0        1196         66.822         31.71        19.139      03/02/2024
    {'1'}      2438         2003         2872         4093         2460         2956         52.11            0        1088         66.263        25.384        18.124      03/02/2024
    {'1'}      2363         1955         2770         4033         2386         2880         52.11            0        1125         65.468        25.384        17.051      03/02/2024
    {'1'}      2298         1904         2691         3997         2290         2809         52.71            0        1149         64.822        20.722        16.309      03/02/2024
    {'1'}      2243         1850         2635         3945         2236         2762         52.71            0        1190         64.671        20.722        15.919      03/02/2024
    {'2'}      2201         1809         2592         3876         2188         2694         50.71            0        1134         63.675        15.052        15.274      03/02/2024
    {'2'}      2172         1781         2563         3823         2112         2555         50.71            0        1115         63.525        15.052        14.386      03/02/2024
    {'2'}      2112         1687         2536         3796         2062         2417         54.44         5.35         976         62.454        17.032        13.381      03/02/2024
    {'2'}      2019         1527         2511         3798         1992         2298         54.44         6.53         847         62.719        17.032        12.239      03/02/2024
    {'2'}      1920         1377         2463         3826         1916         2188         51.42         3.11         750         62.608        23.416        11.156      03/02/2024
    {'2'}      1815         1239         2392         3873         1876         2108         51.42        29.97         605         62.999        23.416        10.121      03/02/2024
    {'2'}      1737         1109         2365         3942         1869         2027            76        29.86         404         63.427        36.313        9.2818      03/02/2024
    {'2'}      1686          989         2383         4065         1827         1956            76       128.25         294         62.802        36.313        8.6376      03/02/2024
    {'3'}      1657          925         2389         4231         1777         1897         83.33       151.66         181         59.111        52.666        8.2306      03/02/2024
    {'3'}      1650          917         2382         4377         1760         1873         83.33       125.57          72         57.294        52.666        7.9755      03/02/2024

T = FCTable;

T.tradeDate_shift = dateshift(T.tradeDate,'start','day');
T.EFA_categorical = categorical(T.EFA);

old_var_name = "Variable"+[1:6,9:12];
new_var_name = "Data";

T = mergevars(T,old_var_name,'NewVariableName',new_var_name);

T = removevars(T,"Variable"+[7,8])

T = 17590x5 table
     EFA                                                  Data                                                 tradeDate     tradeDate_shift    EFA_categorical
    _____    ______________________________________________________________________________________________    __________    _______________    _______________

    {'1'}    2572      2149      2996      4305      2604      3118      1255    67.705     31.71    20.106    03/02/2024      03/02/2024              1       
    {'1'}    2507      2068      2947      4186      2523      3036      1196    66.822     31.71    19.139    03/02/2024      03/02/2024              1       
    {'1'}    2438      2003      2872      4093      2460      2956      1088    66.263    25.384    18.124    03/02/2024      03/02/2024              1       
    {'1'}    2363      1955      2770      4033      2386      2880      1125    65.468    25.384    17.051    03/02/2024      03/02/2024              1       
    {'1'}    2298      1904      2691      3997      2290      2809      1149    64.822    20.722    16.309    03/02/2024      03/02/2024              1       
    {'1'}    2243      1850      2635      3945      2236      2762      1190    64.671    20.722    15.919    03/02/2024      03/02/2024              1       
    {'2'}    2201      1809      2592      3876      2188      2694      1134    63.675    15.052    15.274    03/02/2024      03/02/2024              2       
    {'2'}    2172      1781      2563      3823      2112      2555      1115    63.525    15.052    14.386    03/02/2024      03/02/2024              2       
    {'2'}    2112      1687      2536      3796      2062      2417       976    62.454    17.032    13.381    03/02/2024      03/02/2024              2       
    {'2'}    2019      1527      2511      3798      1992      2298       847    62.719    17.032    12.239    03/02/2024      03/02/2024              2       
    {'2'}    1920      1377      2463      3826      1916      2188       750    62.608    23.416    11.156    03/02/2024      03/02/2024              2       
    {'2'}    1815      1239      2392      3873      1876      2108       605    62.999    23.416    10.121    03/02/2024      03/02/2024              2       
    {'2'}    1737      1109      2365      3942      1869      2027       404    63.427    36.313    9.2818    03/02/2024      03/02/2024              2       
    {'2'}    1686       989      2383      4065      1827      1956       294    62.802    36.313    8.6376    03/02/2024      03/02/2024              2       
    {'3'}    1657       925      2389      4231      1777      1897       181    59.111    52.666    8.2306    03/02/2024      03/02/2024              3       
    {'3'}    1650       917      2382      4377      1760      1873        72    57.294    52.666    7.9755    03/02/2024      03/02/2024              3       

TG = groupsummary(T,{'EFA_categorical','tradeDate_shift'},@(x){x}, ...
    new_var_name,'IncludeEmptyGroups',true);

TG = renamevars(TG,"fun1_"+new_var_name,new_var_name)

TG = 2202x4 table
    EFA_categorical    tradeDate_shift    GroupCount        Data     
    _______________    _______________    __________    _____________

           1             03/02/2024           6         {6x10 double}
           1             04/02/2024           8         {8x10 double}
           1             05/02/2024           8         {8x10 double}
           1             06/02/2024           8         {8x10 double}
           1             07/02/2024           8         {8x10 double}
           1             08/02/2024           8         {8x10 double}
           1             09/02/2024           8         {8x10 double}
           1             10/02/2024           8         {8x10 double}
           1             11/02/2024           8         {8x10 double}
           1             12/02/2024           8         {8x10 double}
           1             13/02/2024           8         {8x10 double}
           1             14/02/2024           8         {8x10 double}
           1             15/02/2024           8         {8x10 double}
           1             16/02/2024           8         {8x10 double}
           1             17/02/2024           8         {8x10 double}
           1             18/02/2024           8         {8x10 double}

ND = numel(unique(T.tradeDate_shift));
NE = numel(unique(T.EFA_categorical));
assert(height(TG) == ND*NE)

% your cell array
C = reshape(TG.Data,ND,NE)

C = 367x6 cell array
    {6x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}
    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}    {8x10 double}

[James McBrearty]

Hi Voss,

This is excellent. I did change some things around on my end, but I never knew the functionality of groupsummary. I have it working now in a couple of parallel LSTM's now, and it's looking like it's doing a great job.

Many Thanks, James

[Voss]

You're welcome!