Error while querying timetable

Hello All

I receive the error below when trying to run the code:

Error using table (line 233)
All table variables must have the same number of rows.

Error in Lookingfordrops (line 35)
    dropInfo = table(repmat(currentDate, numel(row), 1), columnsToCheck(col)', row, difference(sub2ind(size(difference), row, col)), ...       

The main idea is to iteriate through the days in my table and look for drops in value equal or more than 2400 day after day for a specific cell. In my code I am also focusing on only one column at a time. So in the code below, for example, column 3.

Can anyone advise on what's causing the issue? The code is still a work in progress as I haven't figured out how to store the 'drops' in a new table yet. I'm also attaching the table newTable.

columnsToCheck = (3); 

dropsTable = timetable();

uniqueDates = unique(newTable.Time);

% Loop through each row in newTable
for i = 2:length(uniqueDates)
    currentDate = uniqueDates(i);
    previousDate = uniqueDates(i - 1);
    
    % Extract data for the current and previous dates
    currentData = newTable(newTable.Time == currentDate, columnsToCheck);
    previousData = newTable(newTable.Time == previousDate, columnsToCheck);
    
    % Convert timetable data to table data
    currentTable = table2array(currentData);
    previousTable = table2array(previousData);
    
    % Calculate the difference between the dates
    difference = currentTable - previousTable;
    
    % Drops
    drops = abs(difference) > 2400;
    
    % Find rows with drops
    [row, col] = find(drops);
    
    % Create a table with information about drops
    dropInfo = table(repmat(currentDate, numel(row), 1), columnsToCheck(col)', row, difference(sub2ind(size(difference), row, col)), ...
        'VariableNames', {'Date', 'Column', 'Row', 'DropValue'});
    
    % Append the drop information to the dropsTable
    dropsTable = [dropsTable; dropInfo];
end

Happy New Year and thanks for your time

[Dyuman Joshi]

For making a table, every data set needs to have the same amount of rows, which is not the case.

To solve the issue, you will have to make sure that each array (of whatever data type) used to make the table has same amount of rows.

Note - The error occurs while defining the table dropInfo.

[OcDrive]

Hi Dyuman

It's not my dataset and I got data for more than a 100 years to go through. How would you go about making sure that's the case?

After exporting the dataset to MATLAB I have limited it to only 4 rows per day. Then I assumed that it would automatically generate a timetable of equal rows per day.

Thanks

Answer 1

[Hassaan]

@OcDrive Try this out and let me know if it resolves your issue.

% Define the column(s) to check for drops
columnsToCheck = 3; 

% Initialize an empty timetable to store information about drops
dropsTable = table();

% Extract unique dates from your timetable
uniqueDates = unique(newTable.Time);

% Loop through each unique date, starting from the second date
for i = 2:length(uniqueDates)
    currentDate = uniqueDates(i);
    previousDate = uniqueDates(i - 1);
    
    % Extract data for the current and previous dates
    currentData = newTable(newTable.Time == currentDate, columnsToCheck);
    previousData = newTable(newTable.Time == previousDate, columnsToCheck);
    
    % Convert timetable data to table data
    currentTable = table2array(currentData);
    previousTable = table2array(previousData);
    
    % Calculate the difference between the dates
    difference = currentTable - previousTable;
    
    % Identify where the absolute drops are greater than 2400
    drops = abs(difference) > 2400;
    
    % Find the rows and columns where drops occur
    [row, col] = find(drops);
    
    % Check if any drops are found
    if ~isempty(row)
        % Ensure that each component has the same number of elements
        dateRepeated = repmat(currentDate, numel(row), 1);
        colRepeated = repmat(columnsToCheck, numel(row), 1);
        dropValues = difference(sub2ind(size(difference), row, col));

        % Check if all components have the same number of elements
        if all([numel(dateRepeated), numel(colRepeated), numel(row), numel(dropValues)] == numel(row))
            % Create a table with information about drops
            dropInfo = table(dateRepeated, colRepeated, row, dropValues, ...
                'VariableNames', {'Date', 'Column', 'RowIndex', 'DropValue'});

            % Append the drop information to the dropsTable
            dropsTable = [dropsTable; dropInfo];
        else
            % Display an error message if the lengths don't match
            disp('Error: Variable lengths do not match');
        end
    else
        % Handle the case where there are no drops
        disp(['No drops found on ', datestr(currentDate)]);
    end
end

% Display the final dropsTable
disp(dropsTable);

In this revised code:

• The dropsTable is initialized correctly.
• Added some sanity checks

This should resolve the errors and provide you with the expected results. If dropsTable remains empty, verify the conditions in your data and ensure there are indeed drops greater than 240 to be detected.

Output

In this updated script:

• colRepeated is now set to repeat the entire columnsToCheck array for the number of rows found. This assumes that you are interested in the same column(s) for each drop.
• An additional check ensures all the variables (dateRepeated, colRepeated, row, and dropValues) have the same number of elements before trying to create the dropInfo table. If they don't match, it prints an error message.

Before running the script, ensure that newTable is correctly defined in your workspace and contains the data you want to analyze. Also, adjust the columnsToCheck variable as necessary to target the specific columns you're interested in.

Note

Please review and adapt the code according to the specific details of your system and the reaction mechanism.

------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

Professional Interests

• Technical Services and Consulting
• Embedded Systems
• Electrical and Electronics Engineering

[OcDrive]

Thanks Muhammad

When I run it:

Error using table (line 338)
Duplicate dimension and variable name: 'Row'.

Error in Lookingfordedrops (line 44)
        dropInfo = table(dropTimes, col, row, dropValues, ...

When I change the name...:

        Error using Lookingfordedrops (line 51)
All tables being vertically concatenated must have the same variable names.

[Hassaan]

@OcDrive I have updated the code again see and if it runs as on my system the output is generated successfully see the ouput picture. Thank you.

------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

Professional Interests

• Technical Services and Consulting
• Embedded Systems
• Electrical and Electronics Engineering

[Hassaan]

@OcDrive My output generated as follows on the code I shared above and used your asdf.mat file.

------------------------------------------------------------------------------------------------------------------------------------------------

If you find the solution helpful and it resolves your issue, it would be greatly appreciated if you could accept the answer. Also, leaving an upvote and a comment are also wonderful ways to provide feedback.

Professional Interests

• Technical Services and Consulting
• Embedded Systems
• Electrical and Electronics Engineering

[OcDrive]

That's great thank you Muhammad

Answer 2

[Sonesson]

Hello OcDrive,

There are a couple of issues ahead of you as far as I can see.

Your first problem, which causes the error, is mismatching dimensions when you are creating the table dropInfo. For your first iteration everything in dropinfo is a 0x1 vector except columnsToCheck(col)' which is 1x0. To fix this simply remove the " ' ", transposing it (or add another if you are feeling adventurous).

Second issue is that the table() function can not have entry which shares the name of a variable, thus "Row" (table() is not case sensitive in this regard) will not be a valid name, either change the table entry name, or the variable name.

Third issue is when appending the data into the dropsTable. You predefine dropsTable as a 0x0 table and try to append a 0x4 table which will again give mismatching dimensions. To fix this, simply define it as a 0x4 from the get go, or add an if statement to handle the first iteration specially.

Happy new Year!

[OcDrive]

Hi Sam

Thanks for your answer.

It worked, but the returned dropsTable table is empty 0x0 and when I tried to run it again I got another error:

Error using timetable (line 365)
Specify a row times vector, time step, or sample rate when preallocating a timetable to a specified size.

Error in Lookingfordedrops (line 10)
dropsTable = timetable('Size', [0, 4], 'VariableNames', variableNames, 'VariableTypes', variableTypes);

and that's my updated code

% Specify the column
columnsToCheck = (3); 

% Initialize a new timetable to store the identified drops
variableNames = {'60N', '55N', '50N', '45N'};
variableTypes = {'int8', 'int8', 'int8', 'int8'};

dropsTable = timetable('Size', [0, 4], 'VariableNames', variableNames, 'VariableTypes', variableTypes);

% Generate unique dates from newTable
uniqueDates = unique(newTable.Time);

% Loop through each row in newTable
for i = 2:length(uniqueDates)
    currentDate = uniqueDates(i);
    previousDate = uniqueDates(i - 1);
    
    % Extract data for the current and previous dates
    currentData = newTable(newTable.Time == currentDate, columnsToCheck);
    previousData = newTable(newTable.Time == previousDate, columnsToCheck);
    
    % Convert timetable data to table for subtraction
    currentTable = table2array(currentData);
    previousTable = table2array(previousData);
    
    % Calculate the difference between current and previous dates
    difference = currentTable - previousTable;
    
    % Identify drops greater than 24 in absolute value
    drops = abs(difference) > 240;
    
    % Find rows with drops
    [row, col] = find(drops);
    
    % Create a table with information about drops
    dropInfo = table(repmat(currentDate, numel(row), 1), columnsToCheck(col), row, difference(sub2ind(size(difference), row, col)), ...
        'VariableNames', {'Date', 'Column', 'Roww', 'DropValue'});
    
    % Append the drop information to the dropsTable
    dropsTable = [dropsTable; dropInfo];
end