More effective code than loop that takes a lot of time

I have a table T with over 300.000 rows and 39 columns.

I also have a table T2 with 8.800 rows, which is a subtable of T:

 T2 = T(T.b > 1,:);

Now I have written a loop, shown below, which does exactly what I want, but takes a lot of time.

T2.z = zeros(height(T2),1);

for i = 1:height(T2)
    var1 = ismember(T3(:,[24 30]), T2(i,[24 30])) & T3.b == 1;
    var2 = T3.a(var1);
    if isempty(var2)
    else
        var3 = T3.c(var1);
        var4 = T3.d(var1);
        if var2 ~= -1
            T2.z(i) = datenum(var3 + var2);
        else
            T2.z(i) = datenum(var3 + var4);
        end
    end
end

T2.z = datetime(T2.z, 'ConvertFrom', 'datenum');
TTrials.Trialshort(TTrials.Trialshort == datetime('31-Dec--0001 00:00:00')) = NaT;

I know loops aren't the most effective ways in MATLAB, but I don't know how to do this different. Can someone help?

What I want to do in words:

I have a table with data. Each row has a unique combination of 3 columns (say c1, c2 and b (I called it b in the code)), b contains the trialnumber.

T2 is a table with only the records with trial number 2 or higher. Now I want to add a column is this table where z is the addition of 2 other columns in T (a date and a number (amount of days)), of the record in T with the same values for c1 and c2, but where b = 1.

[Geoff Hayes]

Aletta - what does a lot of time mean? Does the processing take seconds or minutes or more? The calls to ismember might be causing a performance hit. Try profiling your code with profile to see where the bottleneck might be.

[Aletta Wilbrink]

I'm waiting for over 30 minutes now ans it's still not done. But I'll try profiling it

[Geoff Hayes]

Note also how you can do

 T.a(ismember(T(:,[24 30]), T2(i,[24 30])) & T.b == 1

up to three times: in the if condition, then in the second if condition, and then in the first assignment. Just do this once and store the value to a local variable that you can re-use.

[Aletta Wilbrink]

That does seem to make the process go a lot faster, thank you!

[Aletta Wilbrink]

This is what the profiler gave me

[Steven Lord]

Would it be possible for you to explain in words (not code) what you're trying to do? It's possible that we can find a more effective way to achieve your goal if we focus on what you're trying to do rather than how this particular implementation tries to achieve that goal.

[Aletta Wilbrink]

I have a table with data. Each row has a unique combination of 3 columns (say c1, c2 and b (I called it b in the code)), b contains the trialnumber.

T2 is a table with only the records with trial number 2 or higher. Now I want to add a column is this table where z is the addition of 2 other columns in T (a date and a number (amount of days)), of the record in T with the same values for c1 and c2, but where b = 1.

I hope this is clear to you.

Answer 1

[Prajit T R]

Hey Aletta

Upon reading your question I get the impression that what you're trying to do is a typical database operation. You may want to check out Database toolbox If you still want to use for loop, you will be better off using the parfor loop provided you have access to the Parallel Computing toolbox. Parallel Computing toolbox

Cheers