Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Flat Table class to 2D array is SUPER Slow

Subject: Flat Table class to 2D array is SUPER Slow

From: Heady

Date: 7 Jan, 2014 15:59:07

Message: 1 of 6

There must be a better way than:

    T = [];
    for i=1:max(t.a)
        for j=1:max(t.b)
            temp = t.c(t.a==1 & t.b==j);
            if ~isempty(temp)
                T(i,j) = temp;
            end;
        end;
    end;

Subject: Flat Table class to 2D array is SUPER Slow

From: Heady

Date: 7 Jan, 2014 16:56:17

Message: 2 of 6

This takes 168 seconds for a 156x1080 table with all the time spent in table.subsref because of the dot notation, I guess.

Even rewriting it to avoid the table is a painful 86 seconds and defeats the entire purpose of using a table.

Even so, it should less than a minute and a half, shouldn't it?

K>> tic;
    T = nan.*ones(max(t.a),max(t.b));
    for i=1:max(t.a)
        for j=1:max(t.b)
            temp = t.c(t.a==i & t.b==j);
            if ~isempty(temp)
                T(i,j) = temp;
            end;
        end;
    end;
    toc;

Elapsed time is 168.390597 seconds.

K>> tic
    profile on
    mA = t.a;
    mB = t.b;
    mC = t.c;
    T = nan.*ones(max(mA),max(mB));
    for i=1:max(mA)
        I = mA==i;
        for j=1:max(mB)
            temp = mC(I & mB==j);
            if ~isempty(temp)
                T(i,j) = temp;
            end;
        end;
    end;
    toc

Elapsed time is 86.318056 seconds.

Subject: Flat Table class to 2D array is SUPER Slow

From: Peter Perkins

Date: 14 Feb, 2014 17:40:27

Message: 3 of 6

On 1/7/2014 11:56 AM, Heady wrote:
> This takes 168 seconds for a 156x1080 table with all the time spent in
> table.subsref because of the dot notation, I guess.
>
> Even rewriting it to avoid the table is a painful 86 seconds and defeats
> the entire purpose of using a table.
>
> Even so, it should less than a minute and a half, shouldn't it?

You have not provided any information about what's actually in your
data, so it's pretty hard to give suggestions.

I thought at first that you were doing some kind of grouped
calculations. You're creating something like 73008 = 3*156^2 logical
arrays to do the subscripting, and having to make 24336 passes through
the data to do so. That's never gonna be as fast as it could be. One
suggestion might be to split up your data once, and then loop over the
split up data. There's an example in the reference page for accumarray
that show how to do this, entitled "Group Values in Cell Array". That
example is a bit of an advanced maneuver, and we're working on more
accessible ways to do it. One might be to use table's varfun method,
which can do grouped calculations.

But on closer inspection of your code, I noticed that line

    T(i,j) = temp;

is doing a scalar assignment, and so I'm going to guess that a and b are
just permutations of 1:156, or something like that. If that's correct,
then all you're doing is broadcasting c out to an array using a and b as
indices.

This is exactly what accumarray is for.

 >> t = table(randperm(100)',randperm(100)',randn(100,1), ...
                             'VariableNames',{'a' 'b' 'c'});
 >> tic, T = accumarray({t.a t.b},t.c,[100 100]); toc
Elapsed time is 0.000767 seconds.


Better?


>
> K>> tic;
> T = nan.*ones(max(t.a),max(t.b));
> for i=1:max(t.a)
> for j=1:max(t.b)
> temp = t.c(t.a==i & t.b==j);
> if ~isempty(temp)
> T(i,j) = temp;
> end;
> end;
> end;
> toc;
>
> Elapsed time is 168.390597 seconds.
>
> K>> tic
> profile on
> mA = t.a;
> mB = t.b;
> mC = t.c;
> T = nan.*ones(max(mA),max(mB));
> for i=1:max(mA)
> I = mA==i;
> for j=1:max(mB)
> temp = mC(I & mB==j);
> if ~isempty(temp)
> T(i,j) = temp;
> end;
> end;
> end;
> toc
>
> Elapsed time is 86.318056 seconds.

Subject: Flat Table class to 2D array is SUPER Slow

From: Eric Sampson

Date: 14 Feb, 2014 18:59:10

Message: 4 of 6

Peter Perkins <Peter.Remove.Perkins.This@mathworks.com> wrote in message <52FE550B.7080107@mathworks.com>...
> On 1/7/2014 11:56 AM, Heady wrote:

Peter, is there a way to use varfun to update a table 'in-place', ie without it automatically prepending something to the variable names? For instance with rowfun to do something like that I'm thinking that I could say (...,'OutputVariables',mytable.Properties.VariableNames) to prevent the renaming, but I don't see a way to to this with varfun other than caching the original variable names and then applying them to the output table?

Thanks

Subject: Flat Table class to 2D array is SUPER Slow

From: Peter Perkins

Date: 17 Feb, 2014 14:22:24

Message: 5 of 6

Eric, in the current release, you would have to assign to the Variable
names property of the output from varfun. I'll make a note to consider
your suggestion as an enhancement for the future.



On 2/14/2014 1:59 PM, Eric Sampson wrote:
> Peter Perkins <Peter.Remove.Perkins.This@mathworks.com> wrote in message
> <52FE550B.7080107@mathworks.com>...
>> On 1/7/2014 11:56 AM, Heady wrote:
>
> Peter, is there a way to use varfun to update a table 'in-place', ie
> without it automatically prepending something to the variable names? For
> instance with rowfun to do something like that I'm thinking that I could
> say (...,'OutputVariables',mytable.Properties.VariableNames) to prevent
> the renaming, but I don't see a way to to this with varfun other than
> caching the original variable names and then applying them to the output
> table?
>
> Thanks

Subject: Flat Table class to 2D array is SUPER Slow

From: Eric Sampson

Date: 18 Feb, 2014 16:33:13

Message: 6 of 6

Peter Perkins <Peter.Remove.Perkins.This@mathworks.com> wrote in message <ldt5uu$cao$1@newscl01ah.mathworks.com>...
> Eric, in the current release, you would have to assign to the Variable
> names property of the output from varfun. I'll make a note to consider
> your suggestion as an enhancement for the future.
>

Thanks as always Peter :)

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us