Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

2-way random effects ANOVA exceeded maximum variable size

Asked by Ryan on 17 Apr 2013

I am trying to run a 2-way random effects ANOVA on a set of data in MATLAB. This is the code I am using:

[pval table stats terms] = anovan(modelfit, {RATEES RATERS},...
 'model', 'interaction', 'random', 2, 'varnames', {'RATEE' 'RATER'})

However, doing this gives me the error:

??? Maximum variable size allowed by the program is exceeded.
Error in ==> termcross at 13
ab = a(:,acols) .* b(:,bcols);
Error in ==> anovan>maketerm at 1470
         tdum = termcross(termdum{k}, tdum);
Error in ==> anovan>makedummyterms at 1122
   [tdum,vars,tn,df0,tconstr] = maketerm(tm,isnested,varinfo,j,sindex,...
Error in ==> anovan at 199
terminfo = makedummyterms(sindex, termlist, varinfo, ...

If I modify my code to any one of the following, it will work without error, but it isn't giving me the data I need:

[pval table stats terms] = anovan(modelfit, {RATEES RATERS},...
 'random', 2, 'varnames',     {'RATEE' 'RATER'})

or:

[pval table stats terms] = anovan(modelfit, {RATEES RATERS})

It just doesn't work when I try to use the 'interaction' model. Why? Do I really just have too much data? The size of each of my input variable (modelfit, RATEES, RATERS) are all 14879 x 1. That is, 1 column arrays with 14879 values. I know this is a large array, but it seems to me MATLAB should be able to handle this?

If this is really too much, how do I go about getting the interaction effects I need?

In case it is relevant, I am using a 64-bit Mac OS X 10.6.8 (Snow Leopard) and MATLAB R2008a.

I also don't understand how to use the 'random' element of the anovan command. I have to put in an index, but it only lets me choose a single variable to be a random effect? I want both RATEES and RATERS to be random, but it only lets me choose one!

3 Comments

Tom Lane on 18 Apr 2013

Some more info would help. How many unique values in RATEES and RATERS? Do all combinations appear?

If there are N RATEES and M RATERS, there could be N*M combinations. An interaction model includes an effect for each combination.

Maybe you could show the first handful of rows of your data, and explain it a bit.

Ahmed A. Selman on 18 Apr 2013

Is it possible to clear some of your variables before entering the function? In 2006 or so I've faced such a (sad) case and thought to store the variables to some point in a file, clear unneeded (specially the large sized) variables, run the function causing the out of memory error, and store data in the same file, clear things up again and read needed variables to plot only. Even if it works, it's a very long story.. I know :)

Ryan on 18 Apr 2013

Re: Tom:

As I said in the original post, both RATEES and RATERS have 14879 unique values. What is the maximum variable size for MATLAB? is 14879*14879 too big? When I check my memory it says the maximum array size is 7.101e+008, which seems to me that 14879^2 is fine.

Do seeing the first rows really help? I can edit my post later to add that. The data are just the ratings provided by a bunch of different raters for a set of stimuli. I am trying to compute the reliability of the data, and the only way I know to do so is ICC, which necessitates a random-effects ANOVA.

Ryan

Products

No products are associated with this question.

1 Answer

Answer by Tom Lane on 18 Apr 2013

It wasn't clear from the original post that the 14879 values were all different.

I don't really need to see the actual data, only understand the general idea. It seems to me, maybe naively, that you have N=14879 rows, and you want to estimate an overall mean, main effects for N levels of raters, main effects for N levels of ratees, and interaction effects for N^2 combinations of them. That will require a design matrix of size N-by-(1+N+N+N^2), roughly.

Even if N were small, this would seem not doable. But maybe I don't yet understand well enough.

I understand you may be interested in only the variance components for these effects, not actually effect/coefficient values, but it still seems not doable.

6 Comments

Tom Lane on 18 Apr 2013

Not at all. Two-way random effects is doable. What I question is two-way random effects on N points where there are N different levels of both effects and N^2 interactions. Again, maybe I don't understand your problem yet.

Ryan on 23 Apr 2013

I don't understand why it isn't doable in this scenario, though. How is my data substantially different than, for example, the mock data set that MATLAB uses in that link I provided? What about my data is so unique that makes this impossible?

Tom Lane on 23 Apr 2013

That example has 18 rows, but only 3 factories, 2 car models, and 6 combinations (interactions). There are enough rows to estimate all of these effects.

You say you have 14879 rows, 14879 unique ratees, 14879 unique raters, and 14879^2 potential combinations (though only 14879 of these could actually appear in the data).

Tom Lane

Contact us