Array exceeds maximum array size preference

504 views (last 30 days)
luca
luca on 25 Jul 2019
Answered: Muniba Ashfaq on 11 Sep 2020
Hi, given the following
v = [1 2 3 4 1 1 1 1 1 2 3 2];
P = unique(perms(v),'rows');
I've obtained
Error using zeros
Requested 479001600x12 (42.8GB) array exceeds maximum array size preference. Creation of arrays
greater than this limit may take a long time and cause MATLAB to become unresponsive. See array
size limit or preference panel for more information.
Do you know how to fix the problem to get all the possible permutations? One idea was to divide in different cases. For example is the total nuber of permutations is 1000, is there a way to modify the command perms to get just the first 200 elements? Is there a more efficient way?
  4 Comments
Joel Handy
Joel Handy on 25 Jul 2019
There is a lot being asked here with very little info to go on.
We don't have any insight into this algorithm. We don't know what it's supposed to do or how it does it. This makes it hard to suggest ways of generating the permutations in useable chunks.
Regardless, processing 479 billion things is going to take a while. Even if it only takes a microsecond to process each permutation, it will take over 5 days to run.

Sign in to comment.

Answers (2)

Hari Krishna Ravuri
Hari Krishna Ravuri on 29 Jul 2019
I understand that, you are trying to generate all permutations of the array ‘v’ resulting in generating a vector of very large size, which exceeds maximum array size preference. To know more about the variable size preferences, please refer https://www.mathworks.com/help/matlab/matlab_env/set-workspace-and-variable-preferences.html.
As you have mentioned that you are trying to solve NP-complete problem and need all the permutations, I suggest you write a small script which generates all the permutations by rearranging the elements in the array itself i.e. with using constant memory space. Now, in that script, for the first Iteration, create a .mat file and write the permutation to the file and for the remaining iterations, you just need to append every permutation to the file. For writing data to .mat file you can use save function with the syntax save(filename, variables,'-append'). For more information, please refer https://www.mathworks.com/help/matlab/ref/save.html
By writing data to .mat file, you can access all the data without bringing the entire file into the main memory.
“is there a way to modify the command perms to get just the first 200 elements?”
No, as of now, there is no such way mentioned in the documentation. Please refer https://www.mathworks.com/help/matlab/ref/perms.html
Is there a more efficient way?”
As you are solving an NP-complete problem and need all the permutations for sure to get the optimal solution, the best way is to write all the permutations to a file and accessing them, as the entire array will not fit in the main memory. To know more about handling mat files, please refer to https://blogs.mathworks.com/loren/2011/10/14/new-mat-file-functionality-in-r2011b/
If you need a good solution but not the optimal one, you can use the problem heuristics and try to get a good solution in less time.
  1 Comment
Guillaume
Guillaume on 29 Jul 2019
You could indeed generate all the permutations a few at a time (see for example, nextperm, nextvector, another nextperm, and probably more on the FileExchange) and save these to a file (mat or text), and then process them with tall arrays to avoid loading them in memory all at once, it doesn't change the fact that generating 479 millions permutations is going to take a while and processing them even longer.
And this will get exponentially worse, if the vector length is increased. 6,222 millions permutations for 13, 87,178 millions for 14, etc.

Sign in to comment.


Muniba Ashfaq
Muniba Ashfaq on 11 Sep 2020
I was also getting this error. I deleted all unnecessary variables from workspace to free up memory. I just keep those variables that was needed for my MATLAB code execution.
This worked amazingly.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!