MATLAB Answers

num2cell out of memory

8 views (last 30 days)
Jorge Aarón Morán Holguín
Commented: per isakson on 27 Dec 2019
I have a huge dataset that when loaded in the workbench it turns into a 67x20838735 double. Then problem is that when I try to convert it into cell array I get the following message: Out of memory. Type "help memory" for your options.
inputs = num2cell(inputs)
I trien to find a way to solve this problem with loops, but I don't find any way to succeed.

  0 Comments

Sign in to comment.

Answers (2)

Uerm
Uerm on 23 Dec 2019
Hi,
You are working with quite a large variable. I have tried the following
n = rand(67,20838735);
When running whos it says that the variable n uses a bit more than 11 GB of memory. That is more than what most of the normal computers/laptops have :) I think that your computer simply does not have enough memory (RAM). How much RAM does your computer have?
There are some things that you can try.
1) Before converting to cell arrays, you can try to convert the variable from double to single:
n = single(n);
This will make the variable n use half of the memory it did when it was a double.
2) Normally, MATLAB limits the maximum array size to a percentage of the RAM of the computer. You can get rid of this limit by this path:
Click the Home tab --> Click Preferences --> Choose Workspace in the left menu --> On the bottom of the page there is a box saying "Limit the maximum array size to a percentage of RAM". Make sure that this box is empty --> Click Apply and OK.
I suggest that you start with step 2. If that does not work, supply with step 1.
If none of the above works, you can make MATLAB use virtual memory. You can read more about it here:
The tradeoff of using virtual memory is the computation speed!

  1 Comment

Jorge Aarón Morán Holguín
Hello,
My computes has a total 96GB of memory.
>> memory
Maximum possible array: 45018 MB (4.720e+10 bytes) *
Memory available for all arrays: 45018 MB (4.720e+10 bytes) *
Memory used by MATLAB: 93519 MB (9.806e+10 bytes)
Physical Memory (RAM): 96922 MB (1.016e+11 bytes)
* Limited by System Memory (physical + swap file) available.
I have tried both options combined and the problem remains.

Sign in to comment.


per isakson
per isakson on 24 Dec 2019
Edited: per isakson on 24 Dec 2019
Why do you want to convert the numerical matrix to a cell array? It requires an enormous amount of memory for overhead.
>> clearvars
>> num = (1:12);
>> cac = num2cell( num );
>> whos num cac
Name Size Bytes Class Attributes
cac 1x12 1440 cell
num 1x12 96 double
I think the resulting cell array will need (1440/96)*11 GB. That's twice as much as your ram.

  2 Comments

Jorge Aarón Morán Holguín
I am trying to train a neural network using the train() function with quite a big dataset and when loaded it turns into 2 numerical matrixes, X (inputs) and Y (target).
per isakson
per isakson on 27 Dec 2019
But that doesn't answer the question why you try to convert a numerical array into a cell array with the statement
inputs = num2cell(inputs)

Sign in to comment.

Sign in to answer this question.

Products


Release

R2019a