Working With Very Large Array

For my work I need to create and work with very large arrays of data (on the order of 1,000*400,000) to simulate many simultaneous random walks. I was wondering what would be the best way to go about this, as I run out of memory before I even get close to creating an array of that size.

Obviously I'll need to write it to the disk in some way. I essentially need to create the array one row at a time, creating the next row from the previous one.

[Walter Roberson]

Those arrays are not quite 4 gigabytes each, even if you use double precision (which might not be necessary for your purpose.) Make sure you use a 64 bit MATLAB and at least 2 gigabytes of RAM plus 4 gigabytes per array of that size that needs to exist simultaneously.

Answer 1

[Adam]

Try looking into using a matfile

doc matfile

It allows you to work with a large data set from file rather than from memory. Access is slower, of course, but not massively so, especially for the first index of the array. It gets slower for accessing subsequent dimensions (or at least it did last time I tested it which was a little while ago).