Determine Matrix Operation Memory Usage

I'm wondering if there is a way to determine what the memory usage will be for a large matrix operation before calling the function. I have made a large data processing chain and have had users MATLAB instances crash do to excess memory usage in a large operation. I'd like to be able to pass in the entire matrix so that MATLAB can crunch though it more quickly, but I'd also like to be able to deterministically break up the matrix to avoid memory issues. Is there a way that I might be able to determine how much of the matrix MATLAB can handle at a time within the code?

Answer 1

[James Tursa]

This is going to depend on the specific operations you are doing, and might also depend on whether any of the variables involved are shared. Doing certain operations might cause variables to suddenly become unshared, which means a deep data copy. If you post the specific operations you are doing we might be able to offer some insight. But unfortunately there is no method provided by MATLAB for determining the sharing status of your variables. For that you have to resort to unofficial methods, which are not 100% reliable. One of those methods (not yet updated for the latest MATLAB versions) can be found here:

https://www.mathworks.com/matlabcentral/fileexchange/65861-isshared-determines-data-sharing-status-among-workspace-variables

Answer 2

[Steven Lord]

Does MATLAB crash or does it throw an error? Those are two very different things.

• Crash: MATLAB shows a crash log file with a lot of technical information at the top, likely brings up a dialog asking if you want to send that crash log file to MathWorks, or may just disappear.
• Throw an error: MATLAB displays some red text in the Command Window but the prompt appears after that text is displayed. You can continue using MATLAB.

If you're experiencing a crash, please send the crash log file and as much information as you can about what was executing when the crash occurred to MathWorks Technical Support using the telephone icon in the upper-right corner of this page.

If you're experiencing errors, if those errors are related to being out of memory (not to exceeding the maximum variable size) the Big Data processing tools described in this section of the documentation may be of use. The Big Data processing functionality in Parallel Computing Toolbox may also be interesting to you.