Out of memory error when working with small matrices

Calling

S = [[4,1];[1,4]]; eig(S)

Gives an out of memory error in Matlab R2020a on my Ubuntu 18.04 machine with 16GB RAM. ulimit returns "unlimited".

Calling

S = [[4,1];[1,4]]; [L,flag] = chol(S)

Gives the following error: "Error using chol. Requested 140466905415679x140466905415679 (17179869184.0GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive."

Any clue?

[Christine Tobler]

Something's going very wrong there. Are you getting similar problems when you're using functions that are not part of linear algebra (for example S.*S or sum(S))?

If not, could you call "version -lapack" and "version -blas" and tell us the results?

[Makrushka]

Things go wrong only for linear algebra functions. See my reply below.

Answer 1

[Makrushka]

Launching Matlab with the "hack" below fixes the problem, but it would still be interesting to know whether there is a better way to tell Matlab which Intel MKL build to use by default.

LD_PRELOAD="/usr/local/MATLAB/R2020a/bin/glnxa64/mkl.so:/usr/local/MATLAB/R2020a/bin/glnxa64/mklcompat.so:${LD_PRELOAD}" matlab

[Christine Tobler]

So is the problem that MATLAB was using your local MKL version instead of the one that is shipped with MATLAB? Is your goal to use a different verison of MKL, or did this just happen by accident?

The path to the BLAS and LAPACK version to use can be set in MATLAB using the environment variables BLAS_VERSION and LAPACK_VERSION. Is it possible you had set these environment variables for some other project before starting MATLAB?

[Makrushka]

> So is the problem that MATLAB was using your local MKL version instead of the one that is shipped with MATLAB? Is your goal to use a different verison of MKL, or did this just happen by accident?

Yes, correct, it happened by accident since I have the LD_PRELOAD environment variable set normally to load my local MKL installation, while I guess Matlab used its own built-in MKL version during install so the two conflict. My workaround modifies LD_PRELOAD only for the matlab command being launched so that Matlab finds the built-in MKL install first and uses that.

It would be nice to know how to install Matlab with a local MKL install on Linux machines since the tutorials online are typically for Windows only, but I am more than happy to use the Matlab built-in MKL install so problem solved.

Thanks a lot Christine, your comments helped a lot!

[Liam H]

I have tried your suggestion above with LD_PRELOAD. I am getting a "/bin/sh: error while loading shared libraries: libiomp5.so: cannot open shared object file: No such file or directory" error. I can't seem to locate that library, do you know where it is/what other library I need to add to the preload to get this to work for me.

Thanks!

[Makrushka]

Mmm... not sure. The LD_PRELOAD trick is a hack that I needed just because I am tampering with the LD_PRELOAD variable for some other reason. What the Mathworks staff suggested as a more standard workaround is to set the BLAS_VERSION and LAPACK_VERSION variables to empty values before loading Matlab instead. Can you try that?

Also, are you sure you have my exact problem? have you checked with "version -lapack" and "version -blas" in the Matlab command line and are you sure that Matlab is loading a non-default version of either Lapack or BLAS?

[Liam H]

Thanks for the quick reply Makrushka!

The problem I am facing, is that an external program needs BLAS_VERSION and LAPACK_VERSION to another install of LAPACK/BLAS (for speed reasons) and this version doesn't seem to work for MATLAB and MATLAB's version doesn't work for this program. So I need to somehow link the external program with my Openblas version and MATLAB with its own version...

Yes, the MATLAB is certainly loading a non-default version of LAPACK and BLAS.

[Walter Roberson]

Liam H:

Would it work for your startup.m to set BLAS_VERSION and LAPACK_VERSION to point to Mathwork's version? Or, for that matter, to unset the environment variables? Then MATLAB might invoke the correct version.

This would not work if you need to set BLAS_VERSION and LAPACK_VERSION for some program that you need to system() out to from inside MATLAB -- not unless you set the appropriate variables at the time you do the system()

[Makrushka]

Liam H:

Something that also works for me is the following:

(unset -v BLAS_VERSION; unset -v LAPACK_VERSION; matlab)

This unsets the BLAS_VERSION and LAPACK_VERSION environment variables only for the command within the brackets by creating a subshell. Taken from here: https://unix.stackexchange.com/questions/388844/unset-an-environment-variable-for-one-command

Since my issue is with LD_PRELOAD in my case I could use this:

(unset -v LD_PRELOAD; matlab)

If this does not work you might want to consider positing as separate question so that someone from the MATLAB staff can help?

Answer 2

[Steven Lord]

Can you confirm that you haven't created or downloaded your own eig.m and chol.m that's taking precedent over the built-in eig and chol functions? The output of these two function calls should show this information.

which -all eig
which -all chol

Can you show the full and exact text that is displayed in the Command Window when this error occurs? Include all the red and orange text displayed. Copy it verbatim from the Command Window and paste into a comment.

[Makrushka]

which -all eig
built-in (/usr/local/MATLAB/R2020a/toolbox/matlab/matfun/eig)
built-in (/usr/local/MATLAB/R2020a/toolbox/matlab/matfun/@double/eig)    % double method
built-in (/usr/local/MATLAB/R2020a/toolbox/matlab/matfun/@single/eig)    % single method
/usr/local/MATLAB/R2020a/toolbox/parallel/parallel/@codistributed/eig.m  % codistributed method
/usr/local/MATLAB/R2020a/toolbox/parallel/gpu/@gpuArray/eig.m            % gpuArray method
/usr/local/MATLAB/R2020a/toolbox/symbolic/symbolic/@sym/eig.m            % sym method
which -all chol
built-in (/usr/local/MATLAB/R2020a/toolbox/matlab/matfun/chol)
built-in (/usr/local/MATLAB/R2020a/toolbox/matlab/matfun/@double/chol)    % double method
built-in (/usr/local/MATLAB/R2020a/toolbox/matlab/matfun/@single/chol)    % single method
/usr/local/MATLAB/R2020a/toolbox/parallel/parallel/@codistributed/chol.m  % codistributed method
/usr/local/MATLAB/R2020a/toolbox/parallel/gpu/@gpuArray/chol.m            % gpuArray method
/usr/local/MATLAB/R2020a/toolbox/symbolic/symbolic/@sym/chol.m            % sym method

For chol I wrote the error message above, for eig it is just "Out of memory. More information". However, the problem is that Matlab is trying to use my own version of the MKL library, see below.

Answer 3

[Makrushka]

Ok, this seems to be the problem: I have my own version of the Intel MKL library installed while Matlab tries to use its own? I get the following:

This works (not my Intel MKL library install):

>> version -lapack

ans =

'Intel(R) Math Kernel Library Version 2019.0.3 Product Build 20190125 for Intel(R) 64 architecture applications, CNR branch AVX2

Linear Algebra PACKage Version 3.7.0

'

>> version -blas

ans =

'Intel(R) Math Kernel Library Version 2019.0.3 Product Build 20190125 for Intel(R) 64 architecture applications, CNR branch AVX2

'

This does not work (my own Intel MKL version):

>> version -lapack

ans =

'Intel(R) Math Kernel Library Version 2019.0.5 Product Build 20190808 for Intel(R) 64 architecture applications, CNR branch AVX2

Linear Algebra PACKage Version 3.7.0

'

>> version -blas

ans =

'Intel(R) Math Kernel Library Version 2019.0.5 Product Build 20190808 for Intel(R) 64 architecture applications, CNR branch AVX2

'

Is there any way to choose the MKL library version that Matlab uses?