MATLAB Answers

0

Eigs in multinode cluster

Asked by Wister Huang on 18 May 2016
Latest activity Edited by Andrew Knyazev on 12 Jun 2019
I am new to high performance computing and I am now trying to reconfigure one of my matlab code to a HPC.(I know it is probably not preferred, but I kind of need it in urgent.)
The task is to solve the lowest 6 eigen values for a 42500*42500 complex sparse Hermician matrix.
I have access to minimum 4 nodes and each node have more than 16GB memory.
I tried to submit this job but get an error
Out of memory. Type HELP MEMORY for your options.
Error in eigs/checkInputs/LUfactorAminusSigmaB (line 987)
[L,U,pp] = lu(AsB,'vector');
Error in eigs/checkInputs (line 809)
[L,U,pp,qq,dgAsB] = LUfactorAminusSigmaB;
Error in eigs (line 96)
checkInputs(varargin{:});
Error in TwoDsolve (line 65)
[psi,E]=eigs(H,NOS,'sm');
Error in Loop_Main (line 9)
[psi,En] = TwoDsolve( V,m2Dx,m2Dy,Grd,'eigs',NOS);
Apparently it only use My question is: is there a way to configure eigs so that it can use all the memory I required from different nodes? I attach my submission file here as well.
#SBATCH --mail-user=wister.huang@student.unsw.edu.au
#SBATCH --mail-type=ALL
#SBATCH --time=00:60:00
#SBATCH --nodes=4
#SBATCH --mem=16384
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=4
echo "myjob.sub"
hostname
module load matlab/2016a
cd $PBS_O_WORKDIR
unset DISPLAY
matlab -nodisplay -r Control
Thanks, Wister

  0 Comments

Sign in to comment.

2 Answers

Answer by Andrew Knyazev on 11 Aug 2018
Edited by Andrew Knyazev on 12 Jun 2019
 Accepted Answer

EIGS has limited support for distributed memory, so you can run it only on a single node, but see the answer from Christine Tobler below.
Google, e.g., for LOBPCG implementations in BLOPEX, SLEPc, Trilinos, hypre, MAGMA for parallel MPI-based eigensolvers that scale to thousands of nodes and run on GPUs.
https://www.mathworks.com/matlabcentral/fileexchange/48-locally-optimal-block-preconditioned-conjugate-gradient
Revision 4.16 adds support for distributed or codistributed arrays available in MATLAB BigData toolbox, e.g.,:
A = codistributed(diag(1:100)); B = codistributed(diag(101:200));
[blockVectorX,lambda]=lobpcg(randn(100,2),A,1e-5,5,2)

  1 Comment

That's great!

Sign in to comment.


Answer by Christine Tobler on 13 Aug 2018

The eigs function is supported for codistributed matrices in the Parallel Toolbox (see help codistributed/eigs. However, this version of eigs does not support the 'sm' / 'smallestabs' option (except for the case of a diagonal or triangular input matrix).
You could try to use the 'smallestreal' option instead, but this often has convergence issues, as it does not compute a decomposition of A, and also doesn't use a preconditioner, as LOBPCG typically does. However, this should be quick enough to try out, as it is just a one-liner.

  1 Comment

Sorry, I realize I started out with the details when I should have started more generally: To distribute a MATLAB array over several compute nodes, use the Parallel Toolbox, and its distributed or codistributed classes (<https://www.mathworks.com/products/parallel-computing.html Parallel Toolbox Documentation>).

Sign in to comment.