Matlab Engine Array Transfer Size Limitation

2 views (last 30 days)
I am using engGetVariable() to transfer an mxArray from the Matlab Engine to my C++ program. The documentation says: "The limit for the size of data transferred is 2 GB."
Why is this limitation present? Is there a way to increase it without breaking up the array into separate arrays? I need to transfer large arrays, potentially up to 256GB.
I like the simplicity of the Matlab Engine, so I would rather avoid having to break up, transfer, and re-assemble large arrays.
  3 Comments
Jonathan
Jonathan on 11 Nov 2015
Edited: Jonathan on 11 Nov 2015
Yes, actually I wrote one. The problem is that I need to solve a large, sparse matrix (Ax=b) on a cluster using Intel Pardiso Sparse Cluster solver.
Unfortunately, integrating Matlab with MPI and using the unsupported Intel MPI compilers seems infeasible.
In fact, I wrote a test program that solves the equation without the MPI code, compiled with GCC using MEX, and it exhibits some weird memory behavior.
So from the above, I concluded that getting this to work in MEX, running on a cluster with MPI is perhaps infeasible.
With regard to transferring 256GB, since I am doing the transfer within a single node (not over a network), it shouldn't take more than a few seconds in theory.
However, are you suggesting that I write my own MEX function that bypasses Matlab Engine to transfer directly using Linux IPC routines?

Sign in to comment.

Answers (1)

Walter Roberson
Walter Roberson on 11 Nov 2015
The 2 Gb limit is for compatibility with 32 bit versions of MATLAB.
You should be considering using memmapfile() or shared memory

Categories

Find more on Startup and Shutdown in Help Center and File Exchange

Products

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!