MATLAB Answers

Dynamic allocation of mxArray structure

8 views (last 30 days)
Fedor Iskhakov
Fedor Iskhakov on 22 Jun 2017
Commented: Fedor Iskhakov on 23 Jun 2017
Hi everyone,
I have a mex function in C that internally generates an unpredictably large volume of data to be output back to Matlab. I would like the return to be a Struct Matrix of the 1 by N, where N is the unknown large number. The memory should be allocated by chunks, so that in case of insufficient memory the calculation can be stopped and the return that does fit into memory handled safely back to Matlab.
It seems that the combination of mxSetN() and mxRealloc() is a proper combination, but I can't find sufficient info on how to use mxRealloc to allocate space for additional elements of structural matrix.
Here is a minimal code example of what I'm trying to do
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
mwSize i;
mxArray *mxp;
nlhs[0]=mxCreateStructMatrix(1,1,0,NULL); //structure with one element and no fields for now
while (continue calculation)
{
//one more round of calculations that creates a lot of output
//stored in structure g with double* field1,..,K
mxAddField(nlhs[0],"field1");
mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
mxSetField(nlhs[0],i,"field1",mxp);
...
mxAddField(nlhs[0],"fieldK");
mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
mxSetField(nlhs[0],i,"fieldK",mxp);
i++;
mxSetM(nlhs[0],i)
//What then?
}
}

Accepted Answer

Jan
Jan on 22 Jun 2017
Letting an array grow iteratively consumes a lot of resources. Therefore it is a good idea to create the new data in chunks and reply them separately.
As James said already, mxCreateDoubleMatrix stops the Mex function immediately if it is not successful. Then all formerly creatzed data are lost. A workaround would be to call the Mex function for each junk in a loop and collect the data in Matlab:
allChunk = cell(1, 1000);
iChunk = 0;
while ~ready
try
newChunk = myMexfunction();
iChunk = iChunk + 1;
allChunk{iChunk} = newChunk;
if iChunk == numel(allChunk) % Container exhausted
allChunk{iChunk + 1000} = []; % Expand
end
catch
disp('Memory exhausted');
ready = true;
end
end
But if te MEX function could not allocate any memory anymore, Matlab is in danger to crash also. You cannot do a lot of computations anymore, most of all you cannot combine the chunks to a large array.
Perhaps it is better to write the junks to the disk (SSD preferrably), such that an exhaustion is less likely and the memory is not filled at all. This would be much better than a strategy which is expected to cause an out-of-memory crash.

  1 Comment

Fedor Iskhakov
Fedor Iskhakov on 23 Jun 2017
Thanks, saving to disk does sound like most reasonable thing I can do. Running the loop in Matlab would also be more problematic in my application..

Sign in to comment.

More Answers (2)

Philip Borghesani
Philip Borghesani on 22 Jun 2017
Edited: Philip Borghesani on 22 Jun 2017
Although the code example you linked to will do the resizing operation on a structure and in general I like it better then the code you posted. I suggest not ever resizing the main structure but allocating it to some reasonable max size from the beginning. Resizing and reallocating operations are slow and tend to cause memory fragmentation issues. In addition structure arrays have quite a bit of memory overhead proportional to the dimensions of the structure.
Hopefully (and if not try to fix it) the amount of memory used by field1 is relatively large compared with the size of your overall structure. If size is a few thousand or larger then the memory used to store the basic structure is insignificant and it can be allocated to it maximum size initially. Your function can return the max "i" reached as a second output or the calling code can just check for empty fields when processing the data and stop.

  1 Comment

Fedor Iskhakov
Fedor Iskhakov on 23 Jun 2017
I take your point, thanks, I'm aware that I'm looking for a "second best" here. The output format can not be changed in the short term, but in the longer term we should reconsider this whole setting.

Sign in to comment.


James Tursa
James Tursa on 22 Jun 2017
Edited: James Tursa on 22 Jun 2017
"... The memory should be allocated by chunks, so that in case of insufficient memory the calculation can be stopped and the return that does fit into memory handled safely back to Matlab. ..."
You can't do this in a mex routine.
In these lines (and other similar lines in your code):
mxp=mxCreateDoubleMatrix(appropriate,size,mxREAL);
if (mxp==NULL) {mexErrMsgTxt("Failed to allocate memory!")}
mxp will never be NULL and you will never get inside the if-block. Inside a mex routine, if there is an API call that results in an out of memory condition, the mex routine will immediately return control to the caller with an error. In that situation it will never execute the next line in your code (i.e., you will never return from the mxCreateDoubleMatrix function call). The only way you actually get a NULL value from API function calls in your code that allocate memory is if they are used in an Engine application ... it never will happen in a mex routine.

  1 Comment

Fedor Iskhakov
Fedor Iskhakov on 23 Jun 2017
Thanks for pointing this out. I did notice that mxRealloc() behaves exactly like this, but I was sure the check for NaNs returned from mxCreate** routines worked before to detect memory shortage. What a bummer!

Sign in to comment.