"depth" of mexMakeArrayPersistent ?

3 views (last 30 days)
I am writing a mex library (DLL) that needs to receive data asynchronously and place that data into a fairly complicated mex structure array, containing subarrays and of course value arrays. That structure array can grow or shrink, depending on how much data was received. On invocation of the mex function by the user, the user will get returned the array. For this I have created a secondary thread and also created the array (currently using a maximum static size) and called "mexMakeArrayPersistent" on that array. When the user calls the function, I use "mxDuplicateArray" to return a copy of the array that is marked persistent (to decouple writing to and reading from the array, using an additional mutex requested from secondary thread as well as from Matlab main thread).
Question: If I call "mexMakeArrayPersistent" on the root array, will that also cover all subarrays and value arrays ? Will it also cover removed/added arrays created under the root array ?

Accepted Answer

James Tursa
James Tursa on 26 Oct 2015
Edited: James Tursa on 28 Oct 2015
Repeated from this link:
Persistence of mxArray variables:
There are two types of memory that are returned by MATLAB API functions, mxArray's (e.g., from mxCreateDoubleMatrix, mxDuplicateArray, etc), and raw memory (e.g., from mxMalloc, mxCalloc, etc). Almost all of these official API functions place the pointers on a garbage collection list maintained by the MATLAB Memory Manager (the only exception I know of is mxArrayToString, which I think is an oversight by TMW and should be fixed ... this address should have been placed on the garbage collection list but it isn't ... you have to free it manually). When a mex function returns to the caller, two things typically happen:
1) If it is not an error return (e.g., via mexErrMsgTxt), then shared data copies of the plhs[*] mxArray's are created and returned to the caller.
2) All memory (mxArray's and raw memory) on the garbage collection list is free'd. In the case of mxArray's, it is a deep free. So for cell and struct arrays, all of the cell elements and field elements are deep free'd for all levels.
Calling mexMakeArrayPersistent or mexMakeMemoryPersistent does one simple thing ... it removes the address from the garbage collection list. (In the background, it also changes type of the argument from Temporary to Normal).
Data pointers (i.e., the address returned by mxGetPr, mxGetPi, mxGetData, mxGetIr, mxGetJc) are not on the garbage collection list per se. The memory behind them gets free'd when the mxArray they are a part of gets destroyed. E.g., if you call mxDestroyArray on an mxArray, the memory behind its data pointers gets free'd first, then then memory for the mxArray itself gets free'd. This is the case for cell and struct mxArray variables as well, in a deep sense. When an mxArray variable gets attached to a cell or struct mxArray, its address gets removed from the garbage collection list (KEY POINT!). E.g., consider the following sequence of steps:
mxArray *mx, *cell;
mx = mxCreateCellMatrix( 1, 1 ); // (1)
cell = mxCreateDoubleScalar( 2.0 ); // (2)
mxSetCell( mx, 0, cell ); // (3)
mxDestroyArray( mx ); // (4)
Explanation:
(1) mx is a Temporary mxArray, its address is on the garbage collection list. The result of the call mxGetData(mx) will return a pointer to the data area ... this pointer is not in and of itself on the garbage collection list. There is room in the data area for one element (an mxArray address). Currently that data value is NULL.
(2) cell is a Temporary mxArray, its address is on the garbage collection list. The result of the call mxGetPr(cell) will return a pointer to the data area ... this pointer is not in and of itself on the garbage collection list. There is room in the data area for one element (a double value). Currently that data value is 2.0.
(3) The address of cell gets put in the first data element area of mx (i.e., the NULL value gets overwritten by the cell address). The address of cell gets removed from the garbage collection list. The type of cell gets changed from Temporary to Sub-Element (i.e., it now knows it is part of a cell or struct mxArray). Note that the actual address of cell is what gets put into mx ... not a copy of some sort.
(4) The elements of mx (in this case only one element, the mxArray associated with cell) get deep destroyed. Then mx itself gets free'd. Then mx gets removed from the garbage collection list.
Now back to your specific question:
> Question: will calling "mexMakeArrayPersistent" on the root array be sufficient for all included subarrays and value arrays ?
Yes. In your case, you have a persistent (i.e., not on the garbage collection list) mxArray struct that is holding everything. You are required to manually destroy this at some point prior to the mex function getting cleared or the memory will get permanently leaked until MATLAB restarts. When you add any mxArray to this struct (e.g., via mxSetField or mxSetFieldByNumber), that act will remove the element from the garbage collection list (e.g., see step (3) above) and it will become type Sub-Element. Its disposition is now completely determined by the struct it is contained in. When you call mexMakeArrayPersistent on your root array, the only thing that happens is the address of this root array is removed from the garbage collection list (and its type changes from Temporary to Normal). It is not necessary to, at this time, remove the addresses of the subarrays because these addresses were already removed from the garbage collection list the instant they were added to the struct array.
> Question: Will it also cover subsequently deleted/added subarrays and value arrays ?
Yes, provided you do it properly. To properly destroy a subarray, you need to do two things: Call mxDestroyArray on the address, and then NULL out the value in the parent array. E.g.,
mxArray *mx, *cell;
mx = mxCreateCellMatrix( 1, 1 ); // (1)
cell = mxCreateDoubleScalar( 2.0 ); // (2)
mxSetCell( mx, 0, cell ); // (3)
cell = mxGetCell( mx, 0 ); // (4)
mxDestroyArray( cell ); // (5)
mxSetCell( mx, 0, NULL ) : // (6)
mxDestroyArray( mx ); // (7)
Steps (1) - (3) are as above. Now suppose we wanted to destroy the element prior to destroying the struct itself. Step (4) gets the address of the mxArrray element we want to destroy. Step (5) destroys this element. At this point in the code, mx is in an invalid state. It contains an invalid address as one of its data elements. If we were to call mxDestroyArray( mx ) at this time, MATLAB would crash with a seg fault because the MATLAB Memory Manager would try to access invalid memory behind the cell address while trying to destroy this element. What we must do is step (6) first ... NULL out that element value. Then step (7) will work properly.
SInce you are working with threads, I will caution you that none of the API functions that allocate memory (mxArray or raw) are thread safe. Seems like you have that protected, which is good. One word of caution, mxGetDimensions might allocate memory in some circumstances (I am not at all sure of this, but have evidence that it might be the case if mwSize is 32-bits on a 64-bit system).
  2 Comments
Lars Erdmann
Lars Erdmann on 28 Oct 2015
Hallo James,
thank your for the excellent explanation. It completely answers my question.
James Tursa
James Tursa on 30 Jan 2019
From my Answer above:
"(the only exception I know of is mxArrayToString, which I think is an oversight by TMW and should be fixed ... this address should have been placed on the garbage collection list but it isn't ... you have to free it manually)"
Update: As of R2017a the memory allocated by mxArrayToString is on the garbage collection list.

Sign in to comment.

More Answers (1)

Philip Borghesani
Philip Borghesani on 26 Oct 2015
mexMakeArrayPersistent should only be called on the top level cell array or structure. When an mxArray is placed in a cell or field it's ownership and lifetime are bound to the containing object. Calling mexMakeArrayPersistent on an array already contained in another array will error or result in undefined behavior.
The description of what you are doing concerns me however. mx... and mex... functions are not guaranteed to be thread safe. The memory allocation routines are nearly there and may work for you in recent versions of MATLAB but there are no guarantees. For instance an out of memory error in a different thread will most likely crash MATLAB.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!