Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Mex API Wish List

Asked by James Tursa on 14 Jun 2013
Latest activity Commented on by ANDREY on 2 Nov 2013

I thought I would start a thread to list all of the desired changes, upgrades, and additions to the official mex API. So here goes my first cut at a list:

.

----------------------------------------

DOCUMENT EXISTING BEHAVIOR

----------------------------------------

.

1) Fully document the garbage collection behavior for mxArray variables and memory, and how it is affected by the various API functions. For example (but not limited to):

- The fact that using mxSetCell, mxSetField, mxSetFieldByNumber removes the input mxArray from the garbage collection list, and that they change portions of the input mxArray (e.g., change the variable type to 3 indicating sub-element). Also that they work only once on the same input (to put the same mxArray in another element you must bump up the Reference Count ... but the official API includes no functions for doing this).

- The fact that mxSetPr, mxSetPi, mxSetIr, mxSetJc remove the input address from the garbage collection list.

- None of the addresses in the mxArray header (pr, pi, ir, jc) are on the garbage collection list ... their disposition is determined entirely by what happens to the variable they are a part of.

.

2) Fully document variable sharing types and how you get it. E.g.,

- Shared Data Copy (e.g., what you commonly get when you do Y = X)

- Shared Reference Copy (e.g., same mxArray address listed multiple times in a cell array, reference count > 0)

- Shared Parent Copy (e.g., the cell array is shared with another cell array, so elements of one cell array are effectively shared with elements of another cell array, even though there is no CrossLink or Reference Count indicator of this at the element level)

- Shared Handle Copy (e.g., classdef object derived from the handle class)

.

3) Document what happens when you exit a mex routine. E.g.,

- Shared data copies of the plhs[ ] variables are returned, then everything on the garbage collection list is destroyed.

.

4) Fully document what happens when you call mxDestroyArray. E.g., as near as I can tell this is what actually happens:

If ( Reference Count > 0 ) Then
    Decrement Reference Count by 1
Else
    If ( This is not a shared data copy ) Deep Free all the data areas
    Free the mxArray header
    Remove mxArray header from the garbage collection list if it is there
    Remove mxArray header from the CrossLink and Reverse Crosslink lists if it is a shared data copy of something
Endif

.

--------------------------------------------------------------------

MAKE OFFICIAL SOME UNDOCUMENTED API FUNCTIONS

--------------------------------------------------------------------

.

It would be nice if some of the more useful currently undocumented API functions could be officially documented. Creating shared data copies and reference copies is an essential capability for efficiently working with large variables in mex routines. For example (not a complete list):

.

5) mxCreateSharedDataCopy (returns shared data copy)

6) mxGetReference (returns reference count)

7) mxCreateReference (returns input with reference count bumped up by 1)

8) mxUnshareArray (unshares array suitable for cell array manipulation)

9) mxUnreference (decrements reference count by 1)

10) mxIsSharedArray (returns sharing status)

11) mxSetReferenceCount (sets reference count)

12) mxCreateUninitDoubleMatrix (returns double matrix with uninitialized data ... fast)

13) mxCreateUninitDoubleArray (returns double array with uninitialized data ... fast)

14) mxCreateUninitNumericArray (returns numeric array with uninitialized data ... fast)

15) mxFastZeros (returns double matrix with 0's ... fast)

16) mxGetUserBits (gets highest 8 bits of variable flags)

17) mxSetUserBits (sets highest 8 bits of variable flags)

18) mxIsA (checks to see if variable is a specified class)

.

---------------------------

ADD NEW FUNCTIONS

---------------------------

.

19) Complementary functions for putting mxArray variables and memory back on the garbage collection lists once they have been removed. E.g., something like:

- mexMakeArrayTemporary (reverse of mexMakeArrayPersistent)

- mexMakeMemoryTemporary (reverse of mexMakeMemoryPersistent)

- Note: Latter already has an undocumented function available, mxAddToAllocList

.

20) Add inquiry functions for determining when an address is on the garbage collection list. E.g., something like:

- mexIsArrayPersistent

- mexIsMemoryPersistent

- Note: Latter already has an undocumented function available, mxIsOnAllocList

.

21) Add function to get the actual pointer of a property, not a deep copy of it. E.g.,

- mxGetPropertyPtr

.

22) Add functions to set a property directly, or a shared data copy, without a deep copy. E.g.,

- mxSetPropertyPtr

- mxSetPropertySharedDataCopy

.

23) Add functions to determine the nearby mxArray variables in the shared data copy linked list. E.g.,

- mxGetNextSharedDataCopy (returns next mxArray * in CrossLink linked list)

- mxGetPrevSharedDataCopy (returns previous mxArray * in Reverse CrossLink linked list)

.

24) Add function for determining parent (e.g., cell or struct or class). It is recognized that reference copies may have multiple parents. E.g.,

- mxGetParents (returns list of all immediate parents)

.

25) Add smarter versions of mxSetCell, mxSetField, mxSetFieldByNumber that take into account the variable type and current sharing status of the variable and automatically update the reference count etc in a smart way. E.g., something like this:

void mxSetCellSmart( mxArray *mx, mwIndex i, mxArray *cell )
{
    int VariableType;
    mxArray *cell_copy;
    if ( mx == NULL ) return;
    if ( i < 0 ) return;
    VariableType = mxGetVariableType( cell );
    if ( VariableType == 4 ) { // if temporary
        mxSetCell( mx, i, cell );
    } else if ( VariableType == 3 ) { // if already a sub-element of cell or struct
        mxSetCell( mx, i, cell );
        mxCreateReference( cell );
    } else {
        cell_copy = mxCreateSharedDataCopy( cell );
        mxSetCell( mx, i, cell_copy );
    }
}

.

26) Add function & macro for determing MATLAB version. E.g.,

- matlab_version( ) would return MATLAB version currently running

- MATLAB_VERSION would be a macro indicating what MATLAB version is being used to compile (e.g., 0x2012b)

.

27) Add thread-safe functionality to the API, or create separate thread-safe functions for creating mxArray variables, etc.

.

28) All of the above for Fortran. And fix several of the Fortran example programs (some have bugs and will not work as expected for certain inputs, or will not work at all in a 64-bit environment)

5 Comments

dpb on 19 Jul 2013

I especially vote for the removal of "red-haired stepchild" status of Fortran re: mex.

James Tursa on 19 Jul 2013

@Pavel: A few comments about the information in your Undocumented API link.

First, the mxGetPropertyShared and mxSetPropertyShared functions are only available in the library as mangled C++ names as far as I can tell. So you can only use them in a C++ mex routine (trying to link to them in a C mex routine will fail).

Second, your warning to not call mxDestroyArray on the returned pointer of mxGetPropertyShared is unwarranted. Since the returned pointer is pointing to a shared data copy of the original property, it is perfectly OK to call mxDestroyArray on it. In fact, since it is a temporary variable and on the MATLAB garbage collection list, MATLAB will destory it automatically when the mex routine exits even if you don't explicitly do it via mxDestroyArray. Since it is a shared data copy, the only thing that will happen is the mxArray header gets destroyed and then the mxArray address gets removed from the shared data copy linked list (CrossLink). The data area is not free'd.

Third, there is an FEX submission for getting a pointer to the original property that works with C or C++. Not very elegant but it does seem to work OK. You can find it here:

http://www.mathworks.com/matlabcentral/fileexchange/30672-mxgetpropertyptr-c-mex-function

I have a working C/C++ routine for setting a property via a shared data copy as well, but haven't got it uploaded to the FEX yet.

ANDREY on 2 Nov 2013

I fully support item 28) All of the above for Fortran. Although Eigen lib in C++ provides a very convinient way to write mex files it would be even more natural to write them in Fortran 90/2003 if Mathworks makes a really nice API (based on the existing one by James) and Intel give the users Array Visualizer back on 64 bit systems.

James Tursa

Products

No products are associated with this question.

2 Answers

Answer by Jan Simon on 19 Jul 2013
Edited by Jan Simon on 19 Jul 2013

Documentation for these functions would support the work with the API also:

mxFevalFunctionHandle, mexRunMexFile, mexSetCallbacks
mxTranspose, mxTransposeArray, mxInPlaceBlockTranspose
mxTxtCmp, mxCreateStringFromNChars, mxGetNChars,
mxIsExactSharedCopy, mxIsIdentical
mxArgMustBeIndices, mxColonop_fill_with_doubles, mxCreateConvertedCopy
mxGetFields, mxMallocEx, mxSerialize
utAccurateEstimateCPUSpeed, utEqualWithinEps

I do not know a method to access files from the MEX side using the File-ID from opening the file in Matlab. Tunneling the access back to the Matlab level by mexCallMATLAB is not efficient enuogh for my case. I thought of asking a new question shortly before I found this thread. But now let me append this as further enhancement request.

0 Comments

Jan Simon
Answer by Pavel Holoborodko on 20 Jul 2013

@James

Mangled C++ names are very useful in reversing undocumented functions - since it easy to derive original signature of the function: arguments and return type.

You can write small wrapper to use mangled C++ functions in C. Something like this would work:

mexext.h:

extern "C" 
mxArray* mxGetPropertySharedC(const mxArray*, unsigned int, const char * );

mexext.cpp (compile with C++ compiler):

mxArray* mxGetPropertySharedC(const mxArray* pa, 
                              unsigned int index, 
                              const char *propname)
{
      return mxGetPropertyShared(pa,index,propname);
}

And you are right of course - my note on mxDestroyArray is unnecessary. I have fixed my post and added some findings on mxArray_tag.

My previous comment got folded, so I repeat my link once more: Undocumented MEX API

2 Comments

James Tursa on 20 Jul 2013

Yes, I agree that the mangled names can be very useful for signature recovery. But if it is only the mangled names that are exported in the library, you still need a C++ compiler to get at them, even if it is only to compile the wrapper function.

Pavel Holoborodko on 20 Jul 2013

Is there way to install C and C++ compilers separately these days? Usually both of them are distributed in the same package (if not as one executable). Or is it some special compiler like Watcom or similar?

Pavel Holoborodko

Contact us