I am calling from MATLAB a fairly complicated routine written in C, part of which is using the FFTW3 library. Lately, I encontered the strangest crashes, which I tracked down to the following reproducible situation. I have a DCT routine written in C in a MEX file and using FFTW3 library, which I call DCT_MEX (I give below the source and the compiling mex command). This routine is, apparently, working perfectly fine: first, I can call it several hundreds of times with several hundreds of different input arrays, without getting any error; second, I can compare its output to, say, the MATLAB routine DCT (after normalizing my routine properly to get a unitary transform), getting a relative error of the order of the machine precision. Now, for some reasons, I need to call the MATLAB routine CONV2 on some array. There comes the most unexpected bug : later calls of my DCT_MEX routine crash, if, and only if, no calls have been made to it before the call to CONV2 since the begining of the MATLAB session.
I can reproduce the bug with the following script:
M = 100;
N = 100;
w = [10 1];
A = randn(M, N);
B = randn(M, N);
f = randn(w);
experience = 2;
switch experience
case 1
D = dct_mex(B);
case 2
C = conv2(A, f);
D = dct_mex(B);
case 3
D = dct_mex(B);
C = conv2(A, f);
D = dct_mex(B);
end
Below is the DCT_MEX source code (dct_mex.c).
#include "matrix.h"
#include "mex.h"
#include <fftw3.h>
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
/* get input */
const int L = mxGetM(prhs[0]);
const int N = mxGetN(prhs[0]);
const double *signals = (double*) mxGetData(prhs[0]);
/* create output */
plhs[0] = mxCreateNumericMatrix(L, N, mxDOUBLE_CLASS, mxREAL);
double *coefficients = (double*) mxGetData(plhs[0]);
/* prepare dct */
const int LL[] = {L};
fftw_r2r_kind kind[] = {FFTW_REDFT10};
const fftw_plan fftpln = fftw_plan_many_r2r(1, LL, N, (double*) signals, LL, 1, L, coefficients, LL, 1, L, kind, FFTW_ESTIMATE | FFTW_PRESERVE_INPUT);
/* perform dct */
fftw_execute(fftpln);
/* free fftw stuff */
fftw_destroy_plan(fftpln);
}
I compile it with "mex ./dct_mex.c LDFLAGS="\$LDFLAGS -lfftw3 -lm".
Some information: I run MATLAB 8.1.0.604 (R2013a), under x86_64 GNU/Linux 3.19.3-3-ARCH. Version of fftw3 seems to be 3.4.4. The compiler used by the mex command is gcc 5.2.0. At compile time, I get a warning 'The version currently supported with MEX is "4.4.x"', but I get this with numerous other mex routines I compiled and used without problem. By putting some "mexPrintf("..."); mexEvalString("pause");" within dct_mex.c, I can tell that the crash always occurs at the plan creation, i.e. the call to "fftw_plan_many_r2r".
Finally, let me note that there problems have been reported in the past of FFTW within MEX files, seemingly due to conflicts in memory management: see Caling fftw (v3) into a mex file and mex crash with fftw calls. To my knowledge, none of these got satisfactorily answered (the former goes back to 2004!). I think it is time to know what are the problems of (and what are the workarounds for) using FFTW within MEX. Not being able to do this is a huge drawback for MATLAB. 
