GPU fft encounters error

I am using gpu parallel to accelerate computation with Tesla 20, Matlab2015a. However, the program always encounters error at the same gpu operation. Below is the error message:

Error using gpuArray/fft2
MATLAB encountered an unexpected error in evaluation on the GPU. If you continue to see this error, reset the GPU device or restart MATLAB and try again.

I guess that this results from limited gpu memory since when I smaller the data size, the program runs smoothly. However, when the program breaks, I find that there are still 3G space available according to gpuDevice. What could be the problem? Does the system only grants Matlab a limited portion of gpu memory? How can I solve it? Thanks,

[Edric Ellis]

Sometimes the underlying FFT routine (CuFFT) reports unhelpful error conditions when memory is exhausted. What size FFT are you attempting? Which precise card are you using? It's worth noting that all GPU FFTs are carried out with complex numbers - so you need to have enough free space on your GPU to hold the complex result as well as the internal memory required by the FFT algorithm (this can be non-trivial).

[Jingzhao Zhang]

I'm doing 2D FFT on a matrix consisting 40 images of size (1000, 1000). I'm using Nvidia Tesla 20 with 5G memory. It should be a problem caused by memory limit; however, after the error occurs, I find that there's 3G free memory according to command gpuDevice. Is there anyway to use that space?

[Joss Knight]

Are you processing all 1000 images at once? Can you give example code that reproduces the problem?

If you have insufficient memory to support the operation (you may need as much as 8-10x as much memory as is needed for one input array), then after throwing an error, any memory allocated during the operation will become available for use by other functions. Hence gpuDevice will show plenty of free memory.

[Saad Mufti]

I've had similar problems with a simple test program. To benchmark my gpu (same as OP), I steadily increased the array size from 2000 to 10,000. At 10,000 Matlab returned the same error as reported above. However, the puzzling thing is that restarting Matlab lets me get on with more GPU work, but resetting the GPU does not improve the sotuation - still get the same error...

[Joss Knight]

There is a bug in the cuFFT library which causes it to go into an unrecoverable state after a GPU allocation failure in certain circumstances. What I would expect to see, if you are hitting this bug, is first an error which tells you there was an allocation failure, and then from then on, no matter what you do, any call to fft returns error code EXEC_FAILED. Is this what you are seeing? You can continue using everything else in MATLAB, you just cannot do FFTs on the GPU until you restart MATLAB.

Unfortunately the workaround for this for the time being is to avoid such allocation failures by being careful with your resources. You can only recover fully from this problem by restarting MATLAB. Many apologies for this.