It seems to me that there is a problem in Parallel Computing Toolbox (PCT) with CUDA operation. The problem is that the setting of CUDA device in PCT fails, if CUDA device has been set up earlier within the same process by someone else, therefore, I cannot use PCT functionality, working with CUDA in my application which is also used by CUDA.
I will give you a simple example. Here it is:
% Starting MATLAB Environment...
% 1. Load a DLL file used by CUDA (or MEX file, it does not matter, only for example) loadlibrary('MyComputingLib.dll', 'MyComputingLib.h', 'alias', 'cuda_lib')
% 2. perform calculations...
% 3. try to use PCT CUDA... gpuDevice()
% ... and the error message appears:
An unexpected error occurred during CUDA execution. The CUDA error was: setting the device when a process is active is not allowed.
If the aforementioned actions are performed in the reversed order, the error does not occur (My software works correctly with CUDA device setting). Why cannot MATLAB set up CUDA device, when, starting with CUDA 4.0 version, it is possible to set up CUDA device a multiple number of times within one process?
I think this limitation is the rudiment of what has been left from the old CUDA versions (old derivers), when it was impossible to set up a device a multiple number of times after the process start.
Thank you in advance for your answer.
In MATLAB versions up to R2016b, MATLAB's built-in commands must be the first activity on each device within the MATLAB process after a device reset. If you are calling into code that makes a CUDA API call on any device before MATLAB does, MATLAB will error unless you call cudaDeviceReset() on that device before MATLAB tries to access it.
This happens because MATLAB sets device flags when switching to or resetting a device, and this is only allowed before any kernels have run after a reset.
As some have pointed out, if you have only one GPU you can work around this by making sure MATLAB (or any MATLAB workers in a pool) uses the GPU before any of your custom functionality. The more robust solution for your own multi-gpu custom code is to call cudaDeviceReset() on each device you have used before you return control to MATLAB.
This behaviour will no longer occur in the next version of MATLAB.
I seem to be experiencing the same thing. Hope TMW can comment...
However, if what I'm seeing is the same as you, a simple workaround is to do something like this in startup.m
a=gpuArray(1); clear a
I find that as long as gpuArray is the first thing to use the GPU in the MATLAB session, things seem to work normally.