CUDA crashes when training LSTM on GeForce RTX 2080 SUPER

When I try to run the example "Waveform Segmentation Using Deep Learning" Matlab stops the execution at very beginning with:

Error using trainNetwork (line 170)
Unexpected error calling cuDNN: CUDNN_STATUS_EXECUTION_FAILED.

or sometimes with:

Failed to initialize GPU BLAS library.

preceded by multiple copies of theis warning:

Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_LAUNCH_FAILED

when I try to reset the GPU, I get the error:

Error using parallel.gpu.CUDADevice/reset
An unexpected error occurred during CUDA execution. The CUDA error was:
all CUDA-capable devices are busy or unavailable 

and have to exit/restart Matlab. I already set TdrLevel to 0, the GPU is not connected to any screen (although I cannot set it to TCC mode as it is not supported).

I also tried

try
    nnet.internal.cnngpu.reluForward(1);
catch ME
end

as suggested here, but without success.

I use Matlab 2019b Update 1, the gpuDevice output is:

                      Name: 'GeForce RTX 2080 SUPER'
                     Index: 1
         ComputeCapability: '7.5'
            SupportsDouble: 1
             DriverVersion: 10.1000
            ToolkitVersion: 10.1000
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 8.5899e+09
           AvailableMemory: 6.8422e+09
       MultiprocessorCount: 48
              ClockRateKHz: 1845000
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 0
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

My old GPU still works without problems but is just very slow

                      Name: 'GeForce GTX 650 Ti'
                     Index: 2
         ComputeCapability: '3.0'
            SupportsDouble: 1
             DriverVersion: 10.1000
            ToolkitVersion: 10.1000
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 2.1475e+09
           AvailableMemory: 1.6932e+09
       MultiprocessorCount: 4
              ClockRateKHz: 1032500
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 0
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

Is the way to get the 2080 to work?

[Andrea Picciau]

Hi Sebastian,

I'd warmly suggest you get in touch with our technical support team, who'll help you out with the troubleshooting.

Let us know!

Answer 1

[Sebastian Major]

Downgrading(!) the NVIDIA driver to the last stable studio driver (431.86) solved the issue.