MATLAB Answers

CUDA crashes when training LSTM on GeForce RTX 2080 SUPER

33 views (last 30 days)
When I try to run the example "Waveform Segmentation Using Deep Learning" Matlab stops the execution at very beginning with:
Error using trainNetwork (line 170)
Unexpected error calling cuDNN: CUDNN_STATUS_EXECUTION_FAILED.
or sometimes with:
Failed to initialize GPU BLAS library.
preceded by multiple copies of theis warning:
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_LAUNCH_FAILED
when I try to reset the GPU, I get the error:
Error using parallel.gpu.CUDADevice/reset
An unexpected error occurred during CUDA execution. The CUDA error was:
all CUDA-capable devices are busy or unavailable
and have to exit/restart Matlab. I already set TdrLevel to 0, the GPU is not connected to any screen (although I cannot set it to TCC mode as it is not supported).
I also tried
try
nnet.internal.cnngpu.reluForward(1);
catch ME
end
as suggested here, but without success.
I use Matlab 2019b Update 1, the gpuDevice output is:
Name: 'GeForce RTX 2080 SUPER'
Index: 1
ComputeCapability: '7.5'
SupportsDouble: 1
DriverVersion: 10.1000
ToolkitVersion: 10.1000
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 8.5899e+09
AvailableMemory: 6.8422e+09
MultiprocessorCount: 48
ClockRateKHz: 1845000
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 0
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1
My old GPU still works without problems but is just very slow
Name: 'GeForce GTX 650 Ti'
Index: 2
ComputeCapability: '3.0'
SupportsDouble: 1
DriverVersion: 10.1000
ToolkitVersion: 10.1000
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 2.1475e+09
AvailableMemory: 1.6932e+09
MultiprocessorCount: 4
ClockRateKHz: 1032500
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 0
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1
Is the way to get the 2080 to work?
  1 Comment
Andrea Picciau
Andrea Picciau on 17 Oct 2019
Hi Sebastian,
I'd warmly suggest you get in touch with our technical support team, who'll help you out with the troubleshooting.
Let us know!

Sign in to comment.

Accepted Answer

Sebastian Major
Sebastian Major on 2 Nov 2019
Downgrading(!) the NVIDIA driver to the last stable studio driver (431.86) solved the issue.

More Answers (0)

Tags

Products


Release

R2019b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!