CUDA 10.1 not recognized (searching for CUDA 10.0 instead)

Hey Everyone,

I'm working on a fresh install of an Ubuntu 18.04 system. Everything was going well until I got MATLAB 2019a and was trying to compile GPU code.

Namely, when running

mexcuda mexGPUExample.cu

In MATLAB, it will tell me:

Warning: Version 10.0 of the CUDA toolkit could not be found. If installed, set MW_NVCC_PATH environment variable
to location of nvcc compiler.
 
> In mexcuda (line 166) 
Error using mex
No supported compiler was found. For options, visit https://www.mathworks.com/support/compilers.

Error in mexcuda (line 166)
    [varargout{1:nargout}] = mex(mexArguments{:});

So I found my CUDA 10.1 installation at the normal symbolic link place and updated the environment:

setenv('MW_NVCC_PATH','/usr/local/cuda/bin')

Then ran mexcuda again. Still the same error message. However, oddly enough, when I run d = gpuDevice() it works perfectly and shows:

>> d = gpuDevice

d = 

  CUDADevice with properties:

                      Name: 'GeForce GTX 1070'
                     Index: 1
         ComputeCapability: '6.1'
            SupportsDouble: 1
             DriverVersion: 10.1000
            ToolkitVersion: 10
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 8.5126e+09
           AvailableMemory: 8.0890e+09
       MultiprocessorCount: 15
              ClockRateKHz: 1721000
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1

If I update my path variable and run nvcc --version it shows:

$ export PATH=/usr/local/cuda-10.1/bin
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:10:27_PDT_2019
Cuda compilation tools, release 10.1, V10.1.168

Any ideas as to why CUDA is recognized, the GPU works, but MATLAB will not let CUDA 10.1 be CUDA 10.0?

Thanks!

[Dustin Hayden]

May have "figured it out". This site:

https://www.mathworks.com/help/parallel-computing/gpu-support-by-release.html

Indicates that R2019a is only compatible up to 10.0. I presume the error message is simply stating "I won't look for 10.1 because it won't work; 10.0 or bust"?

Answer 1

[Edric Ellis]

Unfortunately, the situation here is admittedly somewhat confusing. There are two relevant pieces of software going on here, both related to CUDA, both potentially with different version numbers. These are:

1. The CUDA driver. It is (nearly) always fine to simply install the latest CUDA driver on your system. NVIDIA guarantees backwards compatibility for the driver, and so older software can run without problems using a newer driver. gpuDevice shows you the CUDA version number of your driver in the DriverVersion property.
2. The CUDA toolkit. This is needed only when building MEX files using mexcuda. In this case, it's important that the version of the toolkit that you use matches the version of the toolkit that was used to build MATLAB. gpuDevice shows you that version in the ToolkitVersion property.

So, the fact that you have a CUDA 10.1 toolkit and are trying to use it with a version of MATLAB built with the 10.0 toolkit is the problem - you should download and install the 10.0 toolkit from the NVIDIA website (it's fine to have multiple toolkit versions on your machine).

[Dustin Hayden]

Thanks for the response! For clarification, when you say CUDA driver, you mean the Nvidia driver (in my case: an Nvidia GeForce GTX 1070). This is version 430, not 10.1. So the DriverVersion property is merely stating that the Nvidia Driver 430 is working with my CUDA toolkit version 10.1?

Regardless, I used that website to get CUDA toolkit originally (the 10.1 version). If there is indeed a CUDA driiver (separate from Nvidia's), then wouldn't the 10.0 CUDA toolkit download the toolkit and the driver?

[Edric Ellis]

Yes, the CUDA driver is the same thing as the Nvidia driver (I suppose you could say that CUDA support is a feature of the Nvidia driver...). Just to add to the confusion, CUDA drivers have a version number like 430 that describes the detailed version number of the driver, but CUDA drivers also can specify the maximum version of the CUDA toolkit that they support. There's a table here: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html (section 1, under heading "CUDA Driver") that shows that a driver version of 430 is compatible with all CUDA toolkit versions up to and including 10.1.

You can install old toolkits without changing your driver - the toolkit is basically the compiler nvcc, associated header files, and libraries like CuBLAS.

[Dustin Hayden]

All right, thanks for the help! Everything works now (and other things that use 10.1 were seemingly unaffected, although now cuda shortcuts to 10.0, so be mindful of that).

1) Get CUDA 10.0

2) Download CUDA 10.0 from the CUDA website (for me, this is https://developer.nvidia.com/cuda-10.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal, but it will differ based on your installation)

3) In the folder that contains the downloaded file, run in the terminal (at least for the file I downloaded):

sudo dpkg -i cuda-repo-ubuntu1804-10-0-local-10.0.130-410.48_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-0-local-10.0.130-410.48/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda-10-0

4) Delete the downloaded deb file and restart the computer. MATLAB will now use the correct NVCC and complile correctly (you might need to update your setpath for MW_NVCC_PATH, if you changed it while debugging).