Select a group gpu's that are discontinuous in gpuDevice

On a 4 GPU machine I get the following output (truncated) from gpuDevice().

1. Titan V

2. GTX970

3. Titan V

4. Titan V

I want to use gpu's 1, 3, 4 in a multi-gpu pool. The problem is, the GTX970 is driving the display as a WDDM device and so cannot be set to 'prohibited' in nvidia-smi. Leaving it in the pool hugely reduces the batch size maximum and slows the system to a crawl.

How do I go about creating a pool that includes only gpu's 1,3,4 and not 2?

I've seen this answer (https://www.mathworks.com/matlabcentral/answers/351824-selecting-specific-gpus-for-parpool), but I cannot figure out how to adapt it to my situation.

Thanks!

Answer 1

[Joss Knight]

There are two ways:

The best way is to set environment variable CUDA_VISIBLE_DEVICES to 0,2,3 before you start MATLAB, or as the first thing you do when you start MATLAB:

setenv('CUDA_VISIBLE_DEVICES', '0,2,3');

The second best way is to start a pool of 3 workers before you start training, and set the device manually:

parpool(3);
spmd
    if labindex == 1
        gpuDevice(1);
    else
        gpuDevice(labindex+1);
    end
end

A third way is to use your GTX970, but reduce the amount of work it has to do during training using the 'WorkerLoad' training option:

trainingOptions(..., 'WorkerLoad', [1, 0.1, 1, 1]);

The above will set the second worker to have a tenth of the data of the other workers. Note that this will not necessarily result in faster training than removing your 970 from the pool altogether. You'd have to experiment.

[Louis Vaickus]

Very helpful.

Thanks!