Full range of performance of the GPU calculation with neural network

Hello,

I have the problem that my GPU (NVIDIA QUADRO M4000) does not bring full performance in neural networks despite GPU parallel compunting. I also tried other cards but did not get any better results either.

Can someone tell me why this may be?

I have already tried it with gpuArrays but only one GPU is used and not two as expected.

    [net,tr] = train(net,traindata, targetclass,'useParallel','yes','useGPU','yes');

[Joss Knight]

Two stupid questions:

1. Do you have two GPUs in your machine? You only mention one. If you do, what happens when you type " parpool('local',2); spmd, gpuDevice, end " ?
2. Did you mean to be using shallow networks, or were you trying to do deep learning? Have you tried using trainNetwork with 'ExecutionEnvironment' 'multi-gpu'?

[MG]

1. Yes, I have two NVIDIA QUADRO M4000 in my machine. When I start pool I can use 6 workers. In the task manager I can see that the GPUs work but only with 15-25% of the real performance.

 >> parpool('local',2); spmd, gpuDevice, end
Starting parallel pool (parpool) using the 'local' profile ...
connected to 2 workers.
Error detected on workers 1 2.
Caused by:
    Error using gpuDevice (line 26)
    An unexpected error occurred during CUDA execution. The CUDA error was:
    all CUDA-capable devices are busy or unavailable

2. I use patternnet: LayerSize = [Layer, Notes]; net = patternnet(LayerSize);

The size of the layers and notes increase in a for-loop to find the best configuration of the net

I hope the answers were understandable and you can help me to solve my problems.

[Joss Knight]

I'm afraid I don't know anything about patternnet. However, it sounds like you're saying your GPUs are both being used but not satisfactorily? Are you comparing to the theoretical maximum compute power? Because that is almost impossible to achieve in practise, and to do so would require a special kind of algorithm, one with no data transfer. 15-25% actually sounds pretty good.

[Joss Knight]

I see you are using 6 workers but you only have two GPUs. So the most each could be used is 33% of the time on each worker. You should only open a pool of two workers, one per GPU.

[MG]

Thank you for your support.

I think the problem in my Script was the small net. When I increase the complexity of the net the GPU performance goes up to 80 %. So maybe my problem is not complex enough to use the full available performance.