GPU error + Working with total XTrain data in custom training loop

5 views (last 30 days)
MAHSA YOUSEFI on 2 Nov 2020
Commented: Joss Knight on 7 Nov 2020
Hi. In my custom training loop, (at the moment) I do not want to use mini-batches. Instead I need to work with all 60,000 images XTrain (full batch). I have to check my solver performance without using minibatch at first. My Matlab runs with high performance processor as default.
By the way, for this (a part of my) code I received following error:
I will be thankful for your help to find a solution for this problem.

Answers (1)

Joss Knight
Joss Knight on 4 Nov 2020
The error is telling you you can't process all your data at once because you don't have enough GPU memory. You either have to use mini-batches, or you could try using the CPU instead.
You could try getting a GPU with more memory, but the raw arithmetic doesn't work in your favour. Even if you had 12GB of memory, the largest resolution image you could fit on it is around 230x230 if you want to fit all 60,000; for training you need approximately as much memory as your input data multiplied by the number of layers in your network.
Implementing custom training with batching is relatively simple with minibatchqueue. Take a look. You could even use an arrayDatastore as input.
Joss Knight
Joss Knight on 7 Nov 2020
It's pretty hard to tell because I don't know what network you're using, presumably not one with too many layers. Clearly when you use grayscale data, which takes 179MB, it works and with colour data it doesn't, so the answers are basically: use a GPU with 3 times as much memory, or use 3 GPUs and train in parallel. If you want to do that via Google colab or some other cloud system then go for it.
Of course, training on 3 GPUs isn't exactly the same as training on 1 GPU with 3 times as much memory but the maths is the same.
Take a look at this documentation page to learn how to do multi-GPU training with a custom training loop.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!