Why do I get "Maximum variable size allowed on the device is exceeded." error when running the "semanticseg" function on my image and custom network with a GPU?

I am trying to run the "semanticseg" function on an image with size (8557x11377) and my own custom network, using a GPU. I receive the following error:

ERROR: Error using nnet.internal.cnngpu.convolveForward2D 
Maximum variable size allowed on the device is exceeded. 

I expect MATLAB to tile the image, run the semantic segmentation, and reassemble the tile, but I am receiving this error on a windows laptop with a GTX 1070 card.

The two questions I have are:

1) Why do I receive this error message?

2) I would like to know what the maximum allowed image size I can have with my own custom network when running the "semanticseg" function.

Answer 1

[MathWorks Support Team]

Currently, the maximum value for the total number of array elements on the GPU is limited by CUDA's CUBLAS/CUFFT APIs to " intmax('int32'). The error is likely because the number of elements of the array input to "gpuarray" exceeds this limit. Internally, "semanticseg" creates a gpuarray, and due to the size of the image and the custom network, the limit of  intmax('int32') is being exceeded.

Since this is a very large image to push into a GPU along with the additional memory required for the network and all of its intermediate data, at the moment, there is no tool to support these large image sizes with deep learning.

One workaround would be to tile the image separately and pass the tiles into the "semanticseg" function. A function like "blockproc" can help with the tiling.

A version written in MATLAB R2017b of a tile-based semantic segmentation helper function is also attached, which can be adapted for your specific application.