Why my deep learning network is producing NaN outputs?

I have a deep CNN network with 39 layers. While training the network is producing NaN output so the loss function is also producing NaN values. My network also has some custom layers which uses 'dlconv' where the weights are intialized using 'initializeGlorot' function. Why the network is producing 'NaN' outputs? Is there any way to solve this? Kindly request to provide suggestions.

Thanking,

Bipin

Answer 1

[Prasanna]

Hi Bipin,

When a deep CNN network produces NaN outputs during training, it typically indicates that there is an issue with the numerical stability of the network. Some reasons for NaN outputs can be:

• Improper weight initializations
• High Learning Rate
• Batch normalizations.

To debug and solve the issue, you can try lowering the learning rate, gradient clipping and check and debug custom layers and their corresponding weight initializations and loss functions. Temporarily, you can use a smaller version of your network to see if the issue persists. This can help isolate whether the problem is due to the network size or specific layers.

For more help on network producing NaN outputs, refer the following links:

• https://www.mathworks.com/matlabcentral/answers/337587-how-to-avoid-nan-in-the-mini-batch-loss-from-traning-convolutional-neural-network
• https://www.mathworks.com/matlabcentral/answers/1917165-training-loss-is-nan-deep-learning
• https://www.mathworks.com/matlabcentral/answers/92319-how-are-nan-values-in-the-input-data-for-a-neural-network-taken-into-account-while-training-the-netw

Hope this helps.

[BIPIN SAMUEL]

Thank you @Prasanna, the initial layer that I was using the network was custom layer and it was generating NaN while initializing the weight values, but it was working properly for the inner layers. When I replace the initial layer with the in-built matlab layers the network is working. So, I will check with that initial layers.