Could anyone help me on what basis the number of hidden layers are chosen for deep neural network.

Let me explain in brief.

I have generated the code for deep neural network for regression purpose using numerical data to predict the formation of clusters.

when I run the code, for four hidden layers i can get the lowest value of mean square error as compared to 2 hidden layers,3 hidden layers,5 hidden layers, and 6 hidden layers.

So,I can say four hidden layers are optimal in my case.

But I would like to know is there any other reason other the mean square error to justify why four hidden layers are optimal.

Also let me know, for an image based on pixel, I can find low level features, high level features and so on.

But for numerical data what represent low level and high level features.

Could anyone please clarify me.

Answer 1

[Matt J]

But I would like to know is there any other reason other the mean square error to justify why four hidden layers are optimal.

No, if you change the loss function or any other thing about your network architecture (e.g., number of neurons per layer), you could very well find you get a different optimal number of layers.

But for numerical data what represent low level and high level features.

In general, you won't know that in advance. The main purpose of a neural network is for the network to learn the relevant features on its own are during training.

[jaah navi]

Yes. I can understand that I cannot know the optimal number of hidden layers in advance.

In my case I tried with 2 hidden layers each of which contains 20 neurons and 3 hidden layers each of which contains 30 neurons and I can find mean square error for 2 hidden layers which contains 20 neurons is less as compared to 3 hidden layers with 30 neurons.

But I am unable to find the reason why mean square error is less for 2 hidden layers containing 20 neurons.

Could you please help me on this.

[Matt J]

I can't tell whether you are talking about training error or testing error. If you add more neurons/layers parameters, you will at some point start to overfit your training data and it is typical then to see an increase in your test error.

[jaah navi]

I am talking about the training error. I need to know why for 2 hidden layers with 20 neurons, mean square error ( training error) is low as compared to 3 hidden layers with 30 neurons.

[Matt J]

It might be a good idea to post the training curves for each case.

[Matt J]

Also, the code...

[Matt J]

jaah navi's comment moved here:

Code:

XTrain
YTrain
inputSize = 2;
numHiddenUnits1 = 20;
numHiddenUnits2 = 20;
numClasses = 1;
layers = [ ...
  sequenceInputLayer(2)
  fullyConnectedLayer(20)
  reluLayer
  fullyConnectedLayer(20)
  reluLayer
  fullyConnectedLayer(1)
  regressionLayer]
maxEpochs = 50;
miniBatchSize = 50;
options = trainingOptions('adam', ...
    'ExecutionEnvironment','cpu', ...
    'MaxEpochs',maxEpochs, ...
    'MiniBatchSize',miniBatchSize, ...
    'GradientThreshold',1, ...
    'InitialLearnRate',0.001,...
    'Verbose',false, ...
    'Plots','training-progress');
 net = trainNetwork(XTrain,YTrain,layers,options);

[Matt J]

Yes, but also for the 30 unit case, and also it would be good to see the training curves, since we don't have your data to train with.

[jaah navi]

I have attached the curves for two hidden layers with 20 nodes and three hidden layers with 30 nodes.

[Matt J]

I don't really see a difference in the final RMSEs that I can be certain is signficant. They could just differ by random fluctuations inherent to stochastic gradient descent. You might try reducing the learning rate to see if you can get better descent, however.

[jaah navi]

For 2 hidden layers at 8 iteration (0.5) the error tends to bend down (reduces) but for 3 hidden layers there is no such apperance and from the beginning it remains to continue till the end.

[Matt J]

They both look pretty bad to me. What variations in the hyperparameters did you try?

[jaah navi]

Could you please help me to find the variation of mean square error as attached i.e., why 2 hidden layers results in less error as comapred to 3 hidden layers.

[Matt J]

As with the earlier data sets, I think the problem is that the training is not converging for the 3-layer case, and maybe even the 2-layer case as well. Reducing the learning rate may help, or increasing the minibatch size.

[Matt J]

jaah navi's comment moved here:

As mentioned, the training is not converging for the 3-layer case and 2-layer case is there any reason other than the convergence state.

[Matt J]

Reason for what? If the results don't represent converged networks, there is no basis on which to compare them.