MATLAB Central Newsreader - Question about number of layers in ANN

Question about number of layers in ANN

Kloser — Thu, 29 Oct 2009 08:15:23 -0400

Dear All
I want to know, in Neural Networks, whether more layers will promise higher accuracy for classification?
I have tried using two networks to classify some images into classes. There are 100 variables in each sample, meaning that there are 100 input nodes. And for the Networks, The 1-layer network have 60 output nodes, and the 2-layers network have 20 hidden nodes in the hidden layers and 60 output nodes.
But I found that the accuracy in the 1-layer network is higher than that in the 2-layer. I expected that the 2-layer may have higher accuracy because there are more parameters, so I am wondering whether there are bugs in my implementation. But I can't find any bugs up to now.
So can anyone tell me if that is my wrong expectation or other reasons so I have this result? And what is the relationship between the numbers of layers and the accuracy? How should I judge the optimal numbers of layers for neural networks?
Thank you

Re: Question about number of layers in ANN

Gavrilo Bozovic — Thu, 29 Oct 2009 08:24:06 -0400

"Kloser " <kcfcheung@gmail.com> wrote in message <hcbiur$5g7$1@fred.mathworks.com>...
> Dear All
> I want to know, in Neural Networks, whether more layers will promise higher accuracy for classification?
> I have tried using two networks to classify some images into classes. There are 100 variables in each sample, meaning that there are 100 input nodes. And for the Networks, The 1-layer network have 60 output nodes, and the 2-layers network have 20 hidden nodes in the hidden layers and 60 output nodes.
> But I found that the accuracy in the 1-layer network is higher than that in the 2-layer. I expected that the 2-layer may have higher accuracy because there are more parameters, so I am wondering whether there are bugs in my implementation. But I can't find any bugs up to now.
> So can anyone tell me if that is my wrong expectation or other reasons so I have this result? And what is the relationship between the numbers of layers and the accuracy? How should I judge the optimal numbers of layers for neural networks?
> Thank you

If you have more layers, you will have a higher accuracy for classification, but only if you have an extended training set.

If you can't train your neural network on enough inputs, adding layers won't help, on the contrary.

Re: Question about number of layers in ANN

Kloser — Thu, 29 Oct 2009 08:37:02 -0400

Thank you, Gavrilo,
But could you please provide some more detail about the relationship between the input and the number of layers?

"Gavrilo Bozovic" <gavrilo.dot.bozovic@gmail.dot.ch> wrote in message <hcbjf6$a7o$1@fred.mathworks.com>...
> "Kloser " <kcfcheung@gmail.com> wrote in message <hcbiur$5g7$1@fred.mathworks.com>...
> > Dear All
> > I want to know, in Neural Networks, whether more layers will promise higher accuracy for classification?
> > I have tried using two networks to classify some images into classes. There are 100 variables in each sample, meaning that there are 100 input nodes. And for the Networks, The 1-layer network have 60 output nodes, and the 2-layers network have 20 hidden nodes in the hidden layers and 60 output nodes.
> > But I found that the accuracy in the 1-layer network is higher than that in the 2-layer. I expected that the 2-layer may have higher accuracy because there are more parameters, so I am wondering whether there are bugs in my implementation. But I can't find any bugs up to now.
> > So can anyone tell me if that is my wrong expectation or other reasons so I have this result? And what is the relationship between the numbers of layers and the accuracy? How should I judge the optimal numbers of layers for neural networks?
> > Thank you
>
> If you have more layers, you will have a higher accuracy for classification, but only if you have an extended training set.
>
> If you can't train your neural network on enough inputs, adding layers won't help, on the contrary.

Re: Question about number of layers in ANN

Gavrilo Bozovic — Thu, 29 Oct 2009 16:00:08 -0400

The more you have layers, the more axons and synapses you have, and therefore the more parameters you can tune.

But to be able to tune them, you need to have a lot of input data.

Take thi: if you have a series of ten thousand points in 2d, you can fit a 10th order polynom if it approximates better your series.

But if you have only two points, there's no use to try fitting a polynom of an order superior to two, although it is theoretically possible.

That's the same for your network : if you have a too reduced training set, you won't be able to tune the parameters of a large model satisfactorily.

Re: Question about number of layers in ANN

Greg Heath — Thu, 29 Oct 2009 17:07:02 -0400

On Oct 29, 4:15 am, "Kloser " <kcfche...@gmail.com> wrote:
> Dear All
> I want to know, in Neural Networks, whether morelayerswill promise higher accuracy
> for classification?

No. Just, perhaps, fewer weights.

> I have tried using two networks to classify some images into classes. There are 100
> variables in each sample, meaning that there are 100 input nodes. And for the
> Networks, The 1-layer network have 60 output nodes,

The 1-layer network is equivalent to a linear or logistic model.
What is the output transfer function?
How large is your training sample?

For I = 100 inputs, and O=60 outputs you will have Nw = (I+1)*O =
6060 unknown
weights. With Ntrn training samples you will have Neq = Ntrn*O =
60*Ntrn equations.

To mitigate noise and measurement error it is desirable to have

Neq >> Nw

or

Ntrn >> (I+1) = 101; e.g., Ntrn ~1000 to 3000

> and the 2-layersnetwork have 20 hidden nodes in the hiddenlayersand 60 output
> nodes

What are the hidden and output transfer functions?

A NN with 1 hidden laye rwith H nodes is a universal approximator
provided H is
sufficiently large. Since there is no known analytical expression for
determining the
optimal number of hidden nodes, it is usually chosen by trial and
error.

How was H = 20 determined? Are you sure it is sufficiently large?

If H for one hidden layer is sufficiently large, the total number of
hidden nodes
can be reduced by adding more hidden layers. However, there is no
known analytical
expression for determining H1 or H2. The best bet is to use a single
hidden layer
unless there is a priori information on what H1 or H2 should be. Then
the other can be
found by trial and error.

For a single hidden layer with H=20 nodes, you will have

Nw = (I+1)*H+(H+1)*O = O+(I+O+1)*H = 60+(161)*20 = 3280 unknown
weights.

Therefore it is desrable to have

Ntrn >> Nw/O = 3220/60 ~ 55, e.g., Ntrn ~ 550 to 165

.
> But I found that the accuracy in the 1-layer network is higher than that in the 2-layer. > I expected that the 2-layer may have higher accuracy because there are more parameters,

WRONG. Your single layer has twice as many weights.!

> so I am wondering whether there are bugs in my implementation. But I can't find any > bugs up to now.

How about the choice H = 20?
Again, what is Ntrn?

> So can anyone tell me if that is my wrong expectation or other reasons so I have this > result? And what is the relationship between the numbers oflayersand the accuracy? > How should I judge the optimal numbers oflayersfor neural networks?

1 hidden layer is usually sufficient. Sometime it is not necessary.
I always design a linear model before designing a NN with a
single hidden layer.

See my post on Pretraining Advice
See the comp.ai.neural-nets FAQ
Search in comp.ai.neural-nets and comp.soft-sys.matlab
using "number of layers"

Hope this helps.

Greg