Path: news.mathworks.com!newsfeed-00.mathworks.com!newsfeed2.dallas1.level3.net!news.level3.com!postnews.google.com!g23g2000vbr.googlegroups.com!not-for-mail
From: Greg Heath <heath@alumni.brown.edu>
Newsgroups: comp.soft-sys.matlab
Subject: Re: Question about number of layers in ANN
Date: Thu, 29 Oct 2009 10:07:02 -0700 (PDT)
Organization: http://groups.google.com
Lines: 86
Message-ID: <0bd1a876-7ac3-487f-9972-1bf39d0f28d3@g23g2000vbr.googlegroups.com>
References: <hcbiur$5g7$1@fred.mathworks.com>
NNTP-Posting-Host: 69.141.163.135
Mime-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
X-Trace: posting.google.com 1256836023 21583 127.0.0.1 (29 Oct 2009 17:07:03 GMT)
X-Complaints-To: groups-abuse@google.com
NNTP-Posting-Date: Thu, 29 Oct 2009 17:07:03 +0000 (UTC)
Complaints-To: groups-abuse@google.com
Injection-Info: g23g2000vbr.googlegroups.com; posting-host=69.141.163.135; 
	posting-account=mUealwkAAACvQrLWvunjg50tRAnsNtJR
User-Agent: G2/1.0
X-HTTP-UserAgent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 
	2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729),gzip(gfe),gzip(gfe)
Xref: news.mathworks.com comp.soft-sys.matlab:581062


On Oct 29, 4:15 am, "Kloser " <kcfche...@gmail.com> wrote:
> Dear All
> I want to know, in Neural Networks, whether morelayerswill promise higher accuracy
> for classification?

No. Just, perhaps, fewer weights.

> I have tried using two networks to classify some images into classes. There are 100
>  variables in each sample, meaning that there are 100 input nodes. And for the
>  Networks, The 1-layer network  have 60 output nodes,

The 1-layer network is equivalent to a linear or logistic model.
What is the output transfer function?
How large is your training sample?

For I = 100 inputs,  and O=60 outputs you will have Nw = (I+1)*O =
6060 unknown
weights. With Ntrn training samples you will have Neq = Ntrn*O =
60*Ntrn equations.

To mitigate noise and measurement error it is desirable to have

Neq >> Nw

or

Ntrn >> (I+1) = 101; e.g., Ntrn ~1000 to 3000


> and the 2-layersnetwork have 20 hidden nodes in the hiddenlayersand 60 output
> nodes

What are the hidden and output transfer functions?

A NN with 1 hidden laye rwith H nodes is a universal approximator
provided H is
sufficiently large. Since there is no known analytical expression for
determining the
optimal number of hidden nodes, it is usually chosen by trial and
error.

How was H = 20 determined? Are you sure it is sufficiently large?

If H for one hidden layer is sufficiently large, the total number of
hidden nodes
can be reduced by adding more hidden layers. However, there is no
known analytical
expression for determining H1 or H2. The best bet is to use a single
hidden layer
unless there is a priori information on what H1 or H2 should be. Then
the other can be
found by trial and error.

For a single hidden layer with H=20 nodes, you will have

Nw = (I+1)*H+(H+1)*O = O+(I+O+1)*H = 60+(161)*20 = 3280 unknown
weights.

Therefore it is desrable to have

Ntrn >> Nw/O = 3220/60 ~ 55, e.g., Ntrn ~ 550 to 165

.
> But I found that the accuracy in the 1-layer network is higher than that in the 2-layer. > I expected   that the 2-layer may have higher accuracy because there are more  parameters,

WRONG. Your single layer has twice as many weights.!

> so I am wondering whether there are bugs in my implementation. But I can't find any > bugs up to now.

How about the choice H = 20?
Again, what is Ntrn?

> So can anyone tell me if that is my wrong expectation or other reasons so I have this > result? And what is the relationship between the numbers oflayersand the accuracy? > How should I judge the optimal numbers oflayersfor neural networks?

1 hidden layer is usually sufficient. Sometime it is not necessary.
I always design a linear model before designing a NN with a
single hidden layer.

See my post on Pretraining Advice
See the comp.ai.neural-nets FAQ
Search in comp.ai.neural-nets and comp.soft-sys.matlab
using "number of layers"

Hope this helps.

Greg