prune hidden neurons in neural network

I am trying to use 'prune' to prune the hidden layers in the following neural network. To start of with I define 1000 hidden neurons.

load('x1Ay1Ax2Fy2A.mat')

%For the Neural Network model:
BDataARtraining=[x1A y1A];%create a 3rd order data set by staggering observations
BDataARpredictor=[x2F]; %hold out to test forcast

dim=size(BDataARtraining,2); %number of variables of multivariate

model = newfit(BDataARtraining(1:end,1:dim-1)',BDataARtraining(1:end,dim)',100);%create a Feedforward Neural Network with 3 inputs, 1 output and 100 hidden neurons
view(model)
model2=prune(model);
view(model2)

Please could you explain why model2 still has 100 hidden nodes after pruning (see view(model2) ) as I was expecting some to have been removed by pruning? I have tried starting with different numbers of hidden neurons and also get the same number after pruning as what I started with.

Answer 1

[Greg Heath]

Search NEWSREADER and ANSWERS using

 Ntrials Hmin Hmax

for an easier approach to find the smallest successful number of hidden nodes in the range [Hmin Hmax] using Ntrials sets of random initial weights.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Peter Mills]

Hi Greg Are the references you suggest for a pruning or trial and improvement method? What would be the best places to look for how to use the pruning method? Many thanks for your answer Peter

[Greg Heath]

Neither. In general, both are impractical because you have to determine which weights to add or remove.

My technique is more practical:

 1. Given an acceptable maximum training error rate, the most stable 
solutions can be obtained by minimizing the number of hidden nodes.

 2. A good approach is to use a double DO-LOOP over 
    a. Number of hidden nodes 
    b. Trial subsets of random initial weights.

 3. The maximum trial value for H should not exceed the upper bound Hub 
which would cause the number of unknown weights to exceed the number of 
training equations and allow the dreaded phenomenon of OVERFITTING.

 4. It is up to the designer to choose 
     a. 0 <= Hmin 
     b. Hmax <= Hub
     c. dH

so that the search is either

 h = Hmin : dh : Hmax   %( dH > 0 )

or

 h = Hmax : dH : Hmin   % ( dH < 0 )

 5. The search doesn't have to be done in one fell swoop. I often use a 
primary search with dH large, followed by one or more searches with smaller 
dH.

 6. I have posted jillions of examples in both the NEWREADER and ANSWERS.

 7. A good set of search terms would be 

       Hmax Ntrials

Hope this helps.

Greg

[Peter Mills]

Thanks for the info I now have a model that loops through hidden layers. I what to change both the number of hidden layers and the number of hidden nodes in each layer. So say I try 1 to 10 layers and 1 to 4 hidden nodes in each layer that gives 10!/(4!(10−4)!)= 210 combinations. For each combination I am running the code 30 times to account for getting a different answer each time. How can I make this code more efficient? Ok using a larger dH to start with would help but how do I optimise both the number of layers and number of hidden nodes at the same time without making my code very slow?

[Greg Heath]

1. Solutions of equations having more unknowns than the number of equations are obviously unstable.

2. A net with one hidden layer is a universal approximator.

3. Reasons for using more than 1 hidden layer

   a. Reduce the total number of weights
   b. Additional information about the structure 
of the transformation is known (e.g., existence of 
classification subclasses).

In order to optimize both number of layers and number of nodes you have to have another constraint.

What is yours?

Hope this helps.

Greg

PS

>> help prune --- help for network/prune ---

 PRUNE: Delete neural inputs, layers and outputs with sizes
 of ZERO!.