Training

Once the network weights and biases are initialized, the network is ready for training. The network can be trained for function approximation (nonlinear regression), pattern association, or pattern classification. The training process requires a set of examples of proper network behavior--network inputs p and target outputs t. During training the weights and biases of the network are iteratively adjusted to minimize the network performance function net.performFcn. The default performance function for feedforward networks is mean square error mse--the average squared error between the network outputs a and the target outputs t.

The remainder of this chapter describes several different training algorithms for feedforward networks. All these algorithms use the gradient of the performance function to determine how to adjust the weights to minimize performance. The gradient is determined using a technique called backpropagation, which involves performing computations backward through the network. The backpropagation computation is derived using the chain rule of calculus and is described in Chapter 11 of [HDB96].

The basic backpropagation training algorithm, in which the weights are moved in the direction of the negative gradient, is described in the next section. Later sections describe more complex algorithms that increase the speed of convergence.

Backpropagation Algorithm

There are many variations of the backpropagation algorithm, several of which are described in this chapter. The simplest implementation of backpropagation learning updates the network weights and biases in the direction in which the performance function decreases most rapidly, the negative of the gradient. One iteration of this algorithm can be written

• 

where is a vector of current weights and biases, is the current gradient, and is the learning rate.

There are two different ways in which this gradient descent algorithm can be implemented: incremental mode and batch mode. In incremental mode, the gradient is computed and the weights are updated after each input is applied to the network. In batch mode, all the inputs are applied to the network before the weights are updated. The next section describes the batch mode of training; incremental training is discussed in a later chapter.

Batch Training (train)

In batch mode the weights and biases of the network are updated only after the entire training set has been applied to the network. The gradients calculated at each training example are added together to determine the change in the weights and biases. For a discussion of batch training with the backpropagation algorithm, see page 12-7 of [HDB96].

Batch Gradient Descent (traingd)

The batch steepest descent training function is traingd. The weights and biases are updated in the direction of the negative gradient of the performance function. If you want to train a network using batch steepest descent, you should set the network trainFcn to traingd, and then call the function train. There is only one training function associated with a given network.

There are seven training parameters associated with traingd:

• epochs
• show
• goal
• time
• min_grad
• max_fail
• lr

The learning rate lr is multiplied times the negative of the gradient to determine the changes to the weights and biases. The larger the learning rate, the bigger the step. If the learning rate is made too large, the algorithm becomes unstable. If the learning rate is set too small, the algorithm takes a long time to converge. See page 12-8 of [HDB96] for a discussion of the choice of learning rate.

The training status is displayed for every show iterations of the algorithm. (If show is set to NaN, then the training status is never displayed.) The other parameters determine when the training stops. The training stops if the number of iterations exceeds epochs, if the performance function drops below goal, if the magnitude of the gradient is less than mingrad, or if the training time is longer than time seconds. max_fail, which is associated with the early stopping technique, is discussed in Improving Generalization.

The following code creates a training set of inputs p and targets t. For batch training, all the input vectors are placed in one matrix.

• p = [-1 -1 2 2; 0 5 0 5];
  t = [-1 -1 1 1];
  

Create the feedforward network.

• net = newff(p,t,3,{},'traingd');
  

In this simple example, turn off a feature that is introduced later.

• net.divideFcn = '';
  

At this point, you might want to modify some of the default training parameters.

• net.trainParam.show = 50;
  net.trainParam.lr = 0.05;
  net.trainParam.epochs = 300;
  net.trainParam.goal = 1e-5;
  

If you want to use the default training parameters, the preceding commands are not necessary.

Now you are ready to train the network.

• [net,tr]=train(net,p,t);
  

The training record tr contains information about the progress of training. An example of its use is given in Sample Training Session.

Now you can simulate the trained network to obtain its response to the inputs in the training set.

• a = sim(net,p)
  a =
     -1.0026   -0.9962   1.0010   0.9960
  

Try the Neural Network Design demonstration nnd12sd1[HDB96] for an illustration of the performance of the batch gradient descent algorithm.

Gradient Descent with Momentum

In addition to traingd, there are three other variations of gradient descent.

Gradient descent with momentum, implemented by traingdm, allows a network to respond not only to the local gradient, but also to recent trends in the error surface. Acting like a lowpass filter, momentum allows the network to ignore small features in the error surface. Without momentum a network can get stuck in a shallow local minimum. With momentum a network can slide through such a minimum. See page 12-9 of [HDB96] for a discussion of momentum.

Gradient descent with momentum depends on two training parameters. The parameter lr indicates the learning rate, similar to the simple gradient descent. The parameter mc is the momentum constant that defines the amount of momentum. mc is set between 0 (no momentum) and values close to 1 (lots of momentum). A momentum constant of 1 results in a network that is completely insensitive to the local gradient and, therefore, does not learn properly.)

• p = [-1 -1 2 2; 0 5 0 5];
  t = [-1 -1 1 1];
  net = newff(p,t,3,{},'traingdm');
  net.trainParam.lr = 0.05;
  net.trainParam.mc = 0.9;
  net = train(net,p,t);
  y = sim(net,p)
  

Try the Neural Network Design demonstration nnd12mo [HDB96] for an illustration of the performance of the batch momentum algorithm.

Provide feedback about this page

Simulation (sim)

Faster Training

© 1984-2009- The MathWorks, Inc.    -   Site Help   -   Patents   -   Trademarks   -   Privacy Policy   -   Preventing Piracy   -   RSS