Neural Network Toolbox™

Levenberg-Marquardt backpropagation

Syntax

• [net,TR] = trainlm(net,TR,trainV,valV,testV)
  info = trainlm('info')
  

Description

trainlm is a network training function that updates weight and bias values according to Levenberg-Marquardt optimization.

trainlm is often the fastest backpropagation algorithm in the toolbox, and is highly recommended as a first-choice supervised algorithm, although it does require more memory than other algorithms.

trainlm(net,TR,trainV,valV,testV) takes these inputs,

net	Neural network
TR	Initial training record created by train
trainV	Training data created by train
valV	Validation data created by train
testV	Test data created by train

and returns

net	Trained network
TR	Training record of various values over each epoch

Each argument trainV, valV, and testV is a structure of these fields:

X	N x TS cell array of inputs for N inputs and TS time steps. X{i,ts} is an Ri x Q matrix for the ith input and TS time step.
Xi	N x Nid cell array of input delay states for N inputs and Nid delays. Xi{i,j} is an Ri x Q matrix for the ith input and jth state.
Pd	N x S x Nid cell array of delayed input states.
T	NoxTS cell array of targets for No outputs and TS time steps. T{i,ts} is an Si x Q matrix for the ith output and TS time step.
Tl	Nl x TS cell array of targets for Nl layers and TS time steps. Tl{i,ts} is an Si x Q matrix for the ith layer and TS time step.
Ai	Nl x TS cell array of layer delays states for Nl layers, TS time steps. Ai{i,j} is an Si x Q matrix of delayed outputs for layer i, delay j.

Training occurs according to trainlm's training parameters, shown here with their default values:

net.trainParam.epochs	100		Maximum number of epochs to train
net.trainParam.goal	0		Performance goal
net.trainParam.max_fail	5		Maximum validation failures
net.trainParam.mem_reduc	1		Factor to use for memory/speed tradeoff
net.trainParam.min_grad	1e-10		Minimum performance gradient
net.trainParam.mu	0.001		Initial mu
net.trainParam.mu_dec	0.1		mu decrease factor
net.trainParam.mu_inc	10		mu increase factor
net.trainParam.mu_max	1e10		Maximum mu
net.trainParam.show	25		Epochs between displays (NaN for no displays)
net.trainParam.showCommandLine	0		Generate command-line output
net.trainParam.showWindow	1		Show training GUI
net.trainParam.time	inf		Maximum time to train in seconds

Validation vectors are used to stop training early if the network performance on the validation vectors fails to improve or remains the same for max_fail epochs in a row. Test vectors are used as a further check that the network is generalizing well, but do not have any effect on training.

trainlm is the default training function for several network creationfunctions including newff, newcf, newtd, newdtdnn, and newnarx.

trainlm('info') returns useful information about this function.

Network Use

You can create a standard network that uses trainlm with newff, newcf, or newelm.

To prepare a custom network to be trained with trainlm,

1. Set net.trainFcn to 'trainlm'. This sets net.trainParam to trainlm's default parameters.
2. Set net.trainParam properties to desired values.

In either case, calling train with the resulting network trains the network with trainlm.

See newff, newcf, and newelm for examples.

Algorithm

trainlm supports training with validation and test vectors if the network's NET.divideFcn property is set to a data division function. Validation vectors are used to stop training early if the network performance on the validation vectors fails to improve or remains the same for max_fail epochs in a row. Test vectors are used as a further check that the network is generalizing well, but do not have any effect on training.

trainlm can train any network as long as its weight, net input, and transfer functions have derivative functions.

Backpropagation is used to calculate the Jacobian jX of performance perf with respect to the weight and bias variables X. Each variable is adjusted according to Levenberg-Marquardt,

• jj = jX * jX
  je = jX * E
  dX = -(jj+I*mu) \ je
  

where E is all errors and I is the identity matrix.

The adaptive value mu is increased by mu_inc until the change above results in a reduced performance value. The change is then made to the network and mu is decreased by mu_dec.

The parameter mem_reduc indicates how to use memory and speed to calculate the Jacobian jX. If mem_reduc is 1, then trainlm runs the fastest, but can require a lot of memory. Increasing mem_reduc to 2 cuts some of the memory required by a factor of two, but slows trainlm somewhat. Higher states continue to decrease the amount of memory needed and increase training times.

Training stops when any of these conditions occurs:

• The maximum number of epochs (repetitions) is reached.
• The maximum amount of time is exceeded.
• Performance is minimized to the goal.
• The performance gradient falls below min_grad.
• mu exceeds mu_max.
• Validation performance has increased more than max_fail times since the last time it decreased (when using validation).

See Also

newff, newcf, newtd, newdtdnn, newnarx, and template_train

Provide feedback about this page

traingdx

trainoss

© 1984-2008- The MathWorks, Inc.    -   Site Help   -   Patents   -   Trademarks   -   Privacy Policy   -   Preventing Piracy   -   RSS