LevenbergMarquardt backpropagation
net.trainFcn = 'trainlm'
[net,tr] = train(net,...)
trainlm
is a network training function that
updates weight and bias values according to LevenbergMarquardt optimization.
trainlm
is often the fastest backpropagation
algorithm in the toolbox, and is highly recommended as a firstchoice
supervised algorithm, although it does require more memory than other
algorithms.
net.trainFcn = 'trainlm'
sets the network trainFcn
property.
[net,tr] = train(net,...)
trains the network
with trainlm
.
Training occurs according to trainlm
training
parameters, shown here with their default values:
net.trainParam.epochs  1000  Maximum number of epochs to train 
net.trainParam.goal  0  Performance goal 
net.trainParam.max_fail  6  Maximum validation failures 
net.trainParam.min_grad  1e7  Minimum performance gradient 
net.trainParam.mu  0.001  Initial 
net.trainParam.mu_dec  0.1 

net.trainParam.mu_inc  10 

net.trainParam.mu_max  1e10  Maximum 
net.trainParam.show  25  Epochs between displays ( 
net.trainParam.showCommandLine  false  Generate commandline output 
net.trainParam.showWindow  true  Show training GUI 
net.trainParam.time  inf  Maximum time to train in seconds 
Validation vectors are used to stop training early if the network
performance on the validation vectors fails to improve or remains
the same for max_fail
epochs in a row. Test vectors
are used as a further check that the network is generalizing well,
but do not have any effect on training.
trainlm
is the default training function
for several network creation functions including newcf
, newdtdnn
, newff
,
and newnarx
.
You can create a standard network that uses trainlm
with feedforwardnet
or cascadeforwardnet
.
To prepare a custom network to be trained with trainlm
,
Set net.trainFcn
to 'trainlm'
.
This sets net.trainParam
to trainlm
's
default parameters.
Set net.trainParam
properties
to desired values.
In either case, calling train
with the resulting
network trains the network with trainlm
.
See help feedforwardnet
and help
cascadeforwardnet
for examples.
Here a neural network is trained to predict median house prices.
[x,t] = house_dataset; net = feedforwardnet(10,'trainlm'); net = train(net,x,t); y = net(x)
Like the quasiNewton methods, the LevenbergMarquardt algorithm was designed to approach secondorder training speed without having to compute the Hessian matrix. When the performance function has the form of a sum of squares (as is typical in training feedforward networks), then the Hessian matrix can be approximated as
H = J^{T}J
and the gradient can be computed as
g = J^{T}e
where J is the Jacobian matrix that contains first derivatives of the network errors with respect to the weights and biases, and e is a vector of network errors. The Jacobian matrix can be computed through a standard backpropagation technique (see [HaMe94]) that is much less complex than computing the Hessian matrix.
The LevenbergMarquardt algorithm uses this approximation to the Hessian matrix in the following Newtonlike update:
$${x}_{k+1}={x}_{k}{[{J}^{T}J+\mu I]}^{1}{J}^{T}e$$
When the scalar µ is zero, this is just Newton's method, using the approximate Hessian matrix. When µ is large, this becomes gradient descent with a small step size. Newton's method is faster and more accurate near an error minimum, so the aim is to shift toward Newton's method as quickly as possible. Thus, µ is decreased after each successful step (reduction in performance function) and is increased only when a tentative step would increase the performance function. In this way, the performance function is always reduced at each iteration of the algorithm.
The original description of the LevenbergMarquardt algorithm is given in [Marq63]. The application of LevenbergMarquardt to neural network training is described in [HaMe94] and starting on page 1219 of [HDB96]. This algorithm appears to be the fastest method for training moderatesized feedforward neural networks (up to several hundred weights). It also has an efficient implementation in MATLAB^{®} software, because the solution of the matrix equation is a builtin function, so its attributes become even more pronounced in a MATLAB environment.
Try the Neural Network Design demonstration nnd12m
[HDB96] for
an illustration of the performance of the batch LevenbergMarquardt
algorithm.
This function uses the Jacobian for calculations, which assumes
that performance is a mean or sum of squared errors. Therefore, networks
trained with this function must use either the mse
or sse
performance
function.