Documentation |
One-step secant backpropagation
net.trainFcn = 'trainoss'
[net,tr] = train(net,...)
trainoss is a network training function that updates weight and bias values according to the one-step secant method.
net.trainFcn = 'trainoss' sets the network trainFcn property.
[net,tr] = train(net,...) trains the network with trainoss.
Training occurs according to trainoss training parameters, shown here with their default values:
net.trainParam.epochs | 1000 | Maximum number of epochs to train |
net.trainParam.goal | 0 | Performance goal |
net.trainParam.max_fail | 6 | Maximum validation failures |
net.trainParam.min_grad | 1e-10 | Minimum performance gradient |
net.trainParam.searchFcn | 'srchbac' | Name of line search routine to use |
net.trainParam.show | 25 | Epochs between displays (NaN for no displays) |
net.trainParam.showCommandLine | 0 | Generate command-line output |
net.trainParam.showWindow | 1 | Show training GUI |
net.trainParam.time | inf | Maximum time to train in seconds |
Parameters related to line search methods (not all used for all methods):
net.trainParam.scal_tol | 20 | Divide into delta to determine tolerance for linear search. |
net.trainParam.alpha | 0.001 | Scale factor that determines sufficient reduction in perf |
net.trainParam.beta | 0.1 | Scale factor that determines sufficiently large step size |
net.trainParam.delta | 0.01 | Initial step size in interval location step |
net.trainParam.gama | 0.1 | Parameter to avoid small reductions in performance, usually set to 0.1 (see srch_cha) |
net.trainParam.low_lim | 0.1 | Lower limit on change in step size |
net.trainParam.up_lim | 0.5 | Upper limit on change in step size |
net.trainParam.maxstep | 100 | Maximum step length |
net.trainParam.minstep | 1.0e-6 | Minimum step length |
net.trainParam.bmax | 26 | Maximum step size |
You can create a standard network that uses trainoss with feedforwardnet or cascadeforwardnet. To prepare a custom network to be trained with trainoss:
In either case, calling train with the resulting network trains the network with trainoss.
Here a neural network is trained to predict median house prices.
[x,t] = house_dataset; net = feedforwardnet(10,'trainoss'); net = train(net,x,t); y = net(x)
Because the BFGS algorithm requires more storage and computation in each iteration than the conjugate gradient algorithms, there is need for a secant approximation with smaller storage and computation requirements. The one step secant (OSS) method is an attempt to bridge the gap between the conjugate gradient algorithms and the quasi-Newton (secant) algorithms. This algorithm does not store the complete Hessian matrix; it assumes that at each iteration, the previous Hessian was the identity matrix. This has the additional advantage that the new search direction can be calculated without computing a matrix inverse.
The one step secant method is described in [Batt92]. This algorithm requires less storage and computation per epoch than the BFGS algorithm. It requires slightly more storage and computation per epoch than the conjugate gradient algorithms. It can be considered a compromise between full quasi-Newton algorithms and conjugate gradient algorithms.