A question regarding stopping rules, local minimum vs early stopping

Dear all,

I am new to MATLAB and I have been working with neural networks. I know that there are several stopping criteria for the algorithms but more often than not it is the minimum MSE in the validation set that leads to termination. I am well aware of how important this is but for the purposes of comparison I would like the training to stop at a local minimum instead. Could you please tell me how I can accomplish that?

What I thought about doing afterwards is compare these two stopping rules in a test set and take note of the regression R^2 that is obtained. Of course the validation stopping rule is vastly superior in that respect but I would like to gather some evidence for that.

Thank you in advance.

Answer 1

[Greg Heath]

To see the network parameter settings, create the net without a final semicolon:

>> net = fitnet % no semicolon

The properties will be printed out in the command window. Note,in particular, the lines

 trainParam: .showWindow, .showCommandLine, .show, 
 .epochs, .time, .goal, .min_grad, .max_fail, .mu, 
 .mu_dec, .mu_inc, .mu_max

Now find out

 1. What are the maximum allowable number of epochs

 >> maxepochs = net.trainParam.epochs % no semicolon

       maxepochs =   1000

 2. How many continuous epochs of increasing validation error are 
required before validation stoping is implemeted:

 >> valstopepochs = net.trainParam.max_fail % no semicolon

       valstopepochs =    6

3. Therefore, all you have to to is set max_fail to a large number

 >> net.trainParam.max_fail = maxepochs

Now you still have the default 0.7/0.15/0.15 split for comparison with max_fail = 6. However, Since both the default random datadivision AND the random initial weight assignments could be different, you have to make sure the initial random number state before training is the same.

The trainng record tr will reveal all of the training details

 [ x, t ] = simplefit_data;
 net      = fitnet;
 rand('default')
 [ net tr y e ] = train(net,x,t);
 % y = net(x); e = t - y;

%=========================================================

     clear all, close all, clc, plt=0
     [ x , t ] = simplefit_dataset;
     [ I  N  ] = size(x)  % [ 1 94 ]
     [O N  ] = size(t) % [ 1 94 ]
     vart = var(t,1) %  8.3378 reference MSE
     plot(t)   % 4 local extrema  ==> H=4
     net1 = fitnet(4);
     net2 = net1;
     net2.trainParam.max_fail = 1000;

     rng('default')
     [ net1 tr1 y1 e1] = train(net1,x,t);
     stopcrit1 = tr1.stop       % Validation stop
     numepochs1 = tr1.num_epochs % 274
     NMSE1 = mse(e1)/vart  % 0.00033966
     hold on, plot(y1,'g.')

     rng('default')
     [ net2 tr2 y2 e2] = train(net2,x,t);
     stopcrit2 = tr2.stop % Maximum epoch reached
     numepochs2 = tr2.num_epochs % 1000
     NMSE2 = mse(e2)/vart % 0.00033145
     hold on, plot(y2,'r.')

     % Detailed Comparisons

     tr1 = tr1 , tr2 = tr2

Hope this helps.

Thank you for formally accepting my answer

Greg

[Greg Heath]

 epochratio = numepochs2/numepochs1  % 3.6496

 time1 = tr1.time(end)  % 2.64
 time2 = tr2.time(end)   % 8.49
 timeratio = time2/time1  % 3.1259