Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
DUPLICATING THE BEST NEURAL NET DESIGN

Subject: DUPLICATING THE BEST NEURAL NET DESIGN

From: Greg Heath

Date: 18 Mar, 2014 18:26:08

Message: 1 of 3

This post was inspired by

http://www.mathworks.in/matlabcentral/answers/121234-nn-accuracy-on-test-set-low

The goal is to duplicate the best of many candidate designs. Although the net is a classifier, I made my life a slight bit easier by using the test set MSE instead of the test set classification error.

clear all, close all, clc, plt=0;
 tic
[ x, t ] = simpleclass_dataset;
MSE00 = mean(var(t',1)) % 0.1872 Reference MSE
net = patternnet;
net.trainFcn='trainbr';
net.trainParam.goal = MSE00/100;
 rng(4151941)
 Ntrials=10
 ibest = 1
 msetstbest = inf
 for i = 1:Ntrials
   s = rng;
   net = configure(net,x,t);
   wb0 = getwb(net);
   [ net tr y e] = train(net,x,t); % e=t-y
   msetst(i) = tr.best_tperf;
   if tr.best_tperf < msetstbest
       msetstbest = tr.best_tperf;
       ibest = i
       sbest = s
       netbest = net;
       wb0best = wb0;
       trbest = tr;
       wbbest = getwb(net);
   end
 end
 NMSEtst = msetst/MSE00;
 plt = plt+1,figure(plt)
 plot(NMSEtst,'LineWidth',2)
 title(' Normalized Test Set MSE ' )
 xlabel( ' Trial ' )
 ylabel( ' NMSEtst ' )
 time = toc % ~9 sec
 
 % REPEAT THE BEST DESIGN
net2 = patternnet;
net2.trainFcn = 'trainbr';
net2.trainParam.goal = MSE00/100;
rng(sbest)
net2 = configure(net2,x,t);
wb02 = getwb(net2);
[ net2 tr2 y2 e2] = train(net2,x,t);
wb2 = getwb(net2);

isequal(wb02,wb0best) % yes
isequal(net2,netbest) % no
isequal(wb2,wbbest) % no
isequal(tr2,trbest) % no

% However
max(abs(wb2-wbbest)) % 8.5965e-011
max(abs(tr2.best_tperf - trbest.best_tperf)) %1.1662e-013

NOTE: Knowing the initial weights wb0 is not enough to duplicate designs because training functions randomly divide the data into trn/val/tst subsets. Therefore, you
either have to know the state of the random number generator or explicitly specify the trn/val/tst indices.

Hope this helps.

Greg
    

Subject: DUPLICATING THE BEST NEURAL NET DESIGN

From: Greg Heath

Date: 20 Mar, 2014 05:16:08

Message: 2 of 3

"Greg Heath" <heath@alumni.brown.edu> wrote in message <lga340$ha3$1@newscl01ah.mathworks.com>...
> This post was inspired by
>
> http://www.mathworks.in/matlabcentral/answers/121234-nn-accuracy-on-test-set-low
>
> The goal is to duplicate the best of many candidate designs. Although the net is a classifier, I made my life a slight bit easier by using the test set MSE instead of the test set classification error.
>
> clear all, close all, clc, plt=0;
> tic
> [ x, t ] = simpleclass_dataset;
> MSE00 = mean(var(t',1)) % 0.1872 Reference MSE
> net = patternnet;
> net.trainFcn='trainbr';

DELETE the above statement to use the default (TRAINSCG)
See explanation below

> net.trainParam.goal = MSE00/100;
> rng(4151941)
> Ntrials=10
> ibest = 1
> msetstbest = inf
> for i = 1:Ntrials
> s = rng;
> net = configure(net,x,t);
> wb0 = getwb(net);
> [ net tr y e] = train(net,x,t); % e=t-y
> msetst(i) = tr.best_tperf;
> if tr.best_tperf < msetstbest
> msetstbest = tr.best_tperf;
> ibest = i
> sbest = s
> netbest = net;
> wb0best = wb0;
> trbest = tr;
> wbbest = getwb(net);
> end
> end
> NMSEtst = msetst/MSE00;
> plt = plt+1,figure(plt)
> plot(NMSEtst,'LineWidth',2)
> title(' Normalized Test Set MSE ' )
> xlabel( ' Trial ' )
> ylabel( ' NMSEtst ' )
> time = toc % ~9 sec
>
> % REPEAT THE BEST DESIGN
> net2 = patternnet;
> net2.trainFcn = 'trainbr';
> net2.trainParam.goal = MSE00/100;
> rng(sbest)
> net2 = configure(net2,x,t);
> wb02 = getwb(net2);
> [ net2 tr2 y2 e2] = train(net2,x,t);
> wb2 = getwb(net2);
>
> isequal(wb02,wb0best) % yes
> isequal(net2,netbest) % no
> isequal(wb2,wbbest) % no
> isequal(tr2,trbest) % no
>
> % However
> max(abs(wb2-wbbest)) % 8.5965e-011
> max(abs(tr2.best_tperf - trbest.best_tperf)) %1.1662e-013
>
> NOTE: Knowing the initial weights wb0 is not enough to duplicate designs because training functions randomly divide the data into trn/val/tst subsets. Therefore, you
> either have to know the state of the random number generator or explicitly specify the trn/val/tst indices.

COMMENTS:

1. TRAINBR was used in the original (see the above link to ANSWERS).
2. TRAINBR is not recommended for classification. TRAINSCG is the patternnet default.
3. For classifiers, validation classification error rate should be used to select the "best" net. However, since the validation set is part of the design set (design = train + val), the unbiased estimate of the net performance on unseen data (GENERALIZATION) is provided by the test set classification error rate.
4. Using mse instead of pcterrr will probably yield a good classifier. However, now that you know better, you shouldn't do it.
5. For TRAINBR, perf ~= mse(e). INSTEAD, perf = beta*sse(e)+alpha*sse(wb), where alpha and beta are continually updated during training. Since perf is the minimization objective, I GUESS, that for regression, valperf should be the choice for choosing the best net. If used for classification, I would go with valpcterr to choose and tstpcterr for the
generalization estimate.
6. tr contains the training histories of two quantities tr.gamk and tr.ssX. I think alpha and beta can be obtained from them, but I don't understand that part of the source code yet.
Hope this helps.
 
Greg
     

Subject: DUPLICATING THE BEST NEURAL NET DESIGN

From: Greg Heath

Date: 23 Mar, 2014 20:59:08

Message: 3 of 3

REVISION

5. For TRAINBR, perf ~= mse(e). INSTEAD, perf = beta*sse(e)+alpha*sse(wb), where alpha and beta are continually updated during training. Since perf is the minimization objective, I GUESS, that for regression, valperf COULD be the choice for choosing the best net (mseval might be better) . If used for classification, I would go with valpcterr to choose and tstpcterr for the generalization estimate.

WHOOPS! Unfortunately, the older versions of trainbr do not allow a validation set.

Recent research has indicated that, based on sufficiently large test sets, using validation and regularization is, generally better.

Greg
      

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us