"Greg Heath" <heath@alumni.brown.edu> wrote in message <lga340$ha3$1@newscl01ah.mathworks.com>...
> This post was inspired by
>
> http://www.mathworks.in/matlabcentral/answers/121234nnaccuracyontestsetlow
>
> The goal is to duplicate the best of many candidate designs. Although the net is a classifier, I made my life a slight bit easier by using the test set MSE instead of the test set classification error.
>
> clear all, close all, clc, plt=0;
> tic
> [ x, t ] = simpleclass_dataset;
> MSE00 = mean(var(t',1)) % 0.1872 Reference MSE
> net = patternnet;
> net.trainFcn='trainbr';
DELETE the above statement to use the default (TRAINSCG)
See explanation below
> net.trainParam.goal = MSE00/100;
> rng(4151941)
> Ntrials=10
> ibest = 1
> msetstbest = inf
> for i = 1:Ntrials
> s = rng;
> net = configure(net,x,t);
> wb0 = getwb(net);
> [ net tr y e] = train(net,x,t); % e=ty
> msetst(i) = tr.best_tperf;
> if tr.best_tperf < msetstbest
> msetstbest = tr.best_tperf;
> ibest = i
> sbest = s
> netbest = net;
> wb0best = wb0;
> trbest = tr;
> wbbest = getwb(net);
> end
> end
> NMSEtst = msetst/MSE00;
> plt = plt+1,figure(plt)
> plot(NMSEtst,'LineWidth',2)
> title(' Normalized Test Set MSE ' )
> xlabel( ' Trial ' )
> ylabel( ' NMSEtst ' )
> time = toc % ~9 sec
>
> % REPEAT THE BEST DESIGN
> net2 = patternnet;
> net2.trainFcn = 'trainbr';
> net2.trainParam.goal = MSE00/100;
> rng(sbest)
> net2 = configure(net2,x,t);
> wb02 = getwb(net2);
> [ net2 tr2 y2 e2] = train(net2,x,t);
> wb2 = getwb(net2);
>
> isequal(wb02,wb0best) % yes
> isequal(net2,netbest) % no
> isequal(wb2,wbbest) % no
> isequal(tr2,trbest) % no
>
> % However
> max(abs(wb2wbbest)) % 8.5965e011
> max(abs(tr2.best_tperf  trbest.best_tperf)) %1.1662e013
>
> NOTE: Knowing the initial weights wb0 is not enough to duplicate designs because training functions randomly divide the data into trn/val/tst subsets. Therefore, you
> either have to know the state of the random number generator or explicitly specify the trn/val/tst indices.
COMMENTS:
1. TRAINBR was used in the original (see the above link to ANSWERS).
2. TRAINBR is not recommended for classification. TRAINSCG is the patternnet default.
3. For classifiers, validation classification error rate should be used to select the "best" net. However, since the validation set is part of the design set (design = train + val), the unbiased estimate of the net performance on unseen data (GENERALIZATION) is provided by the test set classification error rate.
4. Using mse instead of pcterrr will probably yield a good classifier. However, now that you know better, you shouldn't do it.
5. For TRAINBR, perf ~= mse(e). INSTEAD, perf = beta*sse(e)+alpha*sse(wb), where alpha and beta are continually updated during training. Since perf is the minimization objective, I GUESS, that for regression, valperf should be the choice for choosing the best net. If used for classification, I would go with valpcterr to choose and tstpcterr for the
generalization estimate.
6. tr contains the training histories of two quantities tr.gamk and tr.ssX. I think alpha and beta can be obtained from them, but I don't understand that part of the source code yet.
Hope this helps.
Greg
