Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Error in Evaluation Criteria for Best Performance of the Neural Network

Subject: Error in Evaluation Criteria for Best Performance of the Neural Network

From: Subodh Paudel

Date: 13 Mar, 2013 09:03:07

Message: 1 of 2

Dear All,

I have find the optimum value of the hidden neurons, H=32, and then with Ntrials=5, i get the following result. How MATLAB neural network toolbox choose the best performance value.

 Nepochs R2Train R2Validation R2Test MSETrain MSEValidation MSETest
 16 0.90386 0.84352 0.67618 0.096081 0.11011 0.11351
 27 0.91307 0.8324 0.77425 0.086873 0.11793 0.079135
 19 0.90546 0.81643 0.77273 0.094487 0.12916 0.079667
 13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566
 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734

With this result i need to check to check for MSE for the Validation to be minimum, and my best configuration should be, because i have MSEva 0.10809, which is minimum:
  13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566

but when i check the performance from matlab with tr command,
best_epoch: 25
best_perf: 0.085477
best_vperf: 0.10985
best_tperf: 0.074734
The matlab best performance is is best epoch 25.
25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734

How matlab choose this value? How to judge the best performance of the network.

Also, the R-statistics from the matlab is
Rtrain=0.95629, if we square it =0.91449
Rvalidation=0.92598, if we square it =0.857438
Rtest=0.91665, if we square it =0.84024

These value are different. How the best performance of the neural network is evaluated.

Thank you in advance for the answer.

Subject: Error in Evaluation Criteria for Best Performance of the Neural Network

From: Greg Heath

Date: 14 Mar, 2013 22:56:18

Message: 2 of 2

"Subodh Paudel" <subodhpaudel@gmail.com> wrote in message <khpfcb$8p9$1@newscl01ah.mathworks.com>...
> Dear All,
>
> I have find the optimum value of the hidden neurons, H=32, and then with Ntrials=5, i get the following result. How MATLAB neural network toolbox choose the best performance value.
>
> Nepochs R2Train R2Validation R2Test MSETrain MSEValidation MSETest
> 16 0.90386 0.84352 0.67618 0.096081 0.11011 0.11351
> 27 0.91307 0.8324 0.77425 0.086873 0.11793 0.079135
> 19 0.90546 0.81643 0.77273 0.094487 0.12916 0.079667
> 13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566
> 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734

1. The MSE tabulations are not very useful for understanding since the scale of the data is unknown. One could combine the R2 and MSE tabulations to obtain the more revealing NMSE tabulations. But why bother when NMSE and R2 only differ by unity?

2. The R2trn tabulation values aren't satisfying since comparing them with R2val and R2tst indicate that OVERFIT nets have been OVERTRAINED in spite of using validation stopping.

3. Obtain the standard deviations of MSEval and MSEtst (You only have 5 samples so you might want to check with the CHI-SQUARE formula from Wikipedia instrad of repeating with a higher value for Ntrials). Then combine and compare with mean(MSEval)-mean(MSEtst) to see if it is reasonable to assume that the val and tst data are good representatives of random draws from the same population of data, much less the same population from which the trn data was drawn.

You may be the victim of a bad random trn/val/tst draw.

Did you have the same random draw of data for all of the runs?

4. It is hard to believe that H=32 is optimal. The minute I saw Hopt = 32, I felt that something was wrong. What are I, O, N, Ntrn, Nval and Ntst? The limit for overfitting is
the upper bound

Hub = -1 + ceil( (Ntrn*O-O)/(I + O + 1))

Overfitting is OK as long as you don't overtrain. A good validation set and/or using
MSE regularization should prevent that.

help mse
doc mse

5. What defaults did you override?

6. Forget the MSE tabulations but include the degree-of-freedom-adjusted training set R^2, R2trna. The comparisons of R2trna vs R2trn and R2trna vs R2val and R2tst are
VERY important.

7. To answer your original question, the best estimate of performance on unseen population data is R2tst at the maximum of R2val:

  R2tst = 0.779 when R2val = 0.846

If this is unacceptable consider what I have said and ask more questions.

> With this result i need to check to check for MSE for the Validation to be minimum, and my best configuration should be, because i have MSEva 0.10809, which is minimum:
> 13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566
>
> but when i check the performance from matlab with tr command,
> best_epoch: 25
> best_perf: 0.085477
> best_vperf: 0.10985
> best_tperf: 0.074734
> The matlab best performance is is best epoch 25.
> 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734
>
> How matlab choose this value? How to judge the best performance of the network.

That matlab best result is is only taking the last (5th) trial into consideration. The training structure results from previous trials is not available once you start a new trial.
 
> Also, the R-statistics from the matlab is
> Rtrain=0.95629, if we square it =0.91449
> Rvalidation=0.92598, if we square it =0.857438
> Rtest=0.91665, if we square it =0.84024
>
> These value are different.

Yes. I don't know why. They might be using different values for the MSE00s.

> How the best performance of the neural network is evaluated.

Already answered.

Hope this helps.

> Thank you in advance for the answer.

You are welcome

Greg

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us