MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Thread Subject: Error in Evaluation Criteria for Best Performance of the Neural Network

 Subject: Error in Evaluation Criteria for Best Performance of the Neural Network From: Subodh Paudel Date: 13 Mar, 2013 09:03:07 Message: 1 of 2 Dear All, I have find the optimum value of the hidden neurons, H=32, and then with Ntrials=5, i get the following result. How MATLAB neural network toolbox choose the best performance value.  Nepochs R2Train R2Validation R2Test MSETrain MSEValidation MSETest  16 0.90386 0.84352 0.67618 0.096081 0.11011 0.11351  27 0.91307 0.8324 0.77425 0.086873 0.11793 0.079135  19 0.90546 0.81643 0.77273 0.094487 0.12916 0.079667  13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566  25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734 With this result i need to check to check for MSE for the Validation to be minimum, and my best configuration should be, because i have MSEva 0.10809, which is minimum:   13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566 but when i check the performance from matlab with tr command, best_epoch: 25 best_perf: 0.085477 best_vperf: 0.10985 best_tperf: 0.074734 The matlab best performance is is best epoch 25. 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734 How matlab choose this value? How to judge the best performance of the network. Also, the R-statistics from the matlab is Rtrain=0.95629, if we square it =0.91449 Rvalidation=0.92598, if we square it =0.857438 Rtest=0.91665, if we square it =0.84024 These value are different. How the best performance of the neural network is evaluated. Thank you in advance for the answer.
 Subject: Error in Evaluation Criteria for Best Performance of the Neural Network From: Greg Heath Date: 14 Mar, 2013 22:56:18 Message: 2 of 2 "Subodh Paudel" wrote in message ... > Dear All, > > I have find the optimum value of the hidden neurons, H=32, and then with Ntrials=5, i get the following result. How MATLAB neural network toolbox choose the best performance value. > > Nepochs R2Train R2Validation R2Test MSETrain MSEValidation MSETest > 16 0.90386 0.84352 0.67618 0.096081 0.11011 0.11351 > 27 0.91307 0.8324 0.77425 0.086873 0.11793 0.079135 > 19 0.90546 0.81643 0.77273 0.094487 0.12916 0.079667 > 13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566 > 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734 1. The MSE tabulations are not very useful for understanding since the scale of the data is unknown. One could combine the R2 and MSE tabulations to obtain the more revealing NMSE tabulations. But why bother when NMSE and R2 only differ by unity? 2. The R2trn tabulation values aren't satisfying since comparing them with R2val and R2tst indicate that OVERFIT nets have been OVERTRAINED in spite of using validation stopping. 3. Obtain the standard deviations of MSEval and MSEtst (You only have 5 samples so you might want to check with the CHI-SQUARE formula from Wikipedia instrad of repeating with a higher value for Ntrials). Then combine and compare with mean(MSEval)-mean(MSEtst) to see if it is reasonable to assume that the val and tst data are good representatives of random draws from the same population of data, much less the same population from which the trn data was drawn. You may be the victim of a bad random trn/val/tst draw. Did you have the same random draw of data for all of the runs? 4. It is hard to believe that H=32 is optimal. The minute I saw Hopt = 32, I felt that something was wrong. What are I, O, N, Ntrn, Nval and Ntst? The limit for overfitting is the upper bound Hub = -1 + ceil( (Ntrn*O-O)/(I + O + 1)) Overfitting is OK as long as you don't overtrain. A good validation set and/or using MSE regularization should prevent that. help mse doc mse 5. What defaults did you override? 6. Forget the MSE tabulations but include the degree-of-freedom-adjusted training set R^2, R2trna. The comparisons of R2trna vs R2trn and R2trna vs R2val and R2tst are VERY important. 7. To answer your original question, the best estimate of performance on unseen population data is R2tst at the maximum of R2val:   R2tst = 0.779 when R2val = 0.846 If this is unacceptable consider what I have said and ask more questions. > With this result i need to check to check for MSE for the Validation to be minimum, and my best configuration should be, because i have MSEva 0.10809, which is minimum: > 13 0.91057 0.84639 0.77873 0.089376 0.10809 0.077566 > > but when i check the performance from matlab with tr command, > best_epoch: 25 > best_perf: 0.085477 > best_vperf: 0.10985 > best_tperf: 0.074734 > The matlab best performance is is best epoch 25. > 25 0.91447 0.84389 0.7868 0.085477 0.10985 0.074734 > > How matlab choose this value? How to judge the best performance of the network. That matlab best result is is only taking the last (5th) trial into consideration. The training structure results from previous trials is not available once you start a new trial.   > Also, the R-statistics from the matlab is > Rtrain=0.95629, if we square it =0.91449 > Rvalidation=0.92598, if we square it =0.857438 > Rtest=0.91665, if we square it =0.84024 > > These value are different. Yes. I don't know why. They might be using different values for the MSE00s. > How the best performance of the neural network is evaluated. Already answered. Hope this helps. > Thank you in advance for the answer. You are welcome Greg