How to choose the appropriate trained NN?
1 view (last 30 days)
Show older comments
Hi!
I have divided my data into trn/tst/val sets. The NN gives different classification accuracy at every training session.Should I choose my model simply based on the highest test set accuracy or should I average the test accuracy over several runs?
Thanks in advance.
0 Comments
Accepted Answer
Greg Heath
on 21 Dec 2014
Doesn't make sense to average error rates in order to choose the best design.
Obtain multiple designs(e.g., ~100: ~10 for each of 10 choices for number of hidden nodes)
Rank them via the degree-of-freedom adjusted training set error and validation set error
Estimate generalization non-design error via the test set error
I tend to choose the smallest successful net by just looking at the three tabulations and plots of error vs # of hidden nodes.
Hope this helps
Thank you for formally accepting my answer
Greg
3 Comments
Greg Heath
on 22 Dec 2014
Different people have different definitions.
I associate model selection with topology, number of layers, choice of transfer functions, ...
I associate parameter selection with a selection of values consistent with topology and optimization of a performance function.
If you type the command (without the ending semicolon)
net = patternnet
you will see the list of defaults chosen for both.
To begin with, run the help and doc examples. Model and parameter selection are automatically chosen by default. If performance is unsatisfactory just run the example Ntrials = 10 or more times to mitigate the random choice of initial weights and trn/val/tst data division. If that fails, increase the number of hidden nodes obtaining Ntrials = 10 or more designs for each candidate value of number of hidden nodes.
For a structured approach to this technique, search including the search words
greg Ntrials.
Hope this helps.
Thank you for formally accepting my answer
Greg
More Answers (0)
See Also
Products
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!