Assuming
N = Ndes + Ntst = Ntrn + Nval + Ntst
and an I-H-O node topology, the best results for hard RW problems depend on repeatedly using the design (training + validation) data to select satisfactory choices for
1. Data division ( e.g., Ntrn/N = 0.6 : 0.1 : 0.8, Nval = Ntst )
2. H ( e.g, H << (Ntrn-1)*O/(I+O+1) )
3. Initial weights (e.g., 10 random initializations for each above combo)
After the candidate design with the best validation performance is chosen, the test data is used ONCE AND ONLY ONCE to obtain an UNBIASED prediction of net peformance on future nondesign data.
The test set prediction is the measure of accuracy. The estimated precision of the accuracy estimate is usually obtained from the statistical formula for standard deviation assuming the errors are Gaussian (Regression) or Binomially (Classification) distributed.
If the performance on the test set is unsatisfactory, the data should be randomly redivided to obtain a new test set and the design process should be repeated.
In your case, the test set performance could be considered much worse than the design set errors. This could have happened by chance via the random data division or be an indication that one or more design parameters needs to be tweaked. However, since you have already used up your allowance of test set estimates, you'll have to redivide, to obtain a new test set and redesign.
Hope this helps.
Greg
