Why does the "trainbr" function not require a validation dataset?

I read that the "trainbr" function in MATLAB (Bayesian regularization back-propagation algorithm for neural network training) does not require a validation dataset, and that in the MATLAB implementation of this algorithm, the validation stops are disabled by default. Please kindly explain to me in a little detail why validation is not necessary for using this neural network training function.

Answer 1

[MathWorks Support Team]

The function "trainbr" that performs Bayesian regularization backpropogation disables validation stops by default. The reasoning for this is that validation is usually used as a form of regularization, but "trainbr" has its own form of validation built into the algorithm.

In other words, "trainbr" does not require a validation dataset because the point of checking validation is to see if the error on the validation set gets better or worse as training goes on. If the error gets worse, you stop training. But the Bayesian error is not just based on how well the model is performing on the dataset- it is also based on how large the weights are. The larger the weights, the higher the error. So throughout training, if the validation step is on, it may not ever let the network explore larger weights, even though larger weights may lead to the global minimum.

If we have a network called "net", this behavior of validation stops is controlled via the parameter 'net.trainParam.max_fail'. 'max_fail' denotes the maximum number of times that we allow the validation to improve or to not improve before terminating training. 

If we set 'max_fail' to 5, the training will terminate when we get 5 consecutive iterations where the validation performance does not improve. If we want to get the validation results without terminating the training, we can set 'max_fail' to a very large number like 10000 or inf (the default).

For further resources, please see the documentation page for "trainbr" or the papers it references:

https://www.mathworks.com/help/deeplearning/ref/trainbr.html

[1] MacKay, David J. C. "Bayesian interpolation." Neural computation. Vol. 4, No. 3, 1992, pp. 415–447.

[2] Foresee, F. Dan, and Martin T. Hagan. "Gauss-Newton approximation to Bayesian learning." Proceedings of the International Joint Conference on Neural Networks, June, 1997.

Answer 2

[Greg Heath]

OVERFITTING + OVERTRAINING combine to form an ugly monster that prevents nets from performing well on nontraining data. Training is so precise that performance is excellent on training data at the expense of performing poorly on nontraining data. Typically, training becomes long and the weights become very large in order to obtain such precision.

Since the net is usually designed to work well on unseen data, several techniques have been invented to prevent overtraining an overfit net. These techniques fall under the term

          GENERALIZATION

 Minimize the training error subject to one of the 
following constraints:

1. NONOVERFITTING:

Minimize the number of weights used.

2. VALIDATION STOPPING (AKA EARLY STOPPING) MATLAB DEFAULT:

Minimize the training subset error until convergence OR the error on the validation subset starts to increase for a specified number of epochs.

3. BAYESIAN REGULARIZATION (MATLAB'S TRAINBR):

 Minimize the  

          sum of squared errors

and either

 a. the weighted sum of squared weights

or

 b.  the weighted sum of absolute weight  values

Hope this helps.

Thank you for formally accepting my answer

Greg