Why Training Set accuracy decrease dramatically after stopping the trainNetwork?

8 views (last 30 days)
After stopping manually trainNetworktrainNetwork, the validation error dropped dramatically:
I tested the Training Set accuracy, and got also about 60%:
predY = classify(net,xTrain);
Any ideas what I'am doing wrong?
Sergy Stepura
Sergy Stepura on 1 Feb 2019
Edited: Sergy Stepura on 4 Feb 2019
The network has simple architecture, 5 fully connected layers with batch normalization + Input layer + Output layer (softmax):
1 '' Image Input 120x1x4 images with 'zerocenter' normalization
2 '' Fully Connected 65 fully connected layer
3 '' Batch Normalization Batch normalization
4 '' ReLU ReLU
5 '' Fully Connected 65 fully connected layer
6 '' Batch Normalization Batch normalization
7 '' ReLU ReLU
8 '' Fully Connected 65 fully connected layer
9 '' Batch Normalization Batch normalization
10 '' ReLU ReLU
11 '' Fully Connected 65 fully connected layer
12 '' Batch Normalization Batch normalization
13 '' ReLU ReLU
14 '' Fully Connected 65 fully connected layer
15 '' Batch Normalization Batch normalization
16 '' ReLU ReLU
17 '' Fully Connected 3 fully connected layer
18 '' Softmax softmax
19 '' Classification Output crossentropyex

Sign in to comment.

Answers (1)

Don Mathis
Don Mathis on 8 Feb 2019
Maybe your minibatch size is too small. The accuracy drop may be due to batchnormalization layers getting finalized, during which time the mean and variance of the incoming activations of each batchnorm layer are computed using the whole training set. If those full-batch statistics don't match the minibatch statistics very well, the finalized batchnorm layers will not be performing a very good normalization.
Don Mathis
Don Mathis on 11 Feb 2019
Also: Why does your plot show "Iterations per epoch: 1"? Were you using miniBatchSize=30000 in that run?
What are you passing to trainingOptions()?

Sign in to comment.


Find more on Deep Learning with Time Series and Sequence Data in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!