Training loss considerably lower than validation loss in regression network

So I succesfully transfer learnied Alexnet to classify an image on the precense of manhole covers or not. After that I used the convolutinoal layers from this network to train a regression network that determines the position of the manhole in the image. However during training the RMSE of the validation and training set are prety much the same but it is the validation loss that is considerable lower than the training loss. I used a custom loss function in my regression layer and training and validation are split 70/30.

I started looking on the internet what could cause this as you would expect the validation loss to be higher or similar as the training loss. On the one hand, somebody mentioned that this could be caused by the validation set being too easy compared to the training set. But even after reshuffeling the training and validation sets from a different random seed point the same thing happend. On the other hand, regularisation in the form of drop-out layers could cause this as they are activeated during training but not during validation. But this is also not the case as my modified alexnet network only consists of the convolutional, max pooling layers and a fully connected layer at the end.

Additionally I manually calculated the loss of the validation and training dataset after training. The validation loss was the same as the final validation loss during training. However, the training loss all of a sudden was lower than the last training loss during training and also lower than the validation loss. This is as expected but why is the loss not correctly computed during training. I don't think this is a problem with my custom loss function as previously I trained with the default loss function of matlab and saw the same thing.

Training loss during training = 0.4002

Training loss after training (manually computed) = 0.0421

Validation loss during training = 0.0720

Validation loss after training (manually computed) = 0.0720

[Greg Heath]

What happens when you repeat your work 9 more times with different random number seeds?

Greg

[lukas mattheuwsen]

Similar results, training validation loss is always considerable lower than training loss on the graph and the output in the command window. However when computer manually the training loss is actually lower than the validation loss. I was just wondering if somebody else had seen this before in their project and had an explanation. During testing the model works fine so that's not the problem.

[Roland Kruse]

Strangely, I have just the opposite problem: while the validation losses agree the training (mini-batch) loss printed during training is much lower than the one I manually calculate afterwards.