LSTM model predicts wrong values?

Hello, Im trying machine learning with LSTM and unsupervised learning.

Here is trained LSTM network, and I checked this model works.

numHiddenUnits = 100;

Layers = [ ...
           sequenceInputLayer(1, Normalization="zscore")
           lstmLayer(numHiddenUnits,'OutputMode','sequence')
           swishLayer
           fullyConnectedLayer(1)
           myRegressionLayer('mae')
          ];

options = trainingOptions('adam', ...
                    'MaxEpochs',40,...
                    'MiniBatchSize',32,...
                    'GradientThreshold',1,...
                    'InitialLearnRate',1e-2, ...
                    'Verbose',false, ...
                    'Plots', 'none');

%XTrain is (n*t sequence data)
%YTrain is (n*t sequence data)
[net, ~] = trainNetwork(XTrain, YTrain, Layers, options);

However, if the model predicts values with inputs in other environment,

%XTrain2 is (n*t sequence data in different environment from XTrain and YTrain environment for unsupervised learning)
dist = predict(net,  XTrain2);

only head of sequence of output is lower than others as shown below.

Here is input data, and it doesn’t seem that there are differents of value between head of sequence and other parts.

In comparison with true data, this head of sequence of output is wrong value, and other sequence parts are comparatively correct values.

I’m sorry for my poor English. Can anyone help me what to do? Thank you.

Answer 1

[Aniket]

Hi 玲,

The issue you are encountering is a common LSTM issue which can arise due to multiple factors. LSTMs can take some time to learn the context of a sequence. Since the ‘XTrain2’ is from a different environment, the model was not quite accurate while predicting the first value.

The following strategies can help resolve the issue:

1. Data Normalisation: The code provided uses “zscore” function for normalisation. Make sure that the data in ‘XTrain2’ is normalized in the same way as ‘XTrain’ beacuse differences in data scaling can lead to poor predictions, especially at the start of a sequence.

If ‘XTrain’ and ‘XTrain2’ have different distributions, the statistics (mean and standard deviation) from ‘XTrain’ should be used to normalize ‘XTrain2’.

You can also make use of Layer Normalisation and Batch Normalisation in MATLAB. Refer to the following documentation links for more details on these layers:

“LayerNormalizationLayer”: https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.layernormalizationlayer.html

“batchnorm”: https://www.mathworks.com/help/deeplearning/ref/dlarray.batchnorm.html

2. Model Regularization: Adding dropout layers or L2 regularization helps the model generalize better and reduce overfitting to the training data environment.

Refer to the following documentation links for more details on these layers:

“dropoutLayer”: https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.dropoutlayer.html

“setL2Factor”: https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.layer.setl2factor.html

I hope this helps resolve the issue.

[玲]

I think maybe normalisation causes this trouble.

I'll try, Thank you!