MATLAB Answers

LSTM time series hyperparameter optimization using bayesian optimization

163 views (last 30 days)
anurag kulshrestha
anurag kulshrestha on 22 Apr 2019
Commented: Jorge Calvo on 27 May 2021 at 13:45
I am working with time series regression problem. I want to optimize the hyperparamters of LSTM using bayesian optimization. I have 3 input variables and 1 output variable.
I want to optimize the number of hidden layers, number of hidden units, mini batch size, L2 regularization and initial learning rate . Code is given below:
numFeatures = 3;
numHiddenUnits = 120;
numResponses = 1;
layers = [ ...
sequenceInputLayer(numFeatures)
lstmLayer(numHiddenUnits,'OutputMode','sequence')
fullyConnectedLayer(numResponses)
regressionLayer];
options = trainingOptions('adam', ...
'MaxEpochs',maxEpochs, ...
'MiniBatchSize',miniBatchSize, ...
'InitialLearnRate',optVars.InitialLearnRate, ...
'Momentum',optVars.Momentum, ...
'GradientThreshold',1, ...
'Shuffle','never', ...
'L2Regularization',optVars.L2Regularization, ...
'Plots','training-progress',...
'Verbose',0);
net = trainNetwork(XTrain,YTrain,layers,options);
YPredicted = predict(net,Xval, 'MiniBatchSize',1);
valError = 1 - mean(YPredicted == Yval);
Thanks in advance.

Answers (2)

Don Mathis
Don Mathis on 10 May 2019
Here's an example using a convolutional network instead of an LSTM network. Your LSTM case should look very similar: https://www.mathworks.com/help/deeplearning/examples/deep-learning-using-bayesian-optimization.html
  1 Comment
Sinan Islam
Sinan Islam on 8 May 2021
LSTM is different from CNN. It is obvious that this example is in great demand. Why not Matlab make a proper example dedicated for optimizing LSTM?

Sign in to comment.


Jorge Calvo
Jorge Calvo on 27 May 2021 at 12:55
If you have R2020b or later, you can use the Experiment Manager app to run Bayesian optimization to determine the best combination of hyperparameters. For more information, see https://www.mathworks.com/help/deeplearning/ug/experiment-using-bayesian-optimization.html.
  2 Comments
Jorge Calvo
Jorge Calvo on 27 May 2021 at 13:45
Each time you run an experiment, the Experiment Manager will find the best combination of hyperparameters for a given setup. To specify what you mean by best, you can select from some standard objective metrics (including validation accuracy, which I think is what the original question was using) or you can define your own.
If you want to do find the best combo of hyperparameters for each of 200 data sets, then you would:
  1. Setup the experiment for the first data set.
  2. Run the experiment.
  3. Modify the setup function to load the next data set.
  4. Run the experiment again.
  5. Repeat steps 3 and 4.
This amounts to running 200 different experiments. On the bright side, unless your objective function depends on the data set, you would not need to recode it.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!