BayesOpt for Custom Neural Network?

18 views (last 30 days)
I'm trying to use Bayesian Optimization for my custom neural network, but when following the tutorials, it isn't clear how I can use BayesOpt with my own network.
My current understanding is I need to use the Validation and Training losses as inputs to the objective function, but I'm sort of at a loss on how to do this.
What I'm Trying to Tune:
  • Hyperparameters (max epochs (just to get in the neighborhood), minibatchsize, initial learning rates, etc.)
  • Number of hidden layers
  • The size of my fully connected layers
Right now I'm doing this iteratively, but I'm looking for a more optimal solution (hence BayesOpt)
This is what I have setup currently in order to iterate over different epochs, hidden layers, and the size of fully connected layers. I know I can add in the other hyperparameters to trainingOptions, but this is all I'm iterating over at the moment, and am leaving the rest to default values.
Thoughts?
function [net,tr] = betNet(X,y,X_test,y_test,X_cv,y_cv,maxE,NHL,fcls)
%fcls = Fully Connected Layer Size
%NHL = Number of Hidden Layers
%maxE = Maximum Epochs
%% ===== Setting up DNN =====
%Sets up our FCL
fcl1 = fullyConnectedLayer(fcls,'BiasInitializer','narrow-normal');
fcl2 = fullyConnectedLayer(2,'BiasInitializer','ones');
ip = sequenceInputLayer(size(X,1));
sml = softmaxLayer('Name','sml');
options = trainingOptions('adam',...
'MaxEpochs',maxE,...
'ExecutionEnvironment','gpu',...
'Shuffle','every-epoch',...
'MiniBatchSize',64,...
'ValidationFrequency',50,...
'ValidationData',{X_cv,y_cv})
% 'Plots','training-progress',...
% layers = [repmat(fcl,1,10) sigmoidLayer classificationLayer]
layers = [ip repmat(fcl1,1,NHL) fcl2 softmaxLayer classificationLayer];
%% ===== Training NN =====
[net,tr] = trainNetwork(X,y,layers,options);
end

Accepted Answer

Grant Anderson
Grant Anderson on 18 Jun 2020
I believe I solved it. I'm letting it run and will find out when it's finished.
% bayesNet
% (X,y,X_test,y_test,X_cv,y_cv,maxE,NHL,fcls)
X = Training_Data.X;
Y = Training_Data.y;
X_cv = Training_Data.X_cv;
Y_cv = Training_Data.y_cv;
%%Choose Variables to Optimize
%MBS = MiniBatchSize
mbsRange = [10 120];
%maxE = maximum epochs
maxErange = [100 10000];
%NHL = number of hidden layers
nhlRange = [5 200];
%fcls = Fully Connected Layer Size
fclsRange = [100 size(Training_Data.X,1)];
optimVars = [
optimizableVariable('MiniBatchSize',mbsRange,'Type','integer');
optimizableVariable('maxEpochs',maxErange,'Type','integer');
optimizableVariable('NHL',nhlRange,'Type','integer');
optimizableVariable('fcls',fclsRange,'Type','integer');
optimizableVariable('InitialLearnRate',[1e-2 1],'Transform','log')];
%%Perform Bayesian Optimization
ObjFcn = makeObjFcn(Training_Data.X,Training_Data.y,Training_Data.X_cv,Training_Data.y_cv);
BayesObject = bayesopt(ObjFcn,optimVars,...
'MaxObj',30,...
'MaxTime',8*60*60,...
'IsObjectiveDeterministic',false,...
'UseParallel',false);
%%Evaluate Final Network
bestIdx = BayesObject.IndexOfMinimumTrace(end);
fileName = BayesObject.UserDataTrace{bestIdx};
load(fileName);
[Y_predicted,scores] = classify(net,test);
testError = perform(net,test_truth_cat,Y_predicted);
testError
valError
%%Objective Function for Optimization
function ObjFcn = makeObjFcn(X,Y,X_cv,Y_cv)
ObjFcn = @valErrorFun;
function [valError,cons,fileName] = valErrorFun(optVars)
%Imports our current optVars for layer construction
NHL = optVars.NHL
fcls = optVars.fcls
%% ===== Setting up DNN =====
%Sets up our FCL
fcl1 = fullyConnectedLayer(fcls,'BiasInitializer','narrow-normal');
fcl2 = fullyConnectedLayer(2,'BiasInitializer','ones');
%lstm = lstmLayer(size(X,1))
% ip = sequenceInputLayer(size(X,1),'Normalization','zerocenter');
ip = sequenceInputLayer(size(X,1));
sml = softmaxLayer('Name','sml');
options = trainingOptions('adam',...
'MaxEpochs',optVars.maxEpochs,...
'ExecutionEnvironment','gpu',...
'Shuffle','every-epoch',...
'MiniBatchSize',optVars.MiniBatchSize,...
'InitialLearnRate',optVars.InitialLearnRate,...
'ValidationFrequency',50,...
'ValidationData',{X_cv,Y_cv})
% 'Plots','training-progress',...
% layers = [repmat(fcl,1,10) sigmoidLayer classificationLayer]
layers = [ip repmat(fcl1,1,NHL) fcl2 softmaxLayer classificationLayer];
%
% global training_state
% training_state = [];
%% ===== Training NN =====
[net,~] = trainNetwork(X,Y,layers,options);
YPredicted = classify(net,test);
valError = perform(net,test_truth_cat,YPredicted);
fileName = num2str(valError)+".mat";
save(fileName,'net','valError');
cons = [];
end
end

More Answers (2)

KASIM Erdem
KASIM Erdem on 5 Nov 2020
Can you share all file ?

Jorge Calvo
Jorge Calvo on 5 Oct 2021

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!