Replicating fitnet training performance with trainNetwork
3 views (last 30 days)
Show older comments
I am trying to design my own deep neural hidden network to approximate a functional relationship between an input vector and one target variable but I fail to obtain good results when designing my own deep neural network and I would like to understand the reason. The fit is pretty much perfect when I use the fitnet & train functions. To narrow down my problem, I brought it down to a one input, one target problem. The total number of samples is 2000.
The actual relationship looks as follows (input and output variables are already mapped to [-1,1]):
Fitnet / train
Using fitnet, I achieve almost arbitrary accuracy using the following options to create a NN with two hidden layers with ten nodes each:
trainFcn = 'trainlm';
hiddenLayerSize = [10 10];
net = fitnet(hiddenLayerSize,trainFcn);
net.trainParam.min_grad=1e-10;
net.trainParam.max_fail = 500;
net.trainParam.epochs=5000;
net.divideFcn = 'dividerand';
net.divideMode = 'sample';
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
net.performFcn = 'mse';
[net,tr] = train(net,x,t, 'useparallel', 'yes');
In this case, the training only takes a couple of seconds and the resultant fit is pretty much perfect:
trainNetwork
However, I fail to obtain a good fit when designing my own deep neural network (I am using an imageInputLayer as suggested in another post)
layers = [imageInputLayer([1 1 1]);
fullyConnectedLayer(10)
tanhLayer
fullyConnectedLayer(10)
tanhLayer
fullyConnectedLayer(1);
regressionLayer];
options = trainingOptions('sgdm', 'maxepochs', 10000, 'minibatchsize', 2000, 'InitialLearnrate', 0.1,'Plots','training-progress');
XNew = reshape(x, [1,1,size(x,1),size(x,2)]); % size(XNew) = [1, 1, 1, 2000]
trainedNet = trainNetwork(XNew, t', layers, options);
I tried playing around with the Initial learning rate (from 0.1 to 1e-5), the minibatchsize (from 10 to all samples), the solver and the activation functions but after a fast decrease the RMSE always gets stuck around 0.1:
and this is the fit I get:
Am I doing anything fundametally wrong or does anyone have tips on how to improve the learning?
Thanks a lot in advance!
3 Comments
Yifan Wei
on 22 Jun 2021
I have faced same issue here. did you get a chance to look at why this happened?
ytzhak goussha
on 30 Jun 2021
You should try to run this without normalization in the imageInputLayer since you already scaled it [-1 1].
The defult is normalization be zero centering by subtracting the mean, the mean is a learnable parameter.
Try:
imageInputLayer([1 1 1],'Name','input','Normalization',"none")
Answers (0)
See Also
Categories
Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!