How to reproduce exact results of a patternnet network using feedforwardnet network.

4 views (last 30 days)
I am trying to reproduce the results of a neural network created by "patternnet" function, using the "feedforwardnet" function instead. I fix
  • the architecture
  • the initialization method, and
  • the seed in the randStream (for generating random numbers)
to be the same. However I still can't get identical results. Although I use the same initialization function I found that the difference is in the initial weights and biases. why are the weights and biases initialized differently? Debugging the functions I found that the "init" function is called more times than expected using the "patternnet" function. Why is this?
I post some code and results to prove that random number generator is not an issue. (A similar question has been asked here, but the problem has not been resolved).
TrainFeatNN = SelectedFeat40.Train;
TrainObsNN = trainToReduce.ObsClasses(:,2:3);
testInd = false(size(SelectedFeat40.Train,1),1);
testInd2 = false(size(SelectedFeat40.Train,1),1);
testInd3 = false(size(SelectedFeat40.Train,1),1);
testInd4 = false(size(SelectedFeat40.Train,1),1);
posclass = 1;
x = TrainFeatNN';
t = TrainObsNN';
%
%
%%Neural Networks patternnet #2
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net = patternnet(hiddenLayerSize, trainFcn);
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','fixunknowns'};
net.divideFcn = 'divideblock';
net.divideMode = 'sample';
net.divideParam.trainRatio = 60/100;
net.divideParam.valRatio = 20/100;
net.divideParam.testRatio = 20/100;
net.trainParam.max_fail = 6;
net.trainParam.epochs = 1000;
net.performFcn = 'crossentropy';
[net,tr] = train(net,x,t);
y = sim(net,x);
PredProb_NN = transpose(y(2,tr.testInd));
testInd(tr.testInd) = true;
[XNN,YNN,~,AUC_NN] = perfcurve(TrainObsNN(testInd,end),PredProb_NN, posclass);
display(sprintf('Method: patternnet (#1): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr.best_epoch,tr.best_perf,AUC_NN)
%
%
%%Neural Networks feedforwardnet #1
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net2 = feedforwardnet(hiddenLayerSize,trainFcn);
net2.performFcn = 'crossentropy';
net2.outputs{numel(hiddenLayerSize)+1}.processParams{2}.ymin = 0;
net2.layers{net2.numLayers}.transferFcn = 'softmax';
net2.plotFcns = {'plotperform','plottrainstate','ploterrhist','plotconfusion','plotroc'};
net2.initFcn='initlay';
net2.layers{1}.initFcn='initnw';
net2.layers{2}.initFcn='initnw';
net2.layers{1}.transferFcn = 'tansig';
net2 = init(net2);
net2.input.processFcns = {'removeconstantrows','mapminmax'};
net2.output.processFcns = {'removeconstantrows','fixunknowns'};
net2.divideFcn = 'divideblock';
net2.divideMode = 'sample';
net2.divideParam.trainRatio = 60/100;
net2.divideParam.valRatio = 20/100;
net2.divideParam.testRatio = 20/100;
net2.trainParam.max_fail = 6;
net2.trainParam.epochs = 1000;
[net2,tr2] = train(net2,x,t);
y2 = sim(net2,x);
PredProb_NN2 = transpose(y2(2,tr2.testInd));
testInd2(tr2.testInd) = true;
[XNN2,YNN2,~,AUC_NN2] = perfcurve(TrainObsNN(testInd2,end), PredProb_NN2, posclass);
display(sprintf('Method: feedforwardnet (#1): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr2.best_epoch,tr2.best_perf,AUC_NN2))
%
%
%%Neural Networks patternnet #2
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net3 = patternnet(hiddenLayerSize, trainFcn);
net3.input.processFcns = {'removeconstantrows','mapminmax'};
net3.output.processFcns = {'removeconstantrows','fixunknowns'};
net3.divideFcn = 'divideblock';
net3.divideMode = 'sample';
net3.divideParam.trainRatio = 60/100;
net3.divideParam.valRatio = 20/100;
net3.divideParam.testRatio = 20/100;
net3.trainParam.max_fail = 6;
net3.trainParam.epochs = 1000;
net3.performFcn = 'crossentropy';
[net3,tr3] = train(net3,x,t);
y3 = sim(net3,x);
PredProb_NN3 = transpose(y3(2,tr3.testInd));
testInd3(tr3.testInd) = true;
[XNN3,YNN3,~,AUC_NN3] = perfcurve(TrainObsNN(testInd3,end), PredProb_NN3, posclass);
display(sprintf('Method: patternnet (#2): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr3.best_epoch,tr3.best_perf,AUC_NN3))
%
%
%%Neural Networks feedforwardnet #2
%
%
RandStream.setGlobalStream (RandStream ('mrg32k3a','Seed', 1234));
trainFcn = 'trainscg';
hiddenLayerSize = 10;
net4 = feedforwardnet(hiddenLayerSize,trainFcn);
net4.performFcn = 'crossentropy';
net4.outputs{numel(hiddenLayerSize)+1}.processParams{2}.ymin = 0;
net4.layers{net4.numLayers}.transferFcn = 'softmax';
net4.plotFcns = {'plotperform','plottrainstate','ploterrhist','plotconfusion','plotroc'};
net4.initFcn='initlay';
net4.layers{1}.initFcn='initnw';
net4.layers{2}.initFcn='initnw';
net4.layers{1}.transferFcn = 'tansig';
net4 = init(net4);
net4.input.processFcns = {'removeconstantrows','mapminmax'};
net4.output.processFcns = {'removeconstantrows','fixunknowns'};
net4.divideFcn = 'divideblock';
net4.divideMode = 'sample';
net4.divideParam.trainRatio = 60/100;
net4.divideParam.valRatio = 20/100;
net4.divideParam.testRatio = 20/100;
net4.trainParam.max_fail = 6;
net4.trainParam.epochs = 1000;
[net4,tr4] = train(net4,x,t);
y4 = sim(net4,x);
PredProb_NN4 = transpose(y4(2,tr4.testInd));
testInd4(tr4.testInd) = true;
[XNN4,YNN4,~,AUC_NN4] = perfcurve(TrainObsNN(testInd4,end), PredProb_NN4, posclass);
display(sprintf('Method: feedforwardnet (#2): \nBest Epoch: %d,\nBest performance: %.4f,\nAUC: %.4f\n',tr4.best_epoch,tr4.best_perf,AUC_NN4))
%
%
if ~isequal(tr.testInd,tr2.testInd) || ~isequal(testInd,testInd2) || ...
~isequal(tr.testInd,tr3.testInd) || ~isequal(tr.testInd,tr4.testInd)
warning('The test sets are not the same!')
end
if isequal(y,y3) && isequal(y2,y4)
display('Random number generation is *not* an issue.')
else
warning('There is a random number generator issue')
end
if isequal(y,y2) || isequal(y3,y4)
display('The two networks are identical')
else
warning('The two netowrks are *not* identical!')
end
Output:
Method: patternnet (#1): Best Epoch: 122, Best performance: 0.2379, AUC: 0.7348
Method: feedforwardnet (#1): Best Epoch: 136, Best performance: 0.2380, AUC: 0.7331
Method: patternnet (#2): Best Epoch: 122, Best performance: 0.2379, AUC: 0.7348
Method: feedforwardnet (#2): Best Epoch: 136, Best performance: 0.2380, AUC: 0.7331
Random number generation is not an issue. Warning: The two netowrks are not identical!

Accepted Answer

Greg Heath
Greg Heath on 21 Oct 2015
Why didn't you just make a simple search on
patternnet feedforwardnet
and find
http://www.mathworks.com/matlabcentral/answers/196799-how-to-simulate-default-patternnet-with-feedforwardnet-in-matlab
Thank you for formally accepting my answer
Greg

More Answers (0)

Categories

Find more on Deep Learning Toolbox in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!