Using backpropagation on a pre-trained neural network

1 view (last 30 days)
I am developing a project about autoencoders (based on the work of G. Hinton) and I have a neural network which is pre-trained with some Matlab scripts that I have already developed. Now I need to perform a fine-tuning stage through backpropagation, and I am trying to use the 'Neural Network Toolbox'. All my data are already pre-processed (zero mean, unit variance, and so on) and I don't need any more pre-processing, nor post-processing. How do I disable them? Besides, I want this backpropagation stage to begin from the weights and biases found with the pre-training (that are stored in some matrices).
The code I have written so far is
% set the network hidden layers' size and the training function
net = feedforwardnet([layer1, layer2, layer3, layer4, layer3, layer2, layer1], 'traincgp');
% the training data are stored as rows in Dtrain matrix
net = configure(net, Dtrain', Dtrain');
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
% initialize weights and biases to the value found with pre-training
% Note: the weights are symmetric but the biases are not
net.IW{1} = vishid';
net.LW{8,7} = vishid;
net.LW{2,1} = hidpen';
net.LW{7,6} = hidpen;
net.LW{3,2} = hidpen2';
net.LW{6,5} = hidpen2;
net.LW{4,3} = hidtop';
net.LW{5,4} = hidtop;
net.b{1} = hidrecbiases';
net.b{2} = penrecbiases';
net.b{3} = penrecbiases2';
net.b{4} = toprecbiases';
net.b{5} = topgenbiases';
net.b{6} = hidgenbiases2';
net.b{7} = hidgenbiases';
net.b{8} = visbiases';
net.trainParam.epochs = 200;
% set the transfer functions: logsig for every layer
% except for the last one which is linear
for ii=1:size(net.layers, 1)-1
net.layers{ii}.transferFcn = 'logsig';
end;
net.layers{size(net.layers, 1)}.transferFcn = 'purelin';
% train the net
net = train(net, Dtrain', Dtrain');
% save the new weights
weight_i1_b = (net.IW{1})';
weight_12_b = (net.LW{2,1})';
weight_23_b = (net.LW{3,2})';
weight_34_b = (net.LW{4,3})';
weight_45_b = (net.LW{5,4})';
weight_56_b = (net.LW{6,5})';
weight_67_b = (net.LW{7,6})';
weight_78_b = (net.LW{8,7})';
hidrecbiases_b = (net.b{1})';
penrecbiases_b = (net.b{2})';
penrecbiases2_b = (net.b{3})';
toprecbiases_b = (net.b{4})';
topgenbiases_b = (net.b{5})';
hidgenbiases2_b = (net.b{6})';
hidgenbiases_b = (net.b{7})';
visbiases_b = (net.b{8})';
% save everything to file
% ...
The problem is that it's not working the way I expect: the error seems to go down but the result (which is the reconstruction of an image) is worse that what I obtain by just pre-training.
Can anyone tell me what I am missing here? Thank you!
  2 Comments
Greg Heath
Greg Heath on 26 Mar 2015
A clear explanation of what you are trying to do...and why.
minomic
minomic on 26 Mar 2015
Edited: minomic on 26 Mar 2015
Ok, I will try. This famous paper by G. Hinton [ link ] talks about deep autoencoders and how they can be used to reduce the dimensionality of data.
What I am trying to do is to implement such a 'system' for grayscale images. The first part is the pretraining of the RBMs, which are the different layers that compose the autoencoder, and I have done it. Then the second part is the finetuning with the backpropagation.
The idea is this: backpropagation alone does work, but it takes forever to find a good solution. So we pretrain with another method (which is the 'contrastive divergence') to find sensible values for weights and biases, and then we fine-tune from there using backpropagation.
So I need to start from the weights and biases which are found with the pre-training and then use backpropagation algorithm.
The code I have written so far is in my first post, but it does not seem to work. I try to explain why I say so: a simple test which we can do is to give the network an image and ask for its reconstruction. This means the the image is encoded and then decoded (reconstructed from the 'features'). In an optimal case we expect the reconstruction to be equal to the original image, even though a certain loss is inevitable. With the fine-tuning I obtain worse reconstructions if I compare them with the reconstructions I obtain with just the pre-training, and this makes me suspect that there is something wrong in my code.
Can you help me, please? Thank you!

Sign in to comment.

Accepted Answer

minomic
minomic on 27 Mar 2015
After many attempts I found the solution: for some reason (which is not clear to me) the reconstruction script was not working correctly after the fine-tuning, even though it works well when I reconstruct my data after pre-training the network.
So the solution is actually quite simple: given an input 'x', the output produced by the network 'net' is just
y = net(x);
and this reconstruction is much better than the one obtained by just pre-training.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!