Bounded neural network output

Hi guys,

I have a couple of questions related to NNs. I want to define a neural network in which outputs are bounded in a user-defined range. How could I do that?

First thing I tried was to change the output transfer function to tansig, expecting the output to be in the range [-1, 1]. Well, it doesn't. Any idea why?

Then, during some tweaking, I've tried to set all the weights and biases of the net to zero, and expected at least to see a zero output. Well, again, the output is 1.5. Why? What am I missing?

Here the code I've experimented with.

HIDDEN_LAYER_SIZE = 10;
net = fitnet(HIDDEN_LAYER_SIZE);
% cheating to set input and output dimensions
x = [1 1 1; 2 2 2]'; t = [1 1; 2 2]';
net = configure(net, x, t);
net.layers{2}.transferFcn = 'tansig';
display('TANSIG OUTPUT');
net([1 1 1]')  % expecting something in [-1, 1]
display('TANSIG OUTPUT WITH ALL ZERO W and b');
net = setwb(net,zeros(1,net.numWeightElements));
net([1 1 1]')  % expecting 0

Output:

TANSIG OUTPUT

ans =

    1.0254
    1.0338

TANSIG OUTPUT WITH ALL ZERO W and b

ans =

    1.5000
    1.5000

Any help will be highly appreciated!

All the best,

Francesco

Answer 1

[Greg Heath]

 clear all, clc
H = 10;
net = fitnet(H);
b1  = net.b{1}    % zeros(10,1)
IW  = net.IW{1,1} % Empty matrix: 10-by-0
b2  = net.b{2}    % Empty matrix: 0-by-1
LW  = net.LW{2,1} % Empty matrix: 0-by-10

% cheating to set input and output dimensions

%% CHEATING?? I have no idea what that is supposed to mean

 x = [1 1 1; 2 2 2]'; t = [1 1; 2 2]';
%%x = repmat([1 2], 3,1], x = repmat([1 2], 2,1]

 [ I N ] = size(x) % [ 3 2 ]
 [ O N ] = size(t) % [ 2 2]

%% When configured, the number of weights will be

Nw = (I+1)*H+(H+1)*O  %62

 net = configure(net, x, t);
b1 = net.b{1}    % size(b1) = [10 1 ]
IW = net.IW{1,1} % size(IW) = [10 3 ]
b2 = net.b{2}    % size(b2) = [ 2 1 ]
LW = net.LW{2,1} % size(LW) = [ 2 10]

%% With the default 'purelin' output

 y = net(x)     %[ 1.714  0.80213;  0.59833  2.813 ]

%% With the 'tansig' output replacement % expecting something in [-1, 1]

 net.layers{2}.transferFcn = 'tansig';
y = net(x)     %[ 1.7019  1.0578 ;  1.0264  1.9948 ]

1. Ordinarily the net is trained with x and t to try to give an output y with a small error e = t-y. None of these quantities are necessarily bounded in range.

2. Now, even though you configured the net with random weights, you never trained the net. Therefore, your e =

 t-y = [ -0.7019  0.9422; -0.02642  0.005209 ]

is not small.

3. By default, the net transforms the target to fit in [-1,1] The corresponding normalized output, yn, then fits in [-1 1]. However, by default, the net automatically reverses the transform to obtain y from yn.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Kilin]

Hi Greg, and thanks for your reply.

A couple of remarks:

1- I don't want to train the network by now. It will be trained later on in an unsupervised setting using genetic algorithms. So I do not have tranining examples, I will just try out a number of randomly generated networks on a given task and combine those that perform better.

2 - this is the reason why I wrote "% cheating to set input and output dimensions": I saw that a way to configure input and output dimensions was to provide some training data, so I defined some fake data of appropriate dimensions just to do this. As I said, no training will be performed.

3- Regarding the other issues:

- output not between [-1,1] even if output layer has tansig activation: this is due to the fact that the system applies some kind of de-normalization using the fake inputs I provided? By the way, I know that usually training examples are not in a particular range, and I think this is the reason why usually we have a linear output function. However, in my application I want to specifically bound the output range. The output is a control signal, and given that I am going to generate several different networks randomly, there is no point in trying to evaluate networks with physically unrealistic outputs. I thus want to bound the output of the net in a physically plausible range!

- what about the non-zero output issue I mentioned in the test in which all weights and biases are set to zero?

Thanks a lot

[Greg Heath]

1. Configuring with tansig AND a target bounded in [-1,1] are necessary for the output to be bounded in [-1,1]

2. When all weights are zero, the nonzero output must come from de-normalization.

 clear all, clc
rng('default')
H = 10;
net = fitnet(H);
x = [1 1 1; 2 2 2]'; t = [1 1; 2 2]';
net = configure(net, x, t);
y1 = net(x) % [  0.9252  1.1196 ; 2.916  2.3395]
net.layers{2}.transferFcn = 'tansig';
y2 = net(x) % [ 1.0912  1.1792 ; 1.9965  1.9664 ]
net = configure(net, x, t);
y3 = net(x) % [ 1.9323  1.0115 ; 1.8776  1.7376 ]

 net = fitnet(H);
x = [1 1 1; 2 2 2]'; t = [-1 -0.5; 0.5 1]';
net = configure(net, x, t);
y1 = net(x) % [ 1.5652  -0.18302 ;  2.1597  0.98423]
net.layers{2}.transferFcn = 'tansig';
y2 = net(x) % [  0.48824  -0.1832 ; 0.99084  0.81447]
net = configure(net, x, t);
y3 = net(x) % [ -0.96228  -0.91735 ; -0.3731  -0.45783 ]

 net = setwb(net,zeros(1,net.numWeightElements));
b1 = net.b{1}        % zeros( 10, 1 )
IW =net.IW{1,1}    % zeros( 10, 3 )
b2=net.b{2}          % zeros( 2 ,1 )
LW = net.LW{2,1} % zeros( 2 , 10 )
y4 = net(x) % [ -0.25  -0.25 ; 0.25  0.25 ]
net=configure(net,x,t)
y5 = net(x) % [  -0.16752     -0.99733 ; 0.846      0.77261 ]
}

[Kilin]

Hi Greg,

yes, finally I came to the same conclusion. By the way creating a custom neural network instead of using fitness did the trick, and I was finally able to configure everything without unexpected behaviors.

Thanks for your help!

All the best