Asked by Céldor
on 6 Apr 2016

Hello,

I have a problem with some outputs from a trained custom neural network. I am using MATLAB 2014 with NN ToolBox ver 8.2.

I have created a simple feedforward NN for classification. I have used some Inputs and Targets, trained the NN, and tried to simulate Outputs given Inputs from the same range. The NN I created gives me incorrect outputs. First it returns outputs only with 0,1 range while the targets are in rage of -6 ... 3 and when I add a process function 'mapminimax' to input and output, the results are wrong: 1. when targets are 0 ... 1, there is an offset of 0.5 such that the outputs are 0.5 and 1 2. when targets are e.g. -6 ... 3, the outputs are around -3 ... 3

I am trying to understand what I am doing wrong.

PS. I have already asked this question in SO and also I provided some more details code: http://stackoverflow.com/questions/36449224/mapminmax-process-function-causes-that-nn-incorrectly-simulates-outputs

The code to test NN

clear all; close all; clc;

net = NNPatRec;

%net.inputs{1}.processFcns = {'mapminmax'};

%net.outputs{2}.processFcns = {'mapminmax'};

Inputs = -10:10;

%Targets = [-6*ones(1,11) 3*ones(1,10)];

Targets = [zeros(1,11) ones(1,10)];

[net,tr] = train(net,Inputs,Targets);

net(-10:10)

The code to create NN:

function net = NNPatternRecognition

net = nntest;

end

function net = nntest

net = network;

net.numInputs = 1;

net.numLayers = 2;

net.biasConnect = [1 1]';

net.inputConnect = [1; 0];

net.layerConnect = [0 0; 1 0];

net.outputConnect = [0 1];

% Inputs

%net.inputs{1}.processFcns = {'mapminmax'};

net.inputWeights{1}.learnFcn = 'learngdm';

% layers 1 (at input)

net.layers{1}.initFcn = 'initnw';

net.layers{1}.netInputFcn = 'netsum';

net.layers{1}.transferFcn = 'tansig';

net.layers{1}.size = 3;

% layers 2 (hidden)

net.layers{2}.initFcn = 'initnw';

net.layers{2}.netInputFcn = 'netsum';

net.layers{2}.transferFcn = 'purelin';

net.layers{2}.size = 1;

% Network functions

net.adaptFcn = 'adaptwb';

net.derivFcn = 'defaultderiv';

net.divideFcn = 'dividerand'; %'divideblock';

net.initFcn = 'initlay';

net.performFcn = 'crossentropy';

net.trainFcn = 'trainscg';

% Outputs

%net.outputs{2}.processFcns = {'mapminmax'};

%net.outputs{2}.exampleOutput = [0 1];

net.trainParam.showWindow = false;

net.trainParam.showCommandLine = true;

end

Answer by Greg Heath
on 8 Apr 2016

Accepted Answer

close all, clear all, clc, plt=0, tic

x = -10:10; N = length(x)

trueind = 1 + [zeros(1,11) ones(1,10)];

t = full(ind2vec(trueind))

plt = plt+1, figure(plt), hold on

plot( x( 1:11), trueind( 1:11) ,'o' )

plot( x(12:21), trueind(12:21),'ro' )

axis([ -11 11 0 3 ])

title('CLASS INDICES')

rng('default')

net = patternnet;

[ net tr y e ] = train( net, x, t );

outind = vec2ind(y)

plot( x( 1:11), outind( 1:11) ,'x' ,'LineWidth',2)

plot( x(12:21), outind(12:21),'rx' ,'LineWidth',2)

err = outind~=trueind;

Nerr = sum(err) % 1

PctErr = 100*Nerr/N % 4.7619

Hope this helps.

For details, remove the semicolon to get

net = net

ALSO, for a trn/val/tst breakdown use

tr = tr

Hope this helps.

Thank you for formally accepting my answer

Greg

Céldor
on 11 Apr 2016

Hi Greg,

Thanks for your answer. I followed your code.

In the final matrix produced by Brendan and yourself, there are three categories not two as I used in my example. Is it a standard routine in classification to always add one extra category?

Thanks

Greg Heath
on 12 Apr 2016

I don't know where you got that idea. Did you run my program?

Brendan Hamm
on 12 Apr 2016

Sign in to comment.

Answer by Brendan Hamm
on 6 Apr 2016

You would likely have better luck if you just started with the patternnet which is meant for NN classification.

What I see that is wrong with your current implementation is you have a linear transferFcn for your second layer. For classification purposes this should really be a softmax function. That is change

net.layers{2}.transferFcn = 'purelin';

to

net.layers{2}.transferFcn = 'softmax';

There are also 2 functions used for the processFcns for the the input and output:

net.outputs{2}.processFcns == {'removeconstantrows', mapminmax};

net.inputs{1}.processFcns == {'removeconstantrows', mapminmax};

Brendan Hamm
on 7 Apr 2016

mapminmax will map the output values to a minimum and maximum value. By default this is set to the range [-1 1]. So, I suppose I missed this part above, we would need to change the range to be [0,1]. This is controlled in the processParams:

p.outputs{2}.processParams{1}.max_range = 0; % params for remove constant rows

p.outputs{2}.processParams{2}.ymin = 0; % Params for mapminmax

p.outputs{2}.processParams{2}.ymax = 1; % Params for mapminmax

Furthermore with a classification problem the output should really be a probability of belonging to each class. For this reason for a binary classification you would expect 2 outputs for every sample, the probability of class1 and the probability of class2.

So, You really want your targets to be a matrix of ones and zeros where 1 indicates belonging to that class (i.e. you know it belongs to class1 so the probability is 1). You can get this result by calling:

dummyvar(Targets+1)';

I use the +1 as dummyvar expects all inputs to be positive integers (or alternatively categorical or strings) So all in all this could have been accomplished with the following:

Inputs = -10:10;

Targets = [ones(1,11) 2*ones(1,10)]; % I'll Increment by 1 here (Class name is irrelevant)

p = patternnet(10); % For classification problems

Targets = dummyvar(Targets)'; % Make this into an indicator matrix

p = train(p,Inputs,Targets); % train the NN

Tpred = p(Inputs) % Verify the results with our training set.

Tpred =

Columns 1 through 3

1.0000 1.0000 1.0000

0.0000 0.0000 0.0000

Columns 4 through 6

1.0000 1.0000 1.0000

0.0000 0.0000 0.0000

Columns 7 through 9

1.0000 1.0000 1.0000

0.0000 0.0000 0.0000

Columns 10 through 12

1.0000 1.0000 0.0000

0.0000 0.0000 1.0000

Columns 13 through 15

0.0000 0.0000 0.0000

1.0000 1.0000 1.0000

Columns 16 through 18

0.0000 0.0000 0.0000

1.0000 1.0000 1.0000

Columns 19 through 21

0.0000 0.0000 0.0000

1.0000 1.0000 1.0000

Brendan Hamm
on 7 Apr 2016

Céldor
on 8 Apr 2016

Many thanks for this Brendan. Everything's working :)

Regarding mapminmax, I also found what I needed to set yesterday. I had to go through the process of how the command patternnet creates a a netowrk and realised it uses feedforwardnet. But somewhere in the code, there's a line where the ymin is set to 0. I have set ymin by myself and everything started to work :)

And thanks for the comment on preparation of Targets for classification. This is another thing I had to go through by myself and realised that NN works much better if there are multiple outputs, each representing individual class.

Thanks :)

PS. I have just learned a new command: dummyvar

Sign in to comment.

Opportunities for recent engineering grads.

Apply Today
## 0 Comments

Sign in to comment.