Thread Subject: Pattern Classification using Neural Network ( newff)

Hello,

I am trying to classify a 4 class problem (each class has 20 features ) using neural network.
So, in order to reduce the complexity, i used newff function to get the weights.

The problem is i am not very familiar with newff function usage ( the samples are not classified properly- same sample set is being classified welll using k nearest neighbour and bayesian techniques).

It would be great if i can get feed back on the usage of this newff sequence.

%%%%%%%%

net = newff(training_data',group',no_hiddenLayer);
% Create a new feed forward network. 20 neurons in the hidden layer.
%training data is a matrix of training samles.
%group is a matrix, where each row is for example [1 0 0 0] or [0 1 0 0] or [0 0 1 %0] or [0 0 0 1] based on the class to which the training sample belongs.
%so basically i have 4 outputs.




% training parameters -i took this from an example
net.trainParam.goal = 0.1;
net.trainParam.show = 20;
net.trainParam.epochs = 40;
net.trainParam.mc = 0.95;

[net,tr] = train(net,training_data',group');

W = net.IW{1};
V = net.LW{2};

W =W';
V = V';

the best weights are obtained.

Now i do the testing...and classify based on the 4 outputs( which ever is maximum).

Is this approach correct =or am i missing something?


Thanks

Ok, one more thing i need to tell,

The network assumes 2 layers, but i have implemented equations ( back propagation for 1 layer - so this is the problem i guess).

But how do i do the testing using standard function.

( i have testing data as well, which function should i pass it to?)

Thanks,

I did exactly the same project last year. Your code seems correct. I will assume the following based on the description of the problem:

1. You have 20 rows of input features.
2. You need to classify the problem into 4 possible outcomes.
unless you need to pass the weights into another program, the codes you have written:
 W = net.IW{1};
> V = net.LW{2};
>
> W =W';
> V = V'
is completely unnecessay, since MATLAB will stote the weights in the network(net).

Back to the problem, once you have the network trained, all you have to do is test the network.

test = sim(net,new_input).
The trick here is that your output will produce 4 elements, the one that is closest to 1 is your classification.

For example if your classification is [red green blue orange], and you get
[1.2 2.4 5.6 3] , then red is your classification. because 1.2 is closest to 1

In most cases, you will get negative, you need to decide based on your input features, if you will use absoulte values. For example,
test = sim(net, new_inputs) gives [0.7 1.5 -0.8 3.2], if you take absolute value, then -0.8 is closest to 1, hence blue is your classification.

Finally, I have assumed that you correctly put the inputs and outputs in your training, and you normalize the inputs.

if you are still confused, feel free to let me know

One last thing, you can just let MATLAB use its default parameters, and all you need to change is the number of neurons, and try 2 or more hidden layers, since your input features are many.

Hello,

Thanks for the input. I will try this.


Thanks!

On Nov 2, 8:45 pm, "Kishore " <kishore3...@yahoo.co.in> wrote:
> Hello,
>
> I am trying to classify a 4 class problem (each class has
> 20 features ) using neural network.
> So, in order to reduce the complexity, i used newff function to
> get the weights.
>
> The problem is i am not very familiar with newff function usage
> ( the samples are not classified properly- same sample set is being
> classified welll using k nearest neighbour and bayesian techniques).
>
> It would be great if i can get feed back on the usage of this
> newff sequence.
>
> %%%%%%%%
>
> net = newff(training_data',group',no_hiddenLayer);

help newff
doc newff

You have accepted the PURELIN output default. For classification,
LOGSIG is superior. The output then represents the posterior
probability (conditional on the input) for class "1".

> % Create a new feed forward network. 20 neurons in the hidden layer.

H = 20 may be far too many. It's best to use the smallest
satisfactory
value which is usually found by trial and error.

For an I-H-O MLP, Ntrn training vectors, size(p) = [I Ntrn],
size(t) = [ O Ntrn] and training without regularization, a good rule
of thumb is Neq >> Nw where

Neq = Ntrn*O = Ntrn*4 = No. of training equations.
Nw = (I+1)*H+(H+1)*O = No. of unknown weights


Search on

greg-heath Neq Nw
greg-heath pretraining advice newbies

for details

> %training data is a matrix of training samles.
> %group is a matrix, where each row is for example
> [1 0 0 0] or [0 1 0 0] or [0 0 1 %0] or [0 0 0 1]
> based on the class to which the training sample belongs.

Each column should be an output target.

size(t) = [ 4 Ntrn]

I don't recommend using the transpose operator in
the call of newff.

> %so basically i have 4 outputs.
>
> % training parameters -i took this from an example
> net.trainParam.goal = 0.1;

Looks OK:

You would get lower than this if the correct class
output is 0.7 and the other outputs are 0.3

> net.trainParam.show = 20;
> net.trainParam.epochs = 40;
> net.trainParam.mc = 0.95;

delete the last 2 and settle for the defaults in TRAINLM.

help trainlm
doc trainlm

> [net,tr] = train(net,training_data',group');


> W = net.IW{1};
> V = net.LW{2};

Incorrect see the documentation

> W =W';
> V = V';
>
> the best weights are obtained.

It is not necessary to explicitly extract the weights
(and the very important thresholds in net.b).
>
> Now i do the testing...and classify based on the 4 outputs
> ( which ever is maximum).
>
> Is this approach correct or am i missing something?

For any input x with correct outputs y0

size(x) = [ I N]
size(y0) = [ O N]
y = sim(net,x);
error = y0-y;
MSE = mse(error)


Hope this helps.

Greg

On Nov 2, 9:24 pm, "Kishore " <kishore3...@yahoo.co.in> wrote:
> Ok, one more thing i need to tell,
>
> The network assumes 2 layers, but i have implemented equations ( back propagation for 1 layer - so this is the problem i guess).

NO. You have 2 layers. Note the N-1 in the documentation.

> But  how do i do the testing using standard function.
>
> ( i have testing data as well, which function should i  pass it to?)

See my previous post.

Hope this helps.

Greg

On Nov 2, 10:08 pm, "ade77 " <ade1...@gmail.com> wrote:
> I did exactly the same project last year. Your code seems correct. I will assume the following based on the description of the problem:
>
> 1. You have 20 rows of input features.
> 2. You need to classify the problem into 4 possible outcomes.
> unless you need to pass the weights into another program, the codes you have written:
>  W = net.IW{1};> V = net.LW{2};
>
> > W =W';
> > V = V'
>
> is completely unnecessay, since MATLAB will stote the weights in the network(net).
>
> Back to the problem, once you have the network trained, all you have to do is test the network.
>
> test = sim(net,new_input).
> The trick here is that your output will produce 4 elements, the one that is closest to 1 is your classification.

No.

The maximum output is the winner.

> For example if your classification is [red green blue orange], and you get
> [1.2  2.4  5.6  3] , then red is your classification. because 1.2 is closest to 1

No.

The correct class is 3.

That is why it is better to use LOGSIG. Then you can choose
take the maximum probability( or maximum risk by considering
prior probabilities and misclassification costs ...See
the NN book by Duda et al ).

> In most cases, you will get negative, you need to decide based on your input features, if you will use absoulte values.

No, No, No!

> For example,
> test = sim(net, new_inputs) gives [0.7  1.5  -0.8  3.2], if you take absolute value,
> then -0.8 is closest to 1, hence blue is your classification.

No!

> Finally, I have assumed that you correctly put the inputs and outputs in your
> training, and you normalize the inputs.
>
> if you are still confused, feel free to let me know

No comment.

> One last thing, you can just let MATLAB use its default parameters, and all you
> need to change is the number of neurons, and try 2 or more hidden layers,

No!

One hidden layer is sufficient.

> since your input features are many.

Hope this is not too late.

Greg

On Nov 8, 5:42 am, Greg Heath <he...@alumni.brown.edu> wrote:
> On Nov 2, 8:45 pm, "Kishore " <kishore3...@yahoo.co.in> wrote:
>
> > Hello,
>
> > I am trying to classify a 4 class problem (each class has
> > 20 features ) using neural network.
> > So, in order to reduce the complexity, i used newff function to
> > get the weights.
>
> > The  problem is i am not very familiar with newff function usage
> > ( the samples are not classified properly- same sample set is being
> > classified welll using k nearest neighbour and bayesian techniques).
>
> > It would be great if i can get feed back on the usage of  this
> > newff sequence.
>
> > %%%%%%%%
>
> > net = newff(training_data',group',no_hiddenLayer);
>
> help newff
> doc newff
>
> You have accepted the PURELIN output default. For classification,
> LOGSIG is superior. The output then represents the posterior
> probability (conditional on the input) for class "1".

You have accepted the PURELIN output default. For classification,
LOGSIG is superior. The ith output then represents the posterior
probability (conditional on the input) for class "i".

Hope this helps.

Greg

On Nov 8, 5:59 am, Greg Heath <he...@alumni.brown.edu> wrote:
> On Nov 2, 10:08 pm, "ade77 " <ade1...@gmail.com> wrote:
>
>
>
>
>
> > I did exactly the same project last year. Your code seems correct. I will assume the following based on the description of the problem:
>
> > 1. You have 20 rows of input features.
> > 2. You need to classify the problem into 4 possible outcomes.
> > unless you need to pass the weights into another program, the codes you have written:
> >  W = net.IW{1};> V = net.LW{2};
>
> > > W =W';
> > > V = V'
>
> > is completely unnecessay, since MATLAB will stote the weights in the network(net).
>
> > Back to the problem, once you have the network trained, all you have to do is test the network.
>
> > test = sim(net,new_input).
> > The trick here is that your output will produce 4 elements, the one that is closest to 1 is your classification.
>
> No.
>
> The maximum output is the winner.
>
> > For example if your classification is [red green blue orange], and you get
> > [1.2  2.4  5.6  3] , then red is your classification. because 1.2 is closest to 1
>
> No.
>
> The correct class is 3.
>
> That is why it is better to use LOGSIG. Then you can choose
> take the maximum probability( or maximum risk by considering
> prior probabilities and misclassification costs ...See
> the NN book by Duda et al ).

Then you can choose the class with the maximum posterior
probability( or minimum risk by considering prior probabilities
and misclassification costs ...See the NN book by Duda et al ).

Hope this helps.

Greg

Hi Greg,
thanks for your suggestions..

1) Yes, i am considering the maximum of the 4 outputs for classification.

2) I am using 'logsig' function at the output layer,instead of the default 'purelin'.
The hidden layer has 10 neurons. -i am also experimenting with variations of this,but this factor does not have major effect .(i tried 8 - 15 -20 etc)

3) The function now looks like

[net,tr] = train(net,training_data,group);

ALSO - training_data and group were transposed before this step

one observation - after the simulation i.e test, the results seems to be biased towards class1
( i.e i get o/p as [1 0.5 0.5 0.5] for class1 test data ( which is fine)
and
[0.5 0.5 0.5 0.5] for the rest. i.e data belonging to class 2/3/4 .

so,everything is getting classified to class1.

4) But, if i use the default purelin,
altleast this bias does not seem to exist although the misclassifcation rate is high.( same data was classified at a good rate using the Baye's method.)

5) Test was done using
test = sim(net,test_data);

I could

Thanks again.
Kishore.

Inputs please...


Thanks..

On Nov 10, 1:59 pm, "Kishore " <kishore3...@yahoo.co.in> wrote:
> Inputs please...
>
> Thanks..

If the sizes of the training subsets are very dissimilar,
you have to use one of several mitigation techniques
to compensate.

See the FAQ. Also search on

"greg-heath" unbalanced

Hope this helps.

Greg

On Nov 13, 1:43 am, Greg Heath <he...@alumni.brown.edu> wrote:
> On Nov 10, 1:59 pm, "Kishore " <kishore3...@yahoo.co.in> wrote:
>
> > Inputs please...
>
> > Thanks..
>
> If the sizes of the training subsets are very dissimilar,
> you have to use one of several mitigation techniques
> to compensate.
>
> See the FAQ.

For comp.ai.neural-nets.

>Also search on
>
> "greg-heath" unbalanced

in Google Groups

Hope this helps.

Greg