Neural Network Classification Problem: Performance in nntraintool always constant

I am trying to use the Neural Network Toolbox for a manufacturing related problem. I have 25 categorial input parameters and roughly 5.000 datasets for training (txTBL). The categorial output has 144 classes(stxIndexVA). Now I would like to use 180 new data sets with the 25 input variables in order to estimate the categorial output (tvTBL).

    hiddenLayerSize = [5,10,10];   
    net = patternnet(hiddenLayerSize);
    net.performFcn = 'crossentropy';
    net.trainParam.goal = 1e-6;

    [net,tr] = train(net,txTBL,stxIndexVA);
    nntraintool

    y=net(tvTBL); 
    classes = vec2ind(y);

The neural network seems to work, however results are worse than random guessing. For the dataset with 180 instances I have the correct results, so I can calculate a percentage of correctly predicted classes. Maybe my network is just not big enough for this classification data? I tried to increase input up to 10.000 data sets (I have up to 100.000 if needed but I hoped I could do it with less) but the results were the same.

Some problems I encountered and cannot solve although digging through the whole documentation.

1. The training tool always runs through the whole 1000 epochs without improving performance at all (plot is a complete constant). What could be the reason? In one run, the expected plot showed up (although results did not improve) but I wasn't able to reproduce this.
2. From the possible 144 output classes which the network should have learned, it only uses up to 10 classes to predict the 180 cases. How can I make sure the network uses more of the possible outputs? I tried playing around with the amount of layers and neurons but didn't see much improvement.I led myself guide by this tipp, I found in the forum: "A general rule is to have less input parameters than output parameters. Also, it is not necessary to have as many neurons as input parameters, and the number of output neurons should be significantly less than the number of input parameters. For example, if you have your input P with 200 input parameters then it not necessarily beneficial to have 200 input neurons. You may want to try and use 20 input neurons. It is very difficult to give an exact ratio of input parameters to input neurons because each application calls for specific network architectures."
3. Is there a better performFcn for this problem than trainscg which is the standard

Thank you very much for your help.

Answer 1

[Greg Heath]

>I am trying to use the Neural Network Toolbox for a manufacturing related problem. I have 25 categorial input parameters and roughly

Spelling: categoriCal

> 5.000 datasets for training (txTBL). The categorial output has 144 classes(stxIndexVA).

Are the target columns 144-dimensional {0,1} unit vectors?

Have you clustered the data to check for class consistency?

Are the class sizes disparate in size? What range of sizes?

> Now I would like to use 180 new data sets with the 25 input variables in order to estimate the categorial output (tvTBL).

> hiddenLayerSize = [5,10,10];

Why are you using more than one hidden layer??? Why do you think only 5 dimensions of a 25 dimensional input space are relevant?

     net = patternnet(hiddenLayerSize);
     net.performFcn = 'crossentropy';

Unnecessary command. It is the default.

     net.trainParam.goal = 1e-6;
     [net,tr] = train(net,txTBL,stxIndexVA);
     nntraintool
     y=net(tvTBL); 
     classes = vec2ind(y);

Where is the code for comparing predicted classes with target classes?

I'm not sure where you found that quoted advice. If I were you I would try to forget it.

I have posted classification/pattern-recognition advice, tutorials and examples which may be helpful. When searching the NEWSGROUP and ANSWERS, include " greg patternnet" in the search words. However, I suggest you first perform some preliminary data analysis, especially clustering, to better understand the data.

Also, keep track of the RNG states so that you can duplicate good designs.

Hope this helps.

Thank you for formally accepting my answer

Greg