RNG neural Network and outputs

Hello everyone.!

I am new in nn so that might be a silly question, but as I change the rng of my nn, the quality of the solution changes too.

For example, for a specific rng , the best setup is using softmax in the second layer, with 11 neurons in the first one. However with a different rng, the best setup is logsig in the second layer with 11 neurons in the first one. What is going on with that?Is there an optimal rng? Also, although I have formulated my output in a 1-c form, the output I get is not binary. Why? I use patternnet, with 10 input categories with 180 responses and 5 output classes.

Thank you all.!

Answer 1

[Greg Heath]

GEH1 = 'Size of input and target matrices and Hub?'

GEH2 = ' Are target columns {0,1} unit vectors?'

GEH3 = 'I find it better to only look at ~ 100 designs at a time : Ntrials ~ numel(Hmin:dH:Hmax) ~ 10. I don"t recall ever having to make more than ~ 200 designs'

GEH4 = ' Too much space is wasted on statements that assign defaults. Accept as many as you can and delete the corresponding statements. goal and min_grad are the only ones I specify'

GEH5 = ' Use dH > 1 and do not use h as a matrix index'

GEH6 = 'defaults TRAINSCG and CROSSENTROPY are usually preferred for classification'

GEH7 = ' Error rates are what what you are trying to minimize. Check out some of my patternnet posts that yield error rates.'

ope tis helps.

Thank you for formally accepting my answer

Greg

[Andreas]

Dear Dr. Heath,

Sorry for the absence of data about the problem,

I have

I=10

N=173

O=5

Ntrn=121

Ntrneq=605 No. of training equations

Hub=37.5

Targets were constructed through dummyvar command, and I checked that they are in the right form (1 out of c).

Maybe I could use this:

Ndof = Ntrneq-Nw

MSEgoal = 0.01*(Ndof/Ntreq)*MSE00

(MSE00=mean(var(targets',1)) ) ,

as a goal?

Finally, using:

input=minmax(std(input))

could give me a better quality of input for my ordinal data?

By error rate you mean the sum(target~=predicted)?

I've read that adding noisy duplicates of small classes with high error rates can address the problem, but I cannot get the implementation of it.

Also I chose trainlm as it had less overall error (~0.21) than trainscg (~0.31)

Best regards. Andreas Kampianakis

[Greg Heath]

% I=10, N=173, O=5, Ntrn=121, Ntrneq=605, Hub=37.5

 Ntrials = 10, h = 0:4:36 or 1:4:37 yields ~ 100 values.

% Targets were constructed through dummyvar command,

WHAT?

 indices = [ 5 3 1 4 2 ],
 target1 = full(ind2vec(indices))  % Correct
 target2 = dummyvar(indices)     % Error

% Maybe I could use this: % Ndof = Ntrneq-Nw % MSEgoal = 0.01*(Ndof/Ntreq)*MSE00 % (MSE00=mean(var(targets',1)) ) , % as a goal?

No. Only use training data

 MSEtrn00  = mean(var(targets(:,trnind))',1) % biased
 MSEtrn00a = mean(var(targets(:,trnind))',0) % unbiased
 Ntrndof   = Ntrneq-Nw 
 MSEgoal   = 0.01*max(0,Ntrndof/Ntrneq)*MSEtrn00a

% Finally, using: % % input=minmax(std(input)) % could give me a better quality of input for my ordinal data?

No. That makes no sense.

(Do you mean integer data?)

% By error rate you mean the sum(target~=predicted)?

 No. That is Nerr, the total number of errors. 

 PctErr = 100*Nerr/N

However, you also have to calculate the errors for trn/val/tst using the indices in tr

% I've read that adding noisy duplicates of small classes with % high error rates can address the problem, but I cannot get %the implementation of it.

Something like

 noisyduplicate = mean + a*std*randn(size(original))  % scalar

for each input variable (different means and stds).

Good to have Ni (i=1:c) equal

See my BIOID posts

% Also I chose trainlm as it had less overall error (~0.21) than trainscg (~0.31)

Again, val/tst error rates should be the criteria

Hope this helps.

Greg

[Andreas]

Dear Dr. Heath,

I did change my error measurement to

sum(Output2(tr.trainInd)'~=vec2ind(y(:,tr.trainInd)))+sum(Output2(tr.valInd)'~=vec2ind(y(:,tr.valInd)))

in order to get the net with the minimum overall errors, and it seems to work good. About the dummyvar, I used the ' sign in order to get the same result with ind2vec.

I don't have integer data, I have categorical ones, so the use of mapstd or minimax is pointless isn't it?

Could the use of trainlm be a mistake?No matter what I've tried trainlm seems to give the best overall results.

Finally, the best I can get is 15% error, although I have followed every suggestion you and others posted.

Could that be the maximum result I can get?

I post you some of my code.

It is always an honor.

Best regards, Andreas Kampianakis

inputs = Input';
targets = dummyvar(Output2)';

Hmin=1;
dH = 4;
Hmax = 37;

for h = Hmin:dH:Hmax
    % Create a Pattern Recognition Network
    hiddenLayerSize = h;

%     [ I N ] = size(inputs) ;
%     [ O N ] = size(targets);
%     Ntrn = N -2*round(0.15*N);%
%     Ntrneq = Ntrn*O  ;%
%     Nw = (I+1)*h+(h+1)*O;
%     Ndof = Ntrneq-Nw

      net = patternnet(hiddenLayerSize);

      %
      %     net.inputs{1}.processFcns = {'removeconstantrows','fixunknowns'};
      %     net.outputs{2}.processFcns = {'removeconstantrows'};

      % net.outputs{2}.processFcns={};

      net.layers{1}.transferFcn = 'tansig';
      net.layers{2}.transferFcn = 'softmax';

      %         MSEgoal = 0.01*mean(var(target',1));
      %         MinGrad = MSEgoal/100;
      %         net.trainParam.goal     = MSEgoal;
      %         net.trainParam.min_grad = MinGrad;

      % Setup Division of Data for Training, Validation, Testing
      net.divideFcn = 'dividerand';
      net.divideParam.trainRatio = 70/100;
      net.divideParam.valRatio = 15/100;
      net.divideParam.testRatio = 15/100;
      net.trainFcn = 'trainlm';
      net.trainParam.max_fail= 6;
      net.trainParam.showWindow=false;
      net.performFcn = 'mse';

% Train the Network

      for i = 1: 15

          net = configure( net, inputs, targets);
          [net,tr, y, e] = train(net,inputs,targets); %br gamaei

        Kapa=sum(Output2(tr.trainInd)'~=vec2ind(y(:,tr.trainInd)))+sum(Output2(tr.valInd)'~=vec2ind(y(:,tr.valInd)));

          if Kapa<BKapa
              Hiddennodes = h;
              BKapa=Kapa;
              Btr=tr;
              By=y;
              Bnet=net;
          end
      end
  end

Answer 2

[Greg Heath]

I typically

 1. Initialize rng once and only once before training (henever you use the 
rng it AUTOMATICALLY moves to another state).
 2. Use a double for loop design over ~ 10 candidates for H, number of 
hidden nodes and ...
 3. For each value of H, design ~ 10 candidate nets with different random 
data divisions AND random intitial weights.
 4. for h = Hmin:dH:Hmax

 net = .

 for i = 1:Ntrials ...

I have posted zillions of examples on both the NEWSGROUP and ANSWERS. Search using

 patternnet Ntrials

or

patternnet Hmin:dH:Hmax

Hope this helps.

Thank you for formally accepting my answer

Greg

[Andreas]

Dear Dr. Heath,

thank you for your answer, it is an honor.

I have read many of your answers at ANSWER and NEWSGROUP, and this is my final piece of code. Although I have figured out a seemingly optimal nn, it does not seem to work at all. With R2 having a value of 0.9280 at max, I think its accuracy is not adequate. My inputs are nominal {1-5}, with 10 categories, from 182 responses and my output is categorical data with 5 classes, formulated through 1-c (using dummyvar). What other options do I have in order to gain better results?

Thank you Dr. Heath.

rng(4500)
inputs = Input';
targets = target';

Hmin=1;
dH = 1;
Hmax = 40;
Ntrials = 50;

for h = Hmin:dH:Hmax
    % Create a Pattern Recognition Network
    hiddenLayerSize = h;
    net = patternnet(hiddenLayerSize);
    net.inputs{1}.processFcns = {'removeconstantrows','fixunknowns','mapstd'};
    net.outputs{2}.processFcns = {'removeconstantrows'};
    net.layers{1}.transferFcn = 'tansig';
    net.layers{2}.transferFcn = 'softmax';

      % Setup Division of Data for Training, Validation, Testing
      net.divideFcn = 'dividerand';
      net.divideParam.trainRatio = 70/100;
      net.divideParam.valRatio = 15/100;
      net.divideParam.testRatio = 15/100;
      net.trainFcn = 'trainlm';
      net.trainParam.max_fail= 6;
      net.performFcn = 'mse';
      net.trainParam.epochs=1000;
      net.trainParam.min_grad=0;
      for i = 1: Ntrials
          net = configure( net, inputs, targets);
          [net,tr, y, e] = train(net,inputs,targets);         
          stopcrit{i,h} = tr.stop;
          bestepoch(i,h) = tr.best_epoch;

          NMSE = mse(targets-y)/MSE00;% Normalization
          R2(i,h) = 1-NMSE; % Rsquare: See Wikipedia
      end
      end
  stopcrit
  bestepoch
  R2

% Test the Network
outputs = net(inputs);
errors = gsubtract(targets,outputs);
performance = perform(net,targets,outputs);