Performance of Feed Forward Neural Network

I do not understand how tr.perf(end) is computed, this is also the performance value the nntraintool shows in the pop-up window. Does someone know the analytical formula for it?

In particular, I execute the following example of a ff NN from matlab:

    % code
[x,t] = simplefit_dataset;
net = feedforwardnet(10);
[net, tr]= train(net,x,t);
view(net)
y = net(x);
perf1 = perform(net,y,t)
perf2=tr.perf(end)

What is the difference between perf1 and perf 2?

perf1 should be the mse on all data points. I also computed mse on only test, validation and training points but none of them seem to match with perf2...

if true
  % code
yPred = net(x(:,tr.testInd));
perf_Test = perform(net,yPred,t(tr.testInd))
 clear yPred;
yPred = net(x(:,tr.valInd));
perf_Val = perform(net,yPred,t(tr.valInd))
 clear yPred;
yPred = net(x(:,tr.trainInd));
perf_Train = perform(net,yPred,t(tr.trainInd))
end

Did anyone go through this already? Thanks in advance

Answer 1

[Greg Heath]

% 1. ALWAYS START CLEAN

 close all, clear all, clc

% 2. ALWAYS INITIALIZE THE RNG SO THAT RESULTS CAN BE DUPLICATED !!!

 rng(0)           % Typical choice 
% rng(4151941)   % Worse example

% 3. DOCUMENTATION EXAMPLE FROM HELP FEEDFORWARDNET & SPECIAL CASE HELP FITNET (IDENTICAL RESULTS)

 [x, t] = simplefit_dataset;
 [I N ] = size(x)       % [ 1 94 ]
 [O N ] = size(t)       % [ 1 94 ]

% 4. Reference MSE is that for naive guess answer y = mean(t)

 MSEref = mean(var(t',1)) %  8.3378

% 5. ALWAYS PLOT THE TARGET TO COUNT THE NUMBER OF LOCAL MAXES AND MINS.

 plot(x,t)
 Nmax = 2, Nmin = 2

% 6. CHOOSE H >= 2*max(Nmax,Nmin)

 H            = 2*max(Nmax,Nmin)  % 4
 net          = fitnet(H);
 [net tr y e] = train(net,x,t); 
% y = net(x); e = t - y;

 NMSE  = mse(e)/MSEref   % 3.3966e-04

 perf1  = mse(e)         % 0.0028
 trnind = tr.trainInd;
 perf2  = mse(e(trnind)) % 0.0033

 hold on, plot(x,y,'r')
 perf1 = perform(net,y,t) % 0.0028
 perf2 = tr.perf(end)     % 0.0033

% NOTE

 % rng(0):       mse(e) = 0.0028/0.0033
 % 
 % rng(4151941): mse(e) = 0.0492/0.0440

Hope this helps.

Thank you for formally accepting my answer

Greg

[giulia Reggiani]

Thank you very much Greg! So if I understood correctly you are saying that, referring to your previous code:

if true
       % these performances:
perf1 = perform(net,y,t); 
perf2 = tr.perf(end) 
perf3=tr.tperf(end)
perf4=tr.vperf(end)
     %should correspond respectively to: 
perf_Total = mse(e);
perf_Train = mse(e(tr.trainInd)) 
perf_Test = mse(e(tr.testInd))
perf_Val = mse(e(tr.valInd))
end

I am following your example and things change according to the random seed. For example, sometimes perf2 and perf_Train are slightly different. Try rng(1000) and see how they differ.

I iterated your code with different seeds and performances do not match at all times. The error is not too big, but when I go to another data set the error increases significantly.

Is it connected to the pre-processing of the data? Maybe tr.perf(end) uses processed data (normalised) and mse(e(tr.trainInd)) uses not processed data? Do you have another explanation for this?

Could you also elaborate on why you use MSEref and NMSE..

Thanks in advance!