How to understand performance function implementation in the post-2012 NN toolbox?

Sadly under the current Neural Network toolbox (R2015b) custom function (for example performance function) implementation is undocumented. `help nncustom` instructs to use the vanilla functions as templates for writing your own; for a cost function it suggests `mse` and the accompanying subfunctions in the `+mse` folder. Unfortunately, code in these files is uncommented, and it is not intuitive as to how these functions are used by the nn toolkit or what is expected of them.

This is the contents of the `+mse` folder. What makes up the cost function seems to be structured in these atom functions. However, their exact roles are unclear.

>> ls +mse

.                forwardprop.m    perfwb.m         
..               name.m           type.m           
apply.m          normalize.m      
backprop.m       parameterInfo.m  
dperf_dwb.m      perfw_to_ew.m

Also, for reference, `apply`, `backprop`, and `forwardprop` functions are concatenated below:

function perfs = apply(t,y,e,param)
%MSE.APPLY

perfs = e .* e;

function dperf = forwardprop(dy,t,y,e,param)
%MSE.FORWARDPROP

dperf = bsxfun(@times,dy,-2*e);

function dy = backprop(t,y,e,param)
%MSE.BACKPROP

dy = -2*e;

I'm having a hard time trying to understand how these work together to compose a Mean Squared Error function. It seems like the classical implementation of MSE was decomposed into steps to achieve more flexibility, but combined with no documentation it's become very hard to understand.

As a note, through comparing file by file I found that the only difference between Mean Absolute Error and Sum Absolute Error implementations is that the file `normalize.m` in MAE has a flag that is set to `true`, while it is `false` on SAE, so it probably causes division by number of outputs.

What are the roles of the cost function's subfunctions, and how they should be implemented?

[杰 李]

I had the same problem. Take the liberty to ask do you now understand what these functions mean?

Answer 1

[Dominykas Mostauskis]

Explanation of performance function's subfunctions. Quoted from this answer.

newfcn.m - Same as mse.m
   +newfcn/apply.m - The main performance calculation
   function perfs = apply(t,y,e,param)
   % Calculate performance for each target individually so perfs is same size
   % as t, y and e.
   +newfcn/backprop.m - Backprop derivatives
   function dy = backprop(t,y,e,param)
   % Return dperf/dy, the derivative of performance with respect to each 
   % output y.
   +newfcn/forwardprop.m - Forward propagate derivatives
   function dperf = forwardprop(dy,t,y,e,param) Return dperf/dwb given 
   % dy/dwb.
   +newfcn/type.m - Indicate that "newfcn" is a performance function.
   function t = type
   % t = 'performance_fcn';
   +newfcn/name.m
   function name = name()
   % Return the name of the function, for instance:  name = 
   % 'My NewPeformance Function';
   +newfcn/normalize.m
   function flag = normalize
   % Return true if mean performance is desired (i.e. mean squared error, 
   % mean absolute error) and false if the absolute performance is desired 
   % (i.e. sum squared error, sum absolute error).
   +newfcn/parameterInfo.m
   function param = parameterInfo
   % Return the same array of parameter definitions as MSE.  Customer can 
   % also add additional parameters if desired.
   +newfcn/perfwb.m - Regularization performance used to minimize weights 
   % +and biases
   function perf = perfwb(wb,param)
   % Return the performance measure for weights and biases.  This performance
   % measure is only used when net.performParam.regularization is set to a 
   % value greater than 0.  If regularization is not going to be used this 
   % function can return 0.
   +newfcn/dperf_dwb.m - Regularization derivatives
   function dperf = dperf_dwb(wb,param)
   % If you are not doing regularization then this function can return zeros 
   % the same size as wb.  Otherwise it should return a derivative of 
   % regularization performance (calculated by perfwb) with respect to each 
   % weight and bias value in wb.
   % Please note that there is one more caveat when following this approach 
   % in R2012b – there is an issue that is planned to be resolved in a future
   % release, but currently defining custom functions with this approach
   % works only with the non-MEX version of the Neural Network code, so it is
   % necessary to call TRAIN with a special syntax – i.e., using the nn7 
   % option.  This workaround is the following syntax for training and 
   % simulating the network:
%%%BEGIN CODE%%%
  net = train(net,x,t,nn7);
  y = net(x,nn7);
%%%END CODE%%%
   % The problem is that a check for custom performance functions fails with
   % the new default MEX implementations.  The MEX calculations do not
   % support custom functions yet, so calculations are supposed to
   %  revert to MATLAB. 
   % We plan to address this for a future release but the workaround in the 
   % meantime is to include nn7 as the last argument for TRAIN or simulating 
   % a network.
   % Additionally, the function files and packages to use for defining custom 
   % versions of other Neural Network components can be found by executing 
   % the following in MATLAB:
%%%BEGIN CODE%%%
doc nncustom
%%%END CODE%%%