How are the input weights organized in a nonlinear autoregressive networks with an external input (narxnet)?

I have trained a Nonlinear autoregressive networks with an external input using the narxnet() command. The network has 5 input delays and 5 feedback delays and 10 hidden neurons and was created with the following command:

net = narxnet(10:15,10:15,10)

The network was trained to predict a single time series given two time series and the feedback delays as inputs. For research purposes, I am investigating the contributions of the two inputs and feedback delays to the output. To understand relative contributions, it is imperative to first understand the organization of the individual input weight matrices contained in the following commands:

net.IW and net.inputWeights

Given the following example code,

     [X,T] = simplenarx_dataset;
     net = narxnet(10:15,10:15,10);
     [Xs,Xi,Ai,Ts] = preparets(net,X,{},T);
     net = train(net,Xs,Ts,Xi,Ai);

here is my question: In the input weight matrices of the above network, how are the inputs and feedback delays organized? Are the input weights the first column(s) in the weight matrix net.IW{1}?

Thanks, Sam

Answer 1

[Greg Heath]

    close all, clear all, clc;

    net = narxnet(10:15,10:15,10)      % Removed semicolon
    % net.numInputs = 2                % Input and Output Feedback
    % net.numInputDelays = 15          % BUG? Not 2*length(10:15) = 12?
    % net.numFeedbackDelays = 0        % Open Loop
    % net.numWeightElements = 10 % 10 zero bias weights (I & O not configured)

    IWc = net.IW                 % 2 inputs
    LWc = net.LW
    bc = net.b

    IW = cell2mat(IWc)           % BUG? No indication of 2 inputs
    LW = cell2mat(LWc)
    b = cell2mat(bc)

    view(net)

    [X,T] = simplenarx_dataset;
    [Xs,Xi,Ai,Ts] = preparets(net,X,{},T);
    rng(0)                                 % So you can duplicate results
    [net tr Ys Es ]=  train(net,Xs,Ts,Xi,Ai);   % ALWAYS include tr !!!

    IWc = net.IW
    LWc = net.LW
    bc = net.b

    IW = cell2mat(IWc)               % [10 12] = size( IW)
    LW = cell2mat(LWc)               % [ 1 10] = size( LW)
    b = cell2mat(bc)                 % [11  1] = size( b)

    view(net)

    tr = tr                          % For your edification

    %      Hope this helps
    %
    %      Thank you for formally accepting my answer
    %
    %      Greg

Answer 2

[Greg Heath]

% net = narxnet(10:15,10:15,10)

Why are the input and feedback delayed by 10 to 15 (6) timesteps?

Are you trying to predict 10 timesteps ahead?

Have you looked at the autocorrelation function of T and the crosscorrelation function of T and X to see if delays of 10-15 are significant?

% The network was trained to predict a single time series given two time series and the feedback delays as inputs.

Do you mean X is 2 dimensional in addition to the output feedback from T?

% Are the input weights the first column(s) in the weight matrix net.IW{1}?

net.IW{1} is not a matrix. It is the (1,1) element of the 2x2 cell IWc = net.IW.

Predict the result of each of the following commands before you type them in without semicolons (Remember, X and T are not defined)

 net = narxnet(10:15,10:15,10);
 view(net);
 isequal( net.IW{1} , net.IW{1,1} )
 isequal( net.IW{2} , net.IW{2,1} )
 isequal( net.IW{3} , net.IW{1,2} )
 isequal( net.IW{4} , net.IW{2,2} ) 

 net.IW
 isequal( net.IW , net.IW{:} )
 isequal( net.IW , net.IW{:,:} ) 
 isequal( net.IW , net.IW(:) ) 
 isequal( net.IW , net.IW(:,:) ) 

 net.IW{:}
 net.IW{:,:}
 net.IW(:)
 net.IW(:,:)

You can find he answers to a lot of questions merely by removing semicolons.

Especially revealing are the commands

 net = net

 tr = tr

where tr is the training history obtained from

 [ net tr ] = train(net,...)

Hope this helps.

Greg

Answer 3

[Greg Heath]

 ID = 10:15;  
 LID = length(ID)              % 6
 FD = 10:15;
 LFD = length(FD)              % 6
 H = 10

 isequal( Xi , [ X(1:15) ; T(1:15) ] )      % 1
 isequal( Xs , [ X(16:100) ; T(16:100) ] )  % 1
 isequal( Ts ,  T(16:100)  )                % 1

%Notice that the delay specification (10:15) causes data points (1:9) to be wasted

 [ O Ns ] = size(Ts)           % [ 1 85 ]
 Neq = Ns*O                    % 85  Output equations
 Nw = (LID+LFD+1)*H+(H+1)*O%141 Unknown weights to estimate from Neq equations

% NOTE: Nw > Neq ==> More unknowns than equations ==> Use regularization or validation stopping to mitigate overfitting

 net =  net                                % Omit semicolon for display
 numinputs         = net.numInputs         % 2 (Input & Feedback)
 numInputDelays    = net.numInputDelays    % 15 size(Xi,2)
 numFeedbackDelays = net.numFeedbackDelays % 0  open loop
 numWeightElements = net.numWeightElements % 141  Nw

%Before evaluating net performance, notice that there are three major reasons why this may not be an acceptable design:

%1. Choice of ID and FD without concern for locating the significant auto (T) and cross (X,T) correlation lags.

%2. Accepting the default division function 'dividerand' which tends to destroy whatever I/O correlations that may exist.

%3. Choice of H without an explicit concern for overfitting (although validation stopping is used)

 ts = cell2mat(Ts);
 MSE00 = mean((var(ts,1)))   % 0.1 Reference MSE
 MSE = perform(net,Ts,Ys)    % 0.0649
 NMSE = MSE/MSE00            % 0.6488
 R2 = 1-NMSE                 % 0.3512  Optimistic but poor

% Poor because the net only models 35% of the target variance (Prefer >= 95%)

% Optimistic because this includes 70% training data

% The breakdown into R2trn, R2val and R2tst can be obtained from the info in the training history tr

tr = tr

% I'll leave this as an excercise for the reader.

Hope this helps.

Greg