NARX net cannot predict nonlinear dynamical system

Hi All, I'm trying to fit a theta -neuron model with narxnet in matlab. The model is given by:

theta_dot = (1 - cos theta)/tau + u *(1 + cos theta) ;

I have trained the network with 5 hidden layers and 10^6 data points (data generated by random noise). The training results are ok but when it comes to prediction the Neural Network performs horribly wrong. I'm attaching the code and training and prediction performance. Any help is appreciated thanks! Using Matlab 2015a on Windows 10.

      %%Fitting a Theta Neuron with a neural network

    close all; clear ; clc

    % theta_dot = (1 - cos theta)/tau + u *(1 + cos theta) ;
    % discrete case theta(k+1) = theta(k)+ theta_dot *dt

    rng(1)

    tau = 1;                                      % Model parameters

    sigma_u = 20 ; mean_u = 0.5;                    % varince in input (gaussian noise)

    time_steps = 1e6 + 1;  dt = 1e-3;             % Simulation time and interval in ms

    t_vec =  (0 : time_steps -1) * dt;            

    u_t = mean_u + sqrt(sigma_u) * randn(time_steps,1);     % Training Input

    theta_t = zeros(time_steps,1);                 % Initialization

    % Euler integration for the theta neuron model

    for nn = 1 : time_steps -1

        theta_t(nn + 1) =  theta_t(nn) + ((1 - cos(theta_t(nn)))/tau + u_t(nn)* (1 + cos(theta_t(nn))))* dt;

    end

    x_t = theta_t;                              % Rename to keep the naming convention

    subplot(4,1,1)

    plot(t_vec , sin(x_t))                      % Plot the evolution of sin(theta) which indicates spike when theta = 2*pi

    %%Create and Train the Neural Network

    % preparing the training data

    training_set = (time_steps -1) * 0.8 ;                       % Use 80% as the training set and 20% for the prediction

    u_t_train = u_t(1 : training_set);          % Setting the training data for the recurrent neural network

    x_t_train = x_t(1 : training_set);

    x_t_train = con2seq(x_t_train');            % Changing the datset to the appropriate datatype
    u_t_train = con2seq(u_t_train');

    % create the neural network

    d1 = 1;                                     % # of input delays
    d2 = 1;                                     % # of feedback delays
    hidden = 5;                                 % # of hidden layers

    narx_net = narxnet(d1,d2,hidden);           % Create a Nonlinear autoregressive neural network with external input

    narx_net.divideFcn = '';                    % No dividing of traing data
    narx_net.trainParam.min_grad = 1e-10;

    % Preparing the shifted data

    [shifted_u_t, input_delay_states, Ai, shifted_x_t] = preparets(narx_net, u_t_train,{}, x_t_train); % Non Feedback Targets is empty

    narx_net = train(narx_net, shifted_u_t, shifted_x_t, input_delay_states);          % train the series parallel model

    % view(narx_net)                                                            % Visualize the Network

    subplot(4,1,2)

    process_sim = sim(narx_net, shifted_u_t, input_delay_states);        % Simulate the neural network
    error_sim= cell2mat(process_sim)-cell2mat(shifted_x_t);

    plot(error_sim)

    subplot(4,1,3)                          % plotting an overlay of training data and the fitting of the NN
    TS = size(shifted_x_t, 2);
    plot(1:TS, sin(cell2mat(shifted_x_t)),'*-b', 1:TS, sin(cell2mat(process_sim)),'--r')

    % Close the neural network loop after Sufficient Training

    narx_net_closed = closeloop(narx_net);

    % view(narx_net_closed)             % Visualizing the Parallel Identification Model

    prediction_set = (time_steps -1) * 0.2;             % 20% data for the prediction

    u_t_predict = u_t(training_set + 1 : training_set + prediction_set);

    x_t_predict = x_t(training_set + 1 : training_set + prediction_set);

    x_t_predict = con2seq(x_t_predict');     % Changing the datset to the appropriate datatype
    u_t_predict = con2seq(u_t_predict');

    [shifted_u_p, input_delay_states1, Ai1,shifted_x_p] = preparets(narx_net_closed,u_t_predict,{},x_t_predict);
    process_prediction = narx_net_closed(shifted_u_p, input_delay_states1, Ai1);
    TS1 = size(shifted_x_p, 2);

    subplot(4,1,4)                          % Plotting an overlay of actual data and prediction of NN
    plot(1:TS1, sin(cell2mat(shifted_x_p)),'*-b', 1:TS1, sin(cell2mat(process_prediction)),'--r')

[Greg Heath]

I think I have solved this problem I will check my files.

Greg

[Greg Heath]

No. I didn't. Will take a look at it.

However, with N = 1e4+1 instead of 1e6+1.

Taking more time than I expected. Your notation is not helpful (in fact, pretty dreadful).

Taking longer than expected to change to my notation.

Why the random input?

P.S. 1 hidden layer with 5 hidden nodes.

Did you realize that the explicit separation into training and prediction subsets is unnecessary if you use DIVIDEBLOCK?

Answer 1

[Greg Heath]

My Normalized OL and CL MSE results for the training data are

 NMSE = mse(error)/mean(var(target',1))

 NMSEtrno = 2.7895e-07
 NMSEtrnc = 0.27786
 ratio    = 9.961e+05

Since the NMSE for the CL net on training data is

     A MILLION TIMES LARGER !!!

than for the OL net, there is no hope for accurate predictions on nontraining data unless you

    TRAIN THE CL NET ON TRAINING DATA!

I have not done this yet.

Hope this helps.

Thank you for formally accepting my answer

Greg