problem with time shift between target and simulation output using neural network

hi,

I'm currently working with neural networks and I'm still beginner. My purpose is to use a MLP to predict flow time series (I know, that NARX-networks may be more suitable for time series predictions, but the requirement is a MLP).

For example I want to predict the flow Q(t+x) with current and historical flow Q(t...t-n) and precipitation P(t...t-m) etc.

The results of my net-trainings (training, validation and test of the network) and an additional validation period show relatively good qualities (correlation and RMSE). This also corresponds to results from papers on this topic. But when I look closer at the output of training and validation period, there is a time shift to the targets of the respective periode. And my problem is that I don't know why. The papers I read don't tell anything about time shifts, however they partly use a simpler net-input than I.

The shift exactly corresponds to my forecast period x, no matter how large x is.

I use a standard MLP from the Matlab-toolbox with default settings; like using the graphical NN-tool (but I also tested other settings). With a simple Q(t) NAR-net it is the same problem. If I try it with regular data like predicting sin(t+x) with sin(t..t-n) or the same with [0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 0 0 ...] there is no time shift, it's all fine. Only if I use real world data or irregular (but most constant) data like [0.12 0.14 0.13 0.1 0.1 0.1 ... (n times) 0.1 ... 0.1 0.1 0.14 0.15 0.12 ...] there is the shift between the target and the output. Although I train the network with the target Q(t+x) the real training output is Q(t).

Is there something I am wrong in my work or something I can try. Had also someone such a problem? I think it is no failure of my implementation, because I also tried the matlab-tool and the sinus function and there are the same outcomes. Someone told me it can be a problem of presenting the input variables (like aggregation of time steps or using other time steps), but there are the paper which use the same input or more sparse input variables. And if I ignore the shift, the accuracy of the values is not bad (thats why the goodness of correlation and rmse is also good obviously).

I use matlab 2012.

I'd be glad to hear from you and if someone can help me!

best regards! Thomas

Answer 1

[Greg Heath]

There is no way we can help without seeing your code and example data.

Applying it to a MATLAB example data set would be most helpful.

 help nndatasets
 doc  nndatasets

Note: t,t-1,...t-n is backwards Could this be your problem?

Hope this helps.

Thank you for formally accepting my answer

Greg

Answer 2

[Greg Heath]

 load data.mat
 whos
% Name      Size         Bytes     Class      
%  data      43465x4   1390880  double

1. Life is MUCH easier if you transpose the data IMMEDIATELY!!!

    hidden_layer_size=15;

2. Why 15?

    %create net
    net = fitnet(hidden_layer_size);
    %created input and target    
        input=data(1:end,:);

3. Do you mean input = data ??

        %predicted for 1 day
        target = circshift(data(1:end,1), -1); % circularly shifts along the dimension 1
        target(end)=nan;
        %predicted for 2 days
        target = circshift(data(1:end,1), -2);
        target(end-1:end)=nan;
        %predicted for 4 days
        target = circshift(data(1:end,1), -4);
        target(end-3:end)=nan;

4. The above indexing is all screwed up. To understand, remove the ending semicolons and apply to

   data = [ 1:10 ;11:20 ; 21:30 ]'

    %train
    net = train(net, input', target');

5. Replace the LHS with

   [ net tr ] =...

6. Then, the training record is readily obtained using

 tr = tr

Hope this helps.

Thank you for formally accepting my answer

Greg

P.S. ALWAYS TEST YOUR CODE WITH A SMALL DATA SET!

[Nam]

Hi Greg, thanks. I answer your questions as below :

1. Life is MUCH easier if you transpose the data IMMEDIATELY!!! I have split the time series into trainning and test set

 trainning_set=data(1:4018,:); % specify trainning samples (10 years from 1980 to 1990 including leap days))
    testing_set = data(4018:5844,:);  % specify testing samples (5 years from 1991 to 1995 including leap days))

    target_set = circshift(trainning_set(:,1),-2);
    target_set(end-1:end)=nan;

    testing_target_set=circshift(testing_set(:,1),-2);
    testing_target_set(end-1:end)=nan;

2. Why 15? It is freely chosen. Could you suggest suitable value?

3. Do you mean input = data ?? Input is trainning time series. It has 4 input variables as Flow, Precipitation, Temperature and Evaporation corresponding to each column.

4. The above indexing is all screwed up. To understand, remove the ending semicolons and apply to I did not run all scripts, only run for each predicted period if you want

I have updated the scripts but it still has same problem ( shift the time corresponding to predicted time)

trainning_set=data(1:4018,:); % specify trainning samples (10 years from 1980 to 1990 including leap days))
  testing_set = data(4018:5844,:);  % specify testing samples (5 years from 1991 to 1995 including leap days))

    target_set = circshift(trainning_set(:,1),-2);
    target_set(end-1:end)=nan;

    testing_target_set=circshift(testing_set(:,1),-2);
    testing_target_set(end-1:end)=nan;

    % number of hidden neurons
    hidden_layer_size = 6;  % the described problem is not dependent on this point, therefor it is freely chosen
    net = fitnet(hidden_layer_size);

    [trained_net,tr] = train(net, trainning_set', target_set');    

    output = sim( trained_net, testing_set');

    %plot
    figure;    
    plot([testing_target_set, output'], 'linewidth',2);

Answer 3

[Nam]

Hi Greg, I have same problem as Thomas when I predicted the flow at t, t+2 and t+4 with current and history flow, temperature, precipitation and evapotranspiration at t to tn. Here is my code and data:

      hidden_layer_size=15;

    %create net
    net = fitnet(hidden_layer_size);

    %created input and target    
        input=data(1:end,:);

        %predicted for 1 day
        target = circshift(data(1:end,1), -1); % circularly shifts along the dimension 1
        target(end)=nan;

        %predicted for 2 days
        target = circshift(data(1:end,1), -2);
        target(end-1:end)=nan;

        %predicted for 4 days
        target = circshift(data(1:end,1), -4);
        target(end-3:end)=nan;

    %train
    net = train(net, input', target');    

    %predicted
    output = sim( net, input');

    figure;
    plot([target, output'], 'linewidth',2);
    legend('org', 'predicted');

Is there something I am wrong in my work? Please help us. Thanks Greg, Best,

Answer 4

[Thomas]

Hi, thanks for your answers!

Here's my code. It's a minimalistic example, only with the most import points. But also shows the problem very well.

%%minimalstic example
% but there is the same problem with more input variables

load Q

%%create net inputs and targets

% start point of t
t = 100; 

% history data of Q -> Q(t-1), Q(t-2), Q(t-3)
inputs = [Q(t-1:end-1,1) Q(t-2:end-2,1) Q(t-3:end-3,1)]';

% timestep t that want to be predicted
targets = Q(t:end,1)';

%%create fitting net (MLP) 
% but it is the same problem for NARnet
% and from here, you can also use the NN graphical tool

% number of hidden neurons
numHiddenNeurons = 6;  % the described problem is not dependent on this 
                       % point, therefor it is freely chosen

net = fitnet(numHiddenNeurons); % same problem if choosing the old version newfit

% default MLP settings, no changes, but the problem even exist with other
% combinations of settings

% train net
[trained_net,tr] = train(net,inputs,targets);

% apply trained net with given data (create net outputs)
outputs = sim(trained_net,inputs);

figure(1)
hold on
bar(targets',0.6,'FaceColor','r','EdgeColor','none')
bar(outputs',0.2,'FaceColor','b','EdgeColor','none')
legend('observation','prediction')
% please zoom very far to see single bars!! the bar plot shows very good
% the time shift
% if you choose a bigger forecasting time, the shift will also be better to
% see

%%the result: targets(1,1)=Q(t), outputs(1,1)=Q(t-1)

%%now try the sinus function, the problem will not be there

x = 1:1:1152;
SIN = sin(x);

inputs = [SIN(1,t-1:end-1);SIN(1,t-2:end-2);SIN(1,t-3:end-3)];
targets = SIN(1,t:end);

% start again from above, creating the net

I have also uploaded two excerpts of the results of the codes. One for the realworl data of Q and one of the sinus function (see first description of problem).

Q:

sinus:

best regards!

[Greg Heath]

You did not quantify your error!

What are the training, validation and testing errors?

[Thomas]

Hi Greg, I hope that's what you meant:

I take the outputs and targets from validation, test and training records via tr and then calculate error=mse(output-target). Is that right?

mse(trainerror) = 1.3085e-05

mse(validationerror) = 1.1622e-05

mse(testerror) = 7.5147e-05

thanks!

[Greg Heath]

They are also available from tr.best_perf, tr.best_vperf and tr.best_tperf.

[Image Analyst]

Thomas's answer moved here:

Hi, does no one have an idea?

Here is another and more general example presentation of the problem:

inputs

P(t) 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 ...

Q(t) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 ...

Q(t-1) -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 ...

target

Q(t+6) 6 7 8 9 10 11 12 13 14 15 16 ...

simulated output from trained net

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 ... --> that's Q(t), an input variable

best regards!

Thomas

[Greg Heath]

What is Q(t-1) when t=1?

[Thomas]

this example is for t > 1