How to calculate confidence intervals with neural network prediction?

18 views (last 30 days)
Hello, I would like to present my neural network result with confidence intervals. Because I think interval estimation is better than point estimation. Correct me if i am wrong. I heard that it can be finished with "Neural Network Time Series Prediction". And can I make it with function fitting(fitnet)? or present with prediction interval instead of?
My neural network function is following:
% Solve an Input-Output Fitting problem with a Neural Network
% Script generated by Neural Fitting app
% Created 29-Oct-2015 00:45:44
%
% This script assumes these variables are defined:
%
% input - input data.
% target - target data.
x = input';
t = target';
% Choose a Training Function
% For a list of all training functions type: help nntrain
% 'trainlm' is usually fastest.
% 'trainbr' takes longer but may be better for challenging problems.
% 'trainscg' uses less memory. Suitable in low memory situations.
trainFcn = 'trainlm'; % Levenberg-Marquardt backpropagation.
% Create a Fitting Network
hiddenLayerSize = 9;
net = fitnet(hiddenLayerSize,trainFcn);
% Choose Input and Output Pre/Post-Processing Functions
% For a list of all processing functions type: help nnprocess
net.input.processFcns = {'removeconstantrows','mapminmax'};
net.output.processFcns = {'removeconstantrows','mapminmax'};
% Setup Division of Data for Training, Validation, Testing
% For a list of all data division functions type: help nndivide
net.divideFcn = 'dividerand'; % Divide data randomly
net.divideMode = 'sample'; % Divide up every sample
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
% Choose a Performance Function
% For a list of all performance functions type: help nnperformance
net.performFcn = 'mse'; % Mean Absolute Percentage Error
% Choose Plot Functions
% For a list of all plot functions type: help nnplot
net.plotFcns = {'plotperform','plottrainstate','ploterrhist', ...
'plotregression', 'plotfit'};
% Train the Network
[net,tr] = train(net,x,t);
% Test the Network
y = net(x);
e = gsubtract(t,y);
performance = perform(net,t,y)
% Recalculate Training, Validation and Test Performance
trainTargets = t .* tr.trainMask{1};
valTargets = t .* tr.valMask{1};
testTargets = t .* tr.testMask{1};
trainPerformance = perform(net,trainTargets,y)
valPerformance = perform(net,valTargets,y)
testPerformance = perform(net,testTargets,y)
% View the Network
view(net)
% Calculate Mean Absolute Percentage Error
err = t-y;
errpct = abs(err)./t*100;
MAE = mean(abs(err));
MAPE = mean(errpct(~isinf(errpct)));
figure
plot(err);
title(sprintf('MAPE: %0.3f%%',MAPE))
xlabel('Days'); ylabel('Error between Actual and Forecasting');
figure
plot(y,'DisplayName','Forecasting','LineStyle','--');hold on;plot(target,'DisplayName','Actual');hold off;
title('Comparing of Actual Customers and Forecasting Customers')
xlabel('Days'); ylabel('Number of Customers');
% Plots
% Uncomment these lines to enable various plots.
%figure, plotperform(tr)
%figure, plottrainstate(tr)
%figure, ploterrhist(e)
%figure, plotregression(t,y)
%figure, plotfit(net,x,t)
% Deployment
% Change the (false) values to (true) to enable the following code blocks.
% See the help for each generation function for more information.
if (true)
% Generate MATLAB function for neural network for application
% deployment in MATLAB scripts or with MATLAB Compiler and Builder
% tools, or simply to examine the calculations your trained neural
% network performs.
genFunction(net,'myNeuralNetworkFunction');
y = myNeuralNetworkFunction(x);
end
if (false)
% Generate a matrix-only MATLAB function for neural network code
% generation with MATLAB Coder tools.
genFunction(net,'myNeuralNetworkFunction','MatrixOnly','yes');
y = myNeuralNetworkFunction(x);
end
if (false)
% Generate a Simulink diagram for simulation or deployment with.
% Simulink Coder tools.
gensim(net);
end
Thanks alot.

Accepted Answer

Greg Heath
Greg Heath on 10 Nov 2015
When my work has required prediction intervals I have just used
[ y - sqrt(mse) , y + sqrt(mse) ]
where
y = net(x)
Hope this helps.
Thank you for formally accepting my answer
Greg
PS Your following label is incorrect:
net.performFcn = 'mse'; % Mean Absolute Percentage Error
  7 Comments
Alex
Alex on 8 Aug 2016
Thank you Greg for the answer. But, I got some queries on that. While, y is an array and mse is a number, how is it possible to calculate the PI? Or, maybe I'm missing something.
P.S.: Sorry for my ignorance.
Salma Hassan
Salma Hassan on 25 Jun 2020
what about this code:
x= predicted score;
CI= mean (x) +- 1.96*standard error
another question does the confident interval should be between 0 and 1. and if i got (0.45,0.67) what does it mean?

Sign in to comment.

More Answers (0)

Categories

Find more on Sequence and Numeric Feature Data Workflows in Help Center and File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!