Electricity Load and Price Forecasting Webinar Case Study

Slides and MATLAB® code for the day-ahead system load and price forecasting case study.

Description

** Update: The webinar recording is available at:
http://www.mathworks.com/company/events/webinars/wbnr51423.html

This example demonstrates building a short term electricity load (and price) forecasting system with MATLAB®. Two non-linear regression models (Neural Networks and Bagged Regression Trees) are calibrated to forecast hourly day-ahead loads given temperature forecasts, holiday information and historical loads. The models are trained on hourly data from the NEPOOL region (courtesy ISO New England) from 2004 to 2007 and tested on out-of-sample data from 2008.

The application includes an (optional) Excel front end which enables users to call the trained load forecasting models through a MATLAB-deployable DLL.

The document titled "Introduction to Load & Price Forecasting Case Study" will guide you through the different components of the analysis.

If you do not have all of the required toolboxes, you can still view the results of running the analysis by clicking on one of the HTML reports below.

NOTE: The Access database shown in the webinar is not provided with this archive due to size restrictions. The equivalent data sets are provided in MAT-files in the folders Load\Data and Price\Data for the load and price forecasting studies respectively. The raw data files can be obtained directly from ISO New England (www.iso-ne.com)

MORE ON LOAD AND PRICE FORECASTS:
Accurate load forecasts are critical for effective operations and planning for utilities. The load forecast influences a number of decisions including which generators to commit for a given period, and acutely affects the wholesale electricity market prices. Load and price forecasting algorithms typically also feature prominently in reduced-form hybrid models for electricity price, which are some of the most accurate models for simulating markets and modeling energy derivatives. The electricity price forecast is also used widely by market participants in many trading and risk management applications.

• Required Products: Database Toolbox; MATLAB Builder EX; MATLAB Compiler; Neural Network Toolbox; Statistics Toolbox
• MATLAB release: MATLAB 7.10 (2010a)

[Ashraf Ul Haque]

Hi, don't get energy.accdb file

[Ashraf Ul Haque]

Could you kindly advise me how can I get energy.accdb file?
Thanks for your cooperation.

[Ameya Deoras]

I have updated the submission to include the relevant data in MAT-files so you shouldn't need the Access database anymore. The Intro document has also been updated to reflect this.

[Jesse Hill]

Would you be able to provide some of the sources/authors that contribute to the background to the code?

Thank you for your help

[Ameya Deoras]

Thanks for your question. The methodology in this example isn't taken from any one source. There are a couple of papers I read which gave me some ideas. Please also check the references within.

* Eugene Feinberg & Dora Genethliou, Applied Mathematics for Power Systems, Chapter 12 Load Forecasting

* K. Liu et al, "Comparison of Very Short-Term Load Forecasting Techniques", IEEE Transactions on Power Systems Vol 11 No 2, May 1996

* Rafal Weron, "Modeling and Forecasting Electricity Loads and Prices" Wiley

[Sam]

I wish to get some helps from you to test the running the function in the spreadsheet.

I have compiled your Forecaster.prj and imported the corresponding .bas files into the Forecaster.xlsm. But when I tried it by clicking the "Compute Forecast" button, it showed the run-time errors in the forecast cells, as shown below:

Error in Forecaster.Class1.1_0: Undefined function or method 'sim' for input arguments of type 'struct'.
Error in => loadForecast.m at line 56

The folder where the sim.m situated was already in my Matlab path when I checked through the Set Path command.

I also noticed that you had indicated in the loadForecast.m the paths for looking up the NN and tree models as well as the data file.

Could you please kindly provide some hints to pin point and solve the problem?

[Sam]

Ameya,

I wish to supplement one more point regarding the problem mentioned above: I could successfully run the loadForecast.m in Matlab and generate the forecast.

So, I wonder if there was something wrong with my setups in the operating system or in the Visual Studio environment at the compile-time and run-time, e.g., the path or whatsoever possible.

I look forward to your hints and helps. Thanks.

[kemal ozsari]

Ameya, your data also has time series property. So is it possible to have a model that is using time-delay dynamic networks? Also may be narx network, with multiple inputs and past y data. If possible and you update your webinar with new models, it will be great. Thx.

[Ameya Deoras]

Thanks for the feedback. You can certainly use time-delay networks for such an application. In this example, I manually create time-lagged versions of the response (load) and use them as variables in the predictor matrix. I use the 168-hour lagged load, the 24-hour lagged load and a smoothed or averaged 24-hour lagged load.

[Jesse Hill]

For the purposes of the 'long' term forecasting, is the period any different than the short term's period? Are we really only changing the factors used to build the model in order to forecast the future load?

In addition, is the temperature data that was provided in the test set a forecast or the historical data? I see that the slides/pdf show that we would be using forecasts, but I wasn't sure if that was the case on the provided data.

Thank you,

Jesse

[Ameya Deoras]

For long term forecasting, you may need to take into account additional predictors such as the economic growth in an area and new industrial or residential developments. The existing model could be modified to remove the lagged load dependence. This would give you a model that would generate a forecast based only on temperature, date/time and holiday information. You could then use it either in a Monte-Carlo fashion or for different scenarios of future temperatures (average, 1 std dev, 2 std dev etc) to create a profile or distribution of load forecasts. Check out my other submission "Energy Trading & Risk Management with MATLAB" for an example of a temperature simulation model.

The temperature data in provided is historical. If you were to use to model to make a forecast, you would use forecast temperature values. In my example, I pretend that today is April 2, 2009 and use actual temperatures for the next day as my assumed temperature forecast. So, only actual temperatures and not forecasts are being used to create the load forecast.

[Hamed]

Hi; I want to run this load forecast without using Excel front page. Can you please identify what is the process? In "LoadForecast.m" file at line 46 I am receiving error showing that "temperature" is not defined. How should I define this temperature?

[Ameya Deoras]

The second input to loadForecast is the temperature forecast for the day for which you want to predict the load. The second requirement is that the day you choose must have some historical data in the database for the week prior. You will therefore also need the database installed. You can contact me to get access to the access database used for this example. Once the database is set up, you can call loadForecast as follows:

y = loadForecast('April 2, 2008', temp, 'No');
where temp is a 24-by-2 array of hourly temperature and dew point forecasts for the day in question.
temp = [38 36 36 36 35 34 35 35 36 38 40 41 42 43 44 44 44 43 42 41 40 40 40 40;31 30 30 30 30 29 30 31 32 32 32 32 32 32 32 32 34 35 36 37 38 38 38 38]';

[kami]

dear ameya
would you please send me the example of this program .includ "temp,holiday,price"

[kami]

please be concider simple example without excel file

[fafa]

Dear ameya
i'm trying to run the price forecast program,but i receive errors,would you please send me an example of this program.

[Ameya Deoras]

Hi. A simple example call to loadForecast has been provided in a previous post. If you need something more, could you please describe it in more detail?

Here is the example again:

% Temperature forecast
temp = [38 36 36 36 35 34 35 35 36 38 40 41 42 43 44 44 44 43 42 41 40 40 40 40;31 30 30 30 30 29 30 31 32 32 32 32 32 32 32 32 34 35 36 37 38 38 38 38]';

% Load forecast for that temperature forecast for April 2, 2008
y = loadForecast('April 2, 2008', temp, 'No');

[Ali]

Ameya,
When I tried to run the fun (genPredictors.m) it showed the errors as shown below:
??? Input argument "data" is undefined.
Error in ==> genPredictors at 27
    dates = datenum(data.Date, 'yyyy-mm-dd HH:MM:SS') + (data.Hour-1)/24;
Could you please kindly solve the problem?

[Thanh]

Dear Ameya,
I have problem with command:

data = fetchDBLoadData('2004-01-01','2008-12-31')

This is the dislay after entering the command:

-----------
??? Error using ==> chkprops at 18
Invalid com.mathworks.toolbox.database.PrefsDBPanel property: ' DefaultRowPreFetch '.

Error in ==> setdbprefs at 91
  p = chkprops(DatabasePrefs,p,prps);

Error in ==> setdbprefs at 88
    setdbprefs(flds,vals)

Error in ==> fetchDBLoadData at 14
setdbprefs(s)
------
That's all. Could you guide me how to treat it?

Thank you very much!

[Jack]

How do we make a prediction with the Matlab generated script above ?

The example should have shown that ... really !!!

[Jack]

The problem with this example it is that Ameya doesn't show how to make a multi step prediction.

This should have been it the example because the title is FORECASTING.

I see a lot or questions everywhere on the web about multi step ahead prediction.

Everyone know how to load datas. I means it's almost like opening an Excel file. But to make a prediction is very difficult.

I've a NARX neural network with 10 hidden neurons and 2 delays. As input I have a 510x5 (called Inputx) and as output I have a 510x1 (called Target).

I want to forecast 10 days ahead but it's really not working...

I tried the following code but I'm stuck now. :-(

Would you mind to help me ? Some code will be awesome. :-(

***////////////////////////////////////////////******** ***/////////////////////////////////////////// ******

inputSeries = tonndata(Inputx,false,false);

targetSeries = tonndata(Target,false,false);

netc = closeloop(net);

netc.name = [net.name ' - Closed Loop'];

[xc,xic,aic,tc] = preparets(netc,inputSeries,{},targetSeries);

yc = netc(xc,xic,aic);

***////////////////////////////////////////////******** ***/////////////////////////////////////////// ******

[Ameya Deoras]

Here is a function that can be used to create an arbitrary predictor matrix with any number of lags:

function mat = createLags(vec, lags)
% CREATELAGS generates lagged versions of an input vector or matrix to
% generate a predictor matrix.
%
% USAGE:
% predictorMatrix = createLags(series, lags)
%
% Here series is a numObs-by-numDim matrix of observations. If numDim > 1,
% it implies the input series is a multidimensional series. lags is a
% vector of integer lags where 0 corresponds to no lag, +1,+2,+3... correspond to
% lags of 1,2,3... steps, and -1,-2,-3 correspond to "leads" of 1,2,3
% steps. predictorMatrix is a numObs-by-numDim*numLags matrix of the
% shifted versions of the input matrix

% x = [1 2 3 4; -1 -2 -3 -4]'
% y = createLags(x, [-1 0 2])

[numObs, numDim] = size(vec);
numLags = length(lags);
mat = NaN(numObs, numDim * numLags);
for i = 1:length(lags)
    mStaInd = max(1, lags(i)+1);
    mEndInd = min(numObs, lags(i)+numObs);
    vStaInd = max(1, 1-lags(i));
    vEndInd = min(numObs, numObs-lags(i));
    
    mat(mStaInd:mEndInd, (i-1)*numDim+1:i*numDim) = vec(vStaInd:vEndInd,:);
end

[Ameya Deoras]

Here is a function that can be used to run multi-step prediction from a 1-step feed-forward network:

function pred = multiPredict(net, X, numSteps)
% multiPredict performs iterated prediction with a feed-forward neural
% network.
%
% SYNTAX: pred = multiPredict(net, X, numSteps)
%
% INPUTS:
% net is a feed-forward neural net
% X is a matrix of size numObservations * numPredictors
% numSteps is the number of steps to predict (eg. 10)
%
% OUTPUTS:
% pred is a matrix of size numObservations * numSteps where every column is
% the ith step prediction for that sample observations. The first column is
% the 1-step prediction, the second columns is the 2-step prediction and so
% on.

numObs = size(X,1);
pred = zeros(numObs, numSteps);
pred(:, 1) = net(X')';
for i = 2:numSteps
    X = [pred(:,i-1) X(:,1:end-1)];
    pred(:,i) = net(X')';
end

[Ameya Deoras]

Jack, there are two places where prediction is shown. In loadForecastNN, look for the line "forecastLoad = sim(net, testX')';". Please also take a look at my comment from 20 May 2011 for an example on using the function forecastLoad to create a prediction.

There are 3 ways of doing multi-step prediction:
1. You can build a NARX network. There are examples in the documentation that show how to do multi-step with such a network.
2. You can build a feedforward network designed for predicting N steps ahead. That is what I do in this example - build a network for 24 step-ahead prediction. This is done by lagging the inputs 24 observation. The function createLags above will do this for you.
3. Run the prediction in a loop. The function multiPredict above does this for a 1-step network. In the context of this example, you can use the N-step model to do 2N, 3N, 4N... step prediction.

While I don't think the lack of multi-step prediction warrants a 1-star rating, you are of course entitled to rate it based on whatever criteria make sense to you. This is not intended to be a tutorial on Neural Networks Toolbox so I can't possibly include everything there is to know about using it. I encourage you to consult the documentation or contact technical support for help with your specific needs.

[Joseph]

This case study has been really helpful, thanks. I am having a problem with changing my network to a time-series forecast tool in which I can feed the current load back through the network to forecast the next day, or possibly the next few hours. It fits the load very well, but due to the complexity in the input preparation I can't figure out how to use it with the sim function (in my regular network I could just use what you did, i.e. sim(net,inputs'), but this won't work with the time-series setup)to actually predict new data coming in. My inputs are similar to your case study. Do you have any recommendations for preparing new inputs for a predicting function in this setup? I have played around with the network code to try and mimic what it does to the inputs, but no luck.

[Ali]

Thank you for this wonderful work,
I've increased the factors like RELATIVE HUMIDITY ,WIND SPEED, but it showed the errors as shown below:

Error using ==> network.sim at 178
Inputs are incorrectly sized for network.
Matrix must have 8 rows.

Error in ==> LoadScriptNN at 94
forecastLoad = sim(net, testX')';

Could you guide me how to treat it?
Thank you very much!

[Cristóbal Samudio]

Suposing that is necessary generate multiple demands time series forecast with the demand model made ​​with the Neural Network Tool, this would be possible??
Thinking specifically of the demand model presented, this would be possible with a single predictor scenario (temperature for exaple).
If it were possible to generate multiple paths to the model hourly demand made​​, which would be the commands to use?

[Cristhiano Moreno]

Hello.

How can i do it with a NAR Network ?
I have a temporal serie with 99 values, and I want to predict the 100 value. And finally plot this 2 series y the same figure.

Somebody can help me, I prefer with the script.

Thk u

[Bine]

 

[Erdal Bizkevelci]

 

[Satya]

Hi Ameya, When I try to use this forecaster, NeuralNet forecasts to Zero load all the day, However the other model shows the forecast load. I tried for several days all the time it is the same case where NN shows zero load. Is there anything that I can do to get NN work?

[Izi]

Ameya,
I have the same problem like Ali in May 2011. Solution?

The fun (genPredictors.m) it showed the errors as shown below:
??? Input argument "data" is undefined.
Error in ==> genPredictors at 27
    dates = datenum(data.Date, 'yyyy-mm-dd HH:MM:SS') + (data.Hour-1)/24;
Could you please kindly solve the problem?

[Ameya Deoras]

Satya, is the Neural Network predicting 0 load in MATLAB or in the Excel spreadsheet? If it is fine in MATLAB you may need to investigate if the model is being loaded correctly in the Excel side.

[Ameya Deoras]

Izi, that error seems to suggest that you are calling the genPredictors function without any inputs. “data” is a required input for the function and needs to be specified. See loadScriptNN for an example of how to use this function.