Discover MakerZone

MATLAB and Simulink resources for Arduino, LEGO, and Raspberry Pi

Learn more

Discover what MATLAB® can do for your career.

Opportunities for recent engineering grads.

Apply Today

Thread Subject:
Forecast Verification/Accuracy toolbox

Subject: Forecast Verification/Accuracy toolbox

From: Nils Tobias

Date: 10 Jun, 2010 20:00:23

Message: 1 of 4

Hi Community,

at the moment I'm working on a project to evaluate the quality of forecasts and I was wondering if there's something similar to the verification package for weather forecast (Forecast verification utilities for R, http://cran.r-project.org/web/packages/verification/) but for Matlab?

I want to assess the quality of a single point forecast (if there is anything for probablistic forecasting I'm also interested) using measures like MAPE, RMSE, Theil's U, hit rate, correlation or any other measure like Brier score or Prognostic Horizon.

If there is anything that compares real values with prognostic values from a model and tries to measure the quality of the forecast performance I'd be happy to get a hint where to find.

Thanks in advance
Nils

Subject: Forecast Verification/Accuracy toolbox

From: BobC Cadenza

Date: 1 Sep, 2010 18:45:24

Message: 2 of 4

Did you ever find anything like this? I could use this for my project.

"Nils Tobias " <nils.kraemer@uni-ulm.de> wrote in message <hurg8n$ia1$1@fred.mathworks.com>...
> Hi Community,
>
> at the moment I'm working on a project to evaluate the quality of forecasts and I was wondering if there's something similar to the verification package for weather forecast (Forecast verification utilities for R, http://cran.r-project.org/web/packages/verification/) but for Matlab?
>
> I want to assess the quality of a single point forecast (if there is anything for probablistic forecasting I'm also interested) using measures like MAPE, RMSE, Theil's U, hit rate, correlation or any other measure like Brier score or Prognostic Horizon.
>
> If there is anything that compares real values with prognostic values from a model and tries to measure the quality of the forecast performance I'd be happy to get a hint where to find.
>
> Thanks in advance
> Nils

Subject: Forecast Verification/Accuracy toolbox

From: Nils Tobias

Date: 1 Sep, 2010 19:32:04

Message: 3 of 4

Unfortunately, I didn't get any response. So far I don't know of any toolbox or anything like the verification package for R. In Matlab you have tools like regression which you got in every statistics program. With that you can calculate the regression coefficient check alpha and beta of your regression (forecast against observation) and also the p-values for each coefficient. For contingency tables I couldn't find any functions that could calculate measures like Proportion Correct, or any other Skill Scores. There are some functions that deal with contingency tables (crosstab/confusionmatrix) however they aren't much of a help.

I wrote a simple m-file that calculates forecast accuracy measures like MAPE, MAE, RMSE and so on. Here's the code (any suggestions are welcome). D is the matrix with all the data. I chose and this is pretty specific for my project this order:
time | forecast1 | observation1 | benchmark1 | forecast2 | observation2 | benchmark2
So I have two forecast that I can evaluate at the same time but this is just because they have the same time. I guess it's easy to customize this to your needs. I think this function is quite flexible so you can calculate things like MAE, RMSE, MAPE, GMRAE, RelMSE, any Score like MAE-Score you just have to call this function more than once.

function [res1, res2] = ForecastAccuracy(D, s1, a, s2, s3)
%ForecastAccuracy Calculates forecast accuracy with user defined
% properties
% s1 can be 'rel' (relative), 'per' (percentage) or 'ben'
% for benchmark, in case of anything else (=do nothing)
% a is the exponent for the error
% s2 can be 'abs' (absolute) or anything else (=do nothing)
% s3 can be 'mean', 'median' or 'gmean'

if isempty(D)
    error('No data available in ForecastAccuracy');
else
    n = length(D);
    E = zeros(n,2); % initialize error matrix with 0
    for k = 1:n
        switch s1
            case 'per' % forecast error divided by observation
                e1 = (D(k,2) - D(k,3))/D(k,3);
                e2 = (D(k,5) - D(k,6))/D(k,6);
            case 'rel' % forecast error divided by benchmark forecast error
                e1 = (D(k,2) - D(k,3))/(D(k,4) - D(k,3));
                e2 = (D(k,5) - D(k,6))/(D(k,7) - D(k,6));
            case 'ben' % forecast error for benchmark
                e1 = (D(k,4) - D(k,3));
                e2 = (D(k,7) - D(k,6));
            otherwise
                e1 = (D(k,2) - D(k,3));
                e2 = (D(k,5) - D(k,6));
        end
        if strcmp(s2,'abs')
            e1 = abs(e1);
            e2 = abs(e2);
        end
        e1 = e1^a;
        e2 = e2^a;
        % fill error matrix with user defined properties a, s1 and s2
        E(k,1) = e1;
        E(k,2) = e2;
    end
    switch s3
        case 'mean'
            res1 = (mean(E)).^(1/a);
            res2 = res1(2);
            res1 = res1(1);
        case 'gmean'
            res1 = (geomean(E)).^(1/a);
            res2 = res1(2);
            res1 = res1(1);
        case 'median'
            res1 = (median(E)).^(1/a);
            res2 = res1(2);
            res1 = res1(1);
        otherwise
            error('Error in ForecastAccuracy! Wrong argument number 5.')
    end
end

I will program more measures as my project goes on, probably for contingency tables but anyways I won't be able to program as many measures as in the verification package for R, especially measures dealing with probability forecasts (ROC curves and so on) because I don't need them right now. But these function I guess can be easily transformed to matlab code.

-NTK

"BobC Cadenza" <bobc@mailinator.com> wrote in message <i5m704$nkl$1@fred.mathworks.com>...
> Did you ever find anything like this? I could use this for my project.
>
> "Nils Tobias " <nils.kraemer@uni-ulm.de> wrote in message <hurg8n$ia1$1@fred.mathworks.com>...
> > Hi Community,
> >
> > at the moment I'm working on a project to evaluate the quality of forecasts and I was wondering if there's something similar to the verification package for weather forecast (Forecast verification utilities for R, http://cran.r-project.org/web/packages/verification/) but for Matlab?
> >
> > I want to assess the quality of a single point forecast (if there is anything for probablistic forecasting I'm also interested) using measures like MAPE, RMSE, Theil's U, hit rate, correlation or any other measure like Brier score or Prognostic Horizon.
> >
> > If there is anything that compares real values with prognostic values from a model and tries to measure the quality of the forecast performance I'd be happy to get a hint where to find.
> >
> > Thanks in advance
> > Nils

Subject: Forecast Verification/Accuracy toolbox

From: Durga Lal Shrestha

Date: 19 Jan, 2011 05:08:04

Message: 4 of 4


You can also check this for deterministic problem.
http://www.mathworks.com/matlabcentral/fileexchange/7968-goodness-of-fit

Tags for this Thread

What are tags?

A tag is like a keyword or category label associated with each thread. Tags make it easier for you to find threads of interest.

Anyone can tag a thread. Tags are public and visible to everyone.

Contact us