Unfortunately, I didn't get any response. So far I don't know of any toolbox or anything like the verification package for R. In Matlab you have tools like regression which you got in every statistics program. With that you can calculate the regression coefficient check alpha and beta of your regression (forecast against observation) and also the pvalues for each coefficient. For contingency tables I couldn't find any functions that could calculate measures like Proportion Correct, or any other Skill Scores. There are some functions that deal with contingency tables (crosstab/confusionmatrix) however they aren't much of a help.
I wrote a simple mfile that calculates forecast accuracy measures like MAPE, MAE, RMSE and so on. Here's the code (any suggestions are welcome). D is the matrix with all the data. I chose and this is pretty specific for my project this order:
time  forecast1  observation1  benchmark1  forecast2  observation2  benchmark2
So I have two forecast that I can evaluate at the same time but this is just because they have the same time. I guess it's easy to customize this to your needs. I think this function is quite flexible so you can calculate things like MAE, RMSE, MAPE, GMRAE, RelMSE, any Score like MAEScore you just have to call this function more than once.
function [res1, res2] = ForecastAccuracy(D, s1, a, s2, s3)
%ForecastAccuracy Calculates forecast accuracy with user defined
% properties
% s1 can be 'rel' (relative), 'per' (percentage) or 'ben'
% for benchmark, in case of anything else (=do nothing)
% a is the exponent for the error
% s2 can be 'abs' (absolute) or anything else (=do nothing)
% s3 can be 'mean', 'median' or 'gmean'
if isempty(D)
error('No data available in ForecastAccuracy');
else
n = length(D);
E = zeros(n,2); % initialize error matrix with 0
for k = 1:n
switch s1
case 'per' % forecast error divided by observation
e1 = (D(k,2)  D(k,3))/D(k,3);
e2 = (D(k,5)  D(k,6))/D(k,6);
case 'rel' % forecast error divided by benchmark forecast error
e1 = (D(k,2)  D(k,3))/(D(k,4)  D(k,3));
e2 = (D(k,5)  D(k,6))/(D(k,7)  D(k,6));
case 'ben' % forecast error for benchmark
e1 = (D(k,4)  D(k,3));
e2 = (D(k,7)  D(k,6));
otherwise
e1 = (D(k,2)  D(k,3));
e2 = (D(k,5)  D(k,6));
end
if strcmp(s2,'abs')
e1 = abs(e1);
e2 = abs(e2);
end
e1 = e1^a;
e2 = e2^a;
% fill error matrix with user defined properties a, s1 and s2
E(k,1) = e1;
E(k,2) = e2;
end
switch s3
case 'mean'
res1 = (mean(E)).^(1/a);
res2 = res1(2);
res1 = res1(1);
case 'gmean'
res1 = (geomean(E)).^(1/a);
res2 = res1(2);
res1 = res1(1);
case 'median'
res1 = (median(E)).^(1/a);
res2 = res1(2);
res1 = res1(1);
otherwise
error('Error in ForecastAccuracy! Wrong argument number 5.')
end
end
I will program more measures as my project goes on, probably for contingency tables but anyways I won't be able to program as many measures as in the verification package for R, especially measures dealing with probability forecasts (ROC curves and so on) because I don't need them right now. But these function I guess can be easily transformed to matlab code.
NTK
"BobC Cadenza" <bobc@mailinator.com> wrote in message <i5m704$nkl$1@fred.mathworks.com>...
> Did you ever find anything like this? I could use this for my project.
>
> "Nils Tobias " <nils.kraemer@uniulm.de> wrote in message <hurg8n$ia1$1@fred.mathworks.com>...
> > Hi Community,
> >
> > at the moment I'm working on a project to evaluate the quality of forecasts and I was wondering if there's something similar to the verification package for weather forecast (Forecast verification utilities for R, http://cran.rproject.org/web/packages/verification/) but for Matlab?
> >
> > I want to assess the quality of a single point forecast (if there is anything for probablistic forecasting I'm also interested) using measures like MAPE, RMSE, Theil's U, hit rate, correlation or any other measure like Brier score or Prognostic Horizon.
> >
> > If there is anything that compares real values with prognostic values from a model and tries to measure the quality of the forecast performance I'd be happy to get a hint where to find.
> >
> > Thanks in advance
> > Nils
