Reliability Diagram for calibration of two-class predictor.

Plot reliability diagram for calibration purposes using predictions and targets vectors.

Description

PLOTRELIABILITY(TARGETS, PREDICTIONS) uses TARGETS and PREDICTIONS vectors to compute TARGETS_MEANS and PREDICTIONS_MEANS. FIG is a rudimentary reliability diagram plotted using these means.

Values of TARGETS are either 0 or 1, representing the two classes. Values of PREDICTIONS are probabilities of the positive class (1), previously obtained using a probabilistic predictive model. TARGETS and PREDICTIONS are row or column vectors of the same size.

The predictions are discretized into ten bins. Cases with predicted value between 0 and 0.1 fall in the first bin, between 0.1 and 0.2 in the second bin, etc. For each bin, the mean predicted value (PREDICTIONS_MEANS) and the true fraction of positive cases (TARGETS_MEANS) are computed. These points are then plotted on the X and Y axes respectively. If the model is well calibrated, the points will fall near the diagonal line.

SHOW_FIG is an optional logical input argument. Its default value is true. Setting it to false prevents the automatic display of the reliability diagram.

Syntax:
plotreliability(targets, predictions);
[fig, targets_means, predictions_means] = plotreliability(targets, predictions, false);

Remarks:
This function does not allow the number of bins or the width of each bin to be changed.

References:
Alexandru Niculescu-Mizil and Rich Caruana (2005) Predicting Good Probabilities With Supervised Learning, in Proceedings of the 22nd International Conference on Machine Learning. See section 4 (Qualitative Analysis of Predictions).

• MATLAB release: MATLAB 7.9 (2009b)
• Other Files: license.txt, ; plotreliability.m