perfcurve - Compute Receiver Operating Characteristic (ROC) curve or other performance curve for classifier output

Syntax

[X,Y] = perfcurve(labels,scores,posclass) [X,Y] = perfcurve(labels,scores,posclass,'param1', val1,'param2',val2,...) [X,Y,THRE,AUC,OPTROCPT,SUBY,SUBYNAMES] = perfcurve(labels,scores,posclass)

Description

[X,Y] = perfcurve(labels,scores,posclass) computes a ROC curve for a vector of classifier predictions scores given true class labels, labels. labels can be a numeric vector, logical vector, character matrix, cell array of strings or categorical vector. scores is a numeric vector of scores returned by a classifier for some data. posclass is the positive class label (scalar), either numeric (for numeric labels), logical (for logical labels), or char. The returned values X and Y are coordinates for the performance curve and can be visualized with plot(X,Y). For more information on labels, scores, and posclass, see Inputs . For more information on X and Y, see Outputs.

[X,Y] = perfcurve(labels,scores,posclass,'param1', val1,'param2',val2,...) specifies optional parameter name/value pairs. See Inputs for a list of inputs, parameter name/value pairs, and respective explanations.

[X,Y,THRE,AUC,OPTROCPT,SUBY,SUBYNAMES] = perfcurve(labels,scores,posclass) returns:

An array of thresholds on classifier scores for the computed values of X and Y (THRE).
The area under curve (AUC) for the computed values of X and Y.
The optimal operating point of the ROC curve (OPTROCPT).
An array of Y values for negative subclasses (SUBY).
A cell array of negative class names (SUBYNAMES).

For more information on each output, see Outputs.

Inputs

`labels`	`labels` can be a numeric vector, logical vector, character matrix, cell array of strings or categorical vector.
`scores`	`scores` is a numeric vector of scores returned by a classifier for some data. This vector must have as many elements as `labels` does.
`posclass`	`posclass` is the positive class label (scalar), either numeric (for numeric `labels`) or char. The specified positive class must be in the array of input labels.

The following name/value pairs are available for the syntax [X,Y] = perfcurve(labels,scores,posclass,'param1', val1,'param2',val2,...)

Name ('param') Value (val) and Description

'negClass' List of negative classes. Can be either a numeric array or an array of chars or a cell array of strings. By default, negClass is set to 'all' and all classes found in the input array of labels that are not the positive class are considered negative. If negClass is a subset of the classes found in the input array of labels, instances with labels that do not belong to either positive or negative classes are discarded.

'xCrit'

Criterion to be computed for X. perfcurve supports the following criteria:

TP — Number of true positive instances.
FN — Number of false negative instances.
FP — Number of false positive instances.
TN — Number of true negative instances.
TP+FP — Sum of TP and FP.
RPP — Rate of positive predictions. RPP=(TP+FP)/(TP+FN+FP+TN)
RNP — Rate of negative predictions. RNP=(TN+FN)/(TP+FN+FP+TN)
accu — Accuracy. accu = (TP+TN)/(TP+FN+FP+TN)
TPR, sens, reca — True positive rate, sensitivity, recall. TPR, sens, reca = TP/(TP+FN)
FNR, miss — False negative rate, miss. FNR,miss=FN/(TP+FN)
FPR, fall — False positive rate, fallout. FPR,fall=FP/(TN+FP)
TNR, spec — True negative rate, specificity. TNR,spec=TN/(TN+FP)
PPV, prec — Positive predictive value, precision. PPV,prec=TP/(TP+FP)
NPV — Negative predictive value. NPV=TN/(TN+FN)
ecost — Expected cost. ecost=(TP*COST(P|P)+FN*COST(N|P)+FP* COST(P|N)+TN*COST(N|N))/(TP+FN+FP+TN)

In addition, you can define an arbitrary criterion by supplying an anonymous function of three arguments, (C,scale,cost), where C is a 2-by-2 confusion matrix, scale is a 2-by-1 array of class scales, and cost is a 2-by-2 misclassification cost matrix.

Caution Some of these criteria return NaN values at one of the two special thresholds, 'reject all' and 'accept all'.

'yCrit' Criterion to compute for Y. perfcurve supports the same criteria as for X.

'xVals' Values for the X criterion. The default value for xVals is 'all' and perfcurve computes X and Y values for all scores. If the value for xVals is not 'all', it must be a numeric array. In this case, perfcurve computes X and Y only for the specified xVals.

'processNaN' Specifies how perfcurve processes NaN scores. The default value is 'ignore' and perfcurve removes instances with NaN scores from the data. If you set the parameter to 'addtofalse', perfcurve adds instances with NaN scores to false classification counts in the respective class. That is, perfcurve always counts instances from the positive class as false negative (FN), and always counts instances from the negative class as false positive (FP).

'prior' Either string or array with two elements. It represents prior probabilities for the positive and negative class, respectively. Default is 'empirical', that is, perfcurve derives prior probabilities from class frequencies. If set to 'uniform', perfcurve sets all prior probabilities equal.

'cost' A 2-by-2 matrix of misclassification costs [C(P|P) C(N|P); C(P|N) C(N|N)], where C(I|J) is the cost of misclassifying class J as class I. The default value is [0 0.5; 0.5 0].

Outputs

`X`	x-coordinates for the performance curve. By default, `X` is false positive rate, FPR, (equivalently, fallout, or 1–specificity). To change this output, use the `'xCrit'` name/value input. For accepted criterion, see `'xCrit'` in Inputs for more information.
`Y`	y-coordinates for the performance curve. By default, `Y` is true positive rate, TPR, (equivalently, recall, or sensitivity). To change this output, use the `'yCrit'` input. For accepted criterion, see `'xCrit'` in Inputs for more information.
`THRE`	An array of thresholds on classifier scores for the computed values of `X` and `Y`. It has the same number of rows as `X` and `Y`. For each threshold, `TP` is the count of true positive observations with scores greater or equal to this threshold, and `FP` is the count of false positive observations with scores greater or equal to this threshold. `perfcurve` defines negative counts, `TN` and `FN`, in a similar way then sorts the thresholds in the descending order which corresponds to the ascending order of positive counts. For the `M` distinct thresholds found in the array of scores, `perfcurve` returns the `X`, `Y` and `THRE` arrays with `M+1` rows. `perfcurve` sets elements `THRE(2:M+1)` to the distinct thresholds, and `THRE(1)` replicates `THRE(2)`. By convention, `THRE(1)` represents the highest `'reject all'` threshold and `perfcurve` computes the corresponding values of `X` and `Y` for `TP=0` and `FP=0`. `THRE(end)` is the lowest `'accept all'` threshold for which `TN=0` and `FN=0`.
`AUC`	The area under curve (`AUC`) for the computed values of `X` and `Y`. If you set `xVals` to `'all'` (the default), `perfcurve` computes `AUC` using the returned `X` and `Y` values. If `xVals` is a numeric array, `perfcurve` computes `AUC` using `X` and `Y` values found from all distinct scores in the interval specified by the smallest and largest elements of `xVals`. More precisely, `perfcurve` finds `X` values for all distinct thresholds as if `xVals` were set to `'all'`, then uses a subset of these (with corresponding `Y` values) between `min(xVals)` and `max(xVals)` to compute `AUC`. The function uses trapezoidal approximation to estimate the area. If the first or last value of `X` or `Y` are `NaN`s, `perfcurve` removes them to allow calculation of `AUC`. This takes care of criteria that produce `NaN`s for the special `'reject all'` or `'accept all'` thresholds, for example, positive predictive value (PPV) or negative predictive value (NPV).
`OPTROCPT`	The optimal operating point of the ROC curve as an array of size 1-by-2 with FPR and TPR values for the optimal ROC operating point. `perfcurve` computes `optrocpt` only for the standard ROC curve and sets to `NaN`s otherwise. To obtain the optimal operating point for the ROC curve, `perfcurve` first finds the slope, `S`, using where `cost(I\|J)` is the cost of assigning an instance of class `J` to class `I`, and `P=TP+FN` and `N=TN+FP` are the total instance counts in the positive and negative class, respectively. `perfcurve` then finds the optimal operating point by moving the straight line with slope `M` from the upper left corner of the ROC plot (`FPR=0`, `TPR=1`) down and to the right until it intersects the ROC curve.
`SUBY`	An array of `Y` values for negative subclasses. If you only specify one negative class, `SUBY` is identical to `Y`. Otherwise `SUBY` is a matrix of size `M`-by-`K`, where `M` is the number of returned values for `X` and `Y`, and `K` is the number of negative classes. `perfcurve` computes `Y` values by summing counts over all negative classes. `SUBY` gives values of the `Y` criterion for each negative class separately. For each negative class, `perfcurve` places a new column in `SUBY` and fills it with `Y` values for TN and FP counted just for this class.
`SUBYNAMES`	A cell array of negative class names. If you provide an input array, `negClass`, of negative class names, `perfcurve` copies it into `SUBYNAMES`. If you do not provide `negClass`, `perfcurve` extracts `SUBYNAMES` from input labels. The order of `SUBYNAMES` is the same as the order of columns in `SUBY`, that is, `SUBY(:,1)` is for negative class `SUBYNAMES{1}` etc.

Examples

Plot the ROC curve for classification by logistic regression:

load fisheriris
x = meas(51:end,1:2);        
% iris data, 2 classes and 2 features
y = (1:100)'>50;             
% versicolor=0, virginica=1
b = glmfit(x,y,'binomial');  
% logistic regression
p = glmval(b,x,'logit');     
% fit probabilities for scores
[X,Y] = perfcurve(species(51:end,:),p,'virginica');
plot(X,Y)
xlabel('False positive rate'); ylabel('True positive rate')
title('ROC for classification by logistic regression')

References

[1] T. Fawcett, ROC Graphs: Notes and Practical Considerations for Researchers, 2004.

[2] M. Zweig and G. Campbell, Receiver-Operating Characteristic (ROC) Plots: A Fundamental Evaluation Tool in Clinical Medicine, Clin. Chem. 39/4, 561-577, 1993.

[3] J. Davis and M. Goadrich, The relationship between precision-recall and ROC curves, in Proceedings of ICML '06, 233-240, 2006.

[4] C. Moskowitz and M. Pepe, Quantifying and comparing the predictive accuracy of continuous prognostic factors for binary outcomes, Biostatistics 5, 113-127, 2004.

[5] Y. Huang, M. Pepe and Z. Feng, Evaluating the Predictiveness of a Continuous Marker, U. Washington Biostatistics Paper Series, 282, 2006.

[6] W. Briggs and R. Zaretzki, The Skill Plot: A Graphical Technique for Evaluating Continuous Diagnostic Tests, Biometrics 63, 250-261, 2008.

[7] http://www2.cs.uregina.ca/~hamilton/courses/831/notes/lift_chart/lift_chart.html; http://www.dmreview.com/news/5329-1.html.

[8] R. Bettinger, Cost-Sensitive Classifier Selection Using the ROC Convex Hull Method, SAS Institute.

[9] http://www.stata.com/statalist/archive/2003-02/msg00060.html