File Exchange

## Confidence intervals for area under the receiver operating curve (AUC)

version 1.1.0.0 (199 KB) by Brian Lau

### Brian Lau (view profile)

Various methods for estimating parametric and non-parametric confidence intervals for the AUC.

Updated 17 Aug 2015

Functions for estimating receiver operating curves (ROC) and the area under the ROC curve (AUC), and various methods for estimating parametric and non-parametric confidence intervals for the AUC estimates. Also included is code for a simple bootstrap test for the estimated area under the ROC against a known value. The available CI estimation methods are:
> Hanley-McNeil, parametric 
> Mann-Whitney, non-parametric 
> Maximum variance, non-parametric 
> Logit, non-parametric 
> Bootstrap, non-parametric 
You can pass arguments through for different bootstrapping options, otherwise the default is a simple percentile bootstrap. This software depends on several functions from the Matlab Statistics toolbox, namely norminv, tiedrank and bootci.

 Hanley, JA, McNeil, BJ (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143:29-36
 Qin, G, Hotilovac, L (2008). Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Stat Meth Med Res, 17:207-21
 Cortex, C, Mohri, M (2004). Confidence intervals for the area under the ROC curve. NIPS Conference Proceedings

Example:
s = randn(50,1) + 1; n = randn(50,1); % simulate binormal model, "signal" and "noise"
% Estimate the AUC and calculate bootstrapped 95% confidence intervals (bias-corrected and accelerated)
[A,Aci] = auc(format_by_class(s,n),0.05,'boot',1000,'type','bca');

### Cite As

Brian Lau (2020). Confidence intervals for area under the receiver operating curve (AUC) (https://www.github.com/brian-lau/MatlabAUC), GitHub. Retrieved .

leo nidas

### leo nidas (view profile)

The function yields wrong results as it cannot handle ties. Based on the wrong point estimate, in case of ties, all inference is wrong as well (including the bootstrap if that is based on re-estimating the AUC the same way). The results of this function disagree with the AUC as yielded from perfcurve.

It would be good to be a fix since this function is potentially useful.

It seems to be working for non-tied data.

Thank you for your submission. Below I provide two examples that reproduce the problem. I ran them in R2014a. Please correct me and point out if I am miss-using it and if I am wrong. Thank you again.

close all
clear all
clc

n=20;

%Example 1 that the function yields a wrong AUC:
x1=[0.5; 0.5; 0.5; 0.5; normrnd(0,1,n,1)];
x2=[0.5; 0.5; 0.5; 0.5; normrnd(1,1,n,1)];
XX=[x1;x2];
DD=[x1.*0;x2./x2];
[X Y T AUC]=perfcurve(DD,XX, 1);
AUC
[A,Aci] = auc([DD,XX])

%Example 2 that obvisously should yield AUC=0.5 but it does not:
x1=[0.5.*ones(20,1)];
x2=[0.5.*ones(20,1)];
XX=[x1;x2];
DD=[x1.*0;x2./x2];
[X Y T AUC]=perfcurve(DD,XX, 1);
AUC
[A,Aci] = auc([DD,XX])

Ian

### Ian (view profile)

Thanks brian, have been using your code before, but nice to see it githubbed here!

Ian

### Ian (view profile)

Thanks brian, have been using your code before, but nice to see it githubbed here!

Ian