File Exchange

image thumbnail

Confidence intervals for area under the receiver operating curve (AUC)

version 1.1.0.0 (199 KB) by Brian Lau
Various methods for estimating parametric and non-parametric confidence intervals for the AUC.

5 Downloads

Updated 17 Aug 2015

GitHub view license on GitHub

Functions for estimating receiver operating curves (ROC) and the area under the ROC curve (AUC), and various methods for estimating parametric and non-parametric confidence intervals for the AUC estimates. Also included is code for a simple bootstrap test for the estimated area under the ROC against a known value. The available CI estimation methods are:
> Hanley-McNeil, parametric [1]
> Mann-Whitney, non-parametric [2]
> Maximum variance, non-parametric [3]
> Logit, non-parametric [2]
> Bootstrap, non-parametric [2]
You can pass arguments through for different bootstrapping options, otherwise the default is a simple percentile bootstrap. This software depends on several functions from the Matlab Statistics toolbox, namely norminv, tiedrank and bootci.

[1] Hanley, JA, McNeil, BJ (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143:29-36
[2] Qin, G, Hotilovac, L (2008). Comparison of non-parametric confidence intervals for the area under the ROC curve of a continuous-scale diagnostic test. Stat Meth Med Res, 17:207-21
[3] Cortex, C, Mohri, M (2004). Confidence intervals for the area under the ROC curve. NIPS Conference Proceedings

Example:
s = randn(50,1) + 1; n = randn(50,1); % simulate binormal model, "signal" and "noise"
% Estimate the AUC and calculate bootstrapped 95% confidence intervals (bias-corrected and accelerated)
[A,Aci] = auc(format_by_class(s,n),0.05,'boot',1000,'type','bca');

Visit https://github.com/brian-lau/MatlabAUC for more info.

Cite As

Brian Lau (2020). Confidence intervals for area under the receiver operating curve (AUC) (https://www.github.com/brian-lau/MatlabAUC), GitHub. Retrieved .

Comments and Ratings (4)

leo nidas

The function yields wrong results as it cannot handle ties. Based on the wrong point estimate, in case of ties, all inference is wrong as well (including the bootstrap if that is based on re-estimating the AUC the same way). The results of this function disagree with the AUC as yielded from perfcurve.

It would be good to be a fix since this function is potentially useful.

It seems to be working for non-tied data.

Thank you for your submission. Below I provide two examples that reproduce the problem. I ran them in R2014a. Please correct me and point out if I am miss-using it and if I am wrong. Thank you again.

close all
clear all
clc

n=20;

%Example 1 that the function yields a wrong AUC:
x1=[0.5; 0.5; 0.5; 0.5; normrnd(0,1,n,1)];
x2=[0.5; 0.5; 0.5; 0.5; normrnd(1,1,n,1)];
XX=[x1;x2];
DD=[x1.*0;x2./x2];
[X Y T AUC]=perfcurve(DD,XX, 1);
AUC
[A,Aci] = auc([DD,XX])

%Example 2 that obvisously should yield AUC=0.5 but it does not:
x1=[0.5.*ones(20,1)];
x2=[0.5.*ones(20,1)];
XX=[x1;x2];
DD=[x1.*0;x2./x2];
[X Y T AUC]=perfcurve(DD,XX, 1);
AUC
[A,Aci] = auc([DD,XX])

Ian

Thanks brian, have been using your code before, but nice to see it githubbed here!

Ian

Thanks brian, have been using your code before, but nice to see it githubbed here!

Ian

Updates

1.1.0.0

-

MATLAB Release Compatibility
Created with R2008a
Compatible with any release
Platform Compatibility
Windows macOS Linux
Tags Add Tags