Net Reclassification Improvement

NRI has been proposed as an alternative to the area under the curve of the the ROC.

Background:
Nowadays, various measures have been proposed to capture discrimination in diagnostic tests, but the area under the receiver operating characteristic (ROC) curve (AUC) is still the most popular metric.
Researchers, usually, evaluate new bio-markers on their ability to increase the AUC. However, it quickly became apparent that, for models containing standard risk factors and possessing reasonably good discrimination, very large ‘independent’ associations of the new marker with the outcome are required to result in a meaningfully larger AUC.
Pencina et. al, proposed a new way of assessing improvement named "Net Reclassification Improvement". The NRI focuses on reclassification tables constructed separately for participants with and without events, and quantifies the correct movement in categories — upwards for events and downwards for non-events.

Description:
Consider a situation in which predicted probabilities of a given event of interest are estimated using two models that share all risk factors, except for one new marker. Let us categorize the predicted probabilities based on these two models into a set of clinically meaningful ordinal categories of absolute risk and then cross-tabulate these two classifications. Define upward movement (up) as a change into higher category based on the new model and downward movement (down) as a change in the opposite direction. D denotes the event indicator.

Results:
This code uses a Matlab GUI to calculate Net gains, NRI, z statistic and p value. Type NRI from command line to start the GUI.

Net gain will be calculated separately for subjects who experienced and did not experienced the event:
(D=1, subjects who experienced the event):
P(up|D=1) - P(down|D=1)
(D=0, subjects who did not experienced the event):
P(up|D=0) - P(down|D=0)

So, NRI will be:

NRI = improvement in reclassification = [ P(up|D=1) - P(down|D=1) ] - [ P(up|D=0) - P(down|D=0) ]

with
z value (z statistic for NRI) and the corresponding p-Value (with alpha error set to 0.05)

Example:
Pencina et. al, proposed an example of NRI for the Framingham Heart Study. Authors reported the initial AUC (0.762) of the Framingham model, and the AUC of the same model with HDL (0.774). As you can see, the increase in AUC is weak and also the difference in AUC is not statistically significant (p-value=0.092).
Evaluating separately subjects who experienced and did not experienced the CHD event they found that:
In the group who experienced the event (D=1, n = 183):
1) for 29 subjects classification improved using the model with HDL (moving up)
2) and for 7 people it became worse (moving down)
In the group who did not experienced the event (D=0, n = 3081)
1) 174 were reclassified down using the model with HDL
2) 173 were reclassified up.

The authors reported the following results:
NRI = 0.121, p < 0.001

and they conclude that addition of HDL improved classification for a net of 12 per cent of individuals with events, with no net loss for non-events.

This is exactly the result of Matlab.

Reference:
Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond.
Pencina J, D'Agostino R, R. D'Agostino Jr, Vasan R.
Statistics in Medicine, 2008 (27:157-172)