Core Path · L6.3.04

Measuring discrimination

14 minROC curveAUC and GiniKS statisticDeLong testOut-of-time validationPSI

By the end you will be able to

  1. 1Build a ROC curve and read the AUC, Gini, accuracy ratio and KS of a score.
  2. 2Put a standard error on an AUC and test whether one score ranks better than another, or worse than at development.
  3. 3Monitor a score's stability with the PSI and CSI, and say what stability does not show.
  4. 4Read discrimination results honestly, out of time and against the incumbent.

Why it matters

The scorecard of L6.3.03: Building a scorecard ranks its development loans with an AUC of 0.760. On the later loans it never saw, 0.739. The scorecard RiskAtoms Bank plc already uses (L3.3.14: Retail credit portfolios), on the same later loans: 0.739. Months of modelling, and out of time the new card ranks no better than the old one. This lesson is about measuring that properly, before anyone switches.