A scorecard or a PD model is first judged on one question: does it put the borrowers who default below the ones who repay? Validators, supervisors and model owners answer it with the same handful of numbers: AUC, Gini, accuracy ratio and KS. This atom shows what each one measures, and what none of them can tell you.
450 borrowers repaid (goods) and 50 defaulted (bads). Each has a score, higher meaning safer. The bads' scores are lower on average, but the two groups overlap. A lender picks a cut-offCut-off: the score below which applicants are declined (predicted to default). and declines everyone below it. Raise it, and more bads are caught, but more goods are turned away too:
Cut-off
Bads declined
Share of bads
Goods declined
Share of goods
500
13
26%
6
1.3%
540
31
62%
52
11.6%
580
42
84%
145
32.2%
620
48
96%
303
67.3%
No single row describes the score. A good score catches many bads before it loses many goods, at every cut-off. Plot the two shares against each other for every cut-off and you have the ROC curve; its area summarises the score in one number.
The rules
At one cut-off. The confusion matrixConfusion matrix: the 2 × 2 table of decisions against outcomes at one cut-off: true positives, false positives, false negatives and true negatives., the true positive rateTrue positive rate (TPR): the share of all bads that the cut-off declines. Also called the hit rate, sensitivity or recall., the false positive rateFalse positive rate (FPR): the share of all goods that the cut-off declines, the good business turned away. and precisionPrecision: of the borrowers declined, the share that were really bad..
Over all cut-offs. The ROC curveROC curve (receiver operating characteristic): the false positive rate against the true positive rate, traced as the cut-off moves., its AUCAUC: the area under the ROC curve. It equals the probability that a randomly chosen bad scores below a randomly chosen good, counting ties as half. and the Gini coefficientGini coefficient: 2 × AUC − 1. 0 for a random score, 1 for a perfect one..
The credit versions. The CAP curveCAP curve (cumulative accuracy profile): the share of all borrowers declined, worst first, against the share of bads caught. and its accuracy ratioAccuracy ratio (AR): the area between the model's CAP and the diagonal, divided by the same area for a perfect model. It equals the Gini coefficient., and the KS statisticKolmogorov-Smirnov (KS) statistic: the largest gap, over all cut-offs, between the share of bads and the share of goods scoring below the cut-off..
SB and SG are the scores of a bad and a good, higher meaning safer; FB and FG are their distribution functions (below c, strictly); π is the share of bads; nB and nG count them; 1{·} is 1 when the condition holds and 0 otherwise; aR and aP are the areas between the diagonal and the CAP of the model and of a perfect model; A is the AUC. Some texts put the score the other way up, or define KS with an absolute value; the numbers are the same. Sources: Fawcett (2006) for ROC and AUC; Hanley and McNeil (1982) for AUC as a probability and its standard error; Engelmann, Hayden and Tasche (2003) for CAP, AR and AR = 2 AUC − 1; Thomas, Crook and Edelman (2017) for KS in credit scoring.
Try it
Three illustrative scores for the same 500 borrowers, a 10% bad rate. The scores are synthetic: goods drawn with a fixed seed from a normal with mean 600 and standard deviation 50, bads from one 30, 70 or 120 points lower, rounded to whole points. Move the cut-off and watch the point travel along the curve while AUC, Gini and KS stay put: they describe the score, not the cut-off. Then switch scores.
Try itMove the cut-off
How well the score separates
Curve
goods (450) bads (50)Score across, in bins of 20 points; each bar is a share of its own group. Everything left of the blue line is declined.Share of goods declined across, share of bads declined up. Purple: the KS gap.
Confusion matrix at 560
bad
good
declined
38 TP
86 FP
approved
12 FN
364 TN
True positive rate (recall)76.0%
False positive rate19.1%
Precision30.6%
Accuracy80.4%
TPR − FPR at this cut-off0.569
KS, the largest gap (at 552)0.596
AUC0.860
Standard error of AUC0.034
Gini = 2 AUC − 10.719
Accuracy ratio from the CAP0.719
Share of borrowers declined24.8%
Score
AUC (SE)
Gini
KS
Implied AUC
Weak
0.666 (0.044)
0.332
0.291
0.664
Typical
0.860 (0.034)
0.719
0.596
0.839
Strong
0.958 (0.020)
0.915
0.800
0.955
The implied AUC is Φ(gap/(σ√2)), the AUC an endless sample from the same normals would give. With only 50 bads each sample AUC is within about two standard errors of it: the typical score's 0.860 has a standard error of 0.034. A Gini quoted to three decimals from a small default sample is far less precise than it looks.
Worked examples
1. One cut-off, by the confusion matrix
Step 1 of 3
1
Count
Cut-off 560
Bad
Good
Declined
TP = 38
FP = 86
Approved
FN = 12
TN = 364
2. AUC, Gini, AR and KS for ten loans, by hand
Step 1 of 5
1
The scores
Four bads scored 520, 555, 580, 610; six goods scored 540, 580, 600, 625, 640, 660. That makes 4 × 6 = 24 bad-good pairs.
3. Ranking well is not predicting well
Step 1 of 3
1
Two sets of PDs
Group the typical score's 500 borrowers into four score bands. Model A gives each band its observed default rate; model B gives half of it.
Common mistakes
Each line below is a tempting slip. Spot what is wrong, then check.
Reaction
Check what stuck
Question 1 · objective 1
A cut-off declines 40 of the 50 borrowers who went on to default. What is its true positive rate? Give a decimal.
Question 2 · objective 1
The same cut-off declines 100 borrowers in all, 40 of them bad. What is its precision? Give a decimal.
Question 3 · objective 2
A scorecard has an AUC of 0.78. What is its Gini coefficient?
Question 4 · objective 3
At the cut-off where the gap is largest, 62% of bads and 21% of goods score below it. What is the KS statistic? Give a decimal.
Question 5 · objective 4
A model's predicted PDs are all doubled. Its PDs used to match observed default rates. What happens to its AUC and its calibration?
Glossary
Cut-off
Cut-off: the score below which applicants are declined (predicted to default).
Confusion matrix
Confusion matrix: the 2 × 2 table of decisions against outcomes at one cut-off: true positives, false positives, false negatives and true negatives.
True positive rate (TPR)
True positive rate (TPR): the share of all bads that the cut-off declines. Also called the hit rate, sensitivity or recall.
False positive rate (FPR)
False positive rate (FPR): the share of all goods that the cut-off declines, the good business turned away.
Precision
Precision: of the borrowers declined, the share that were really bad.
ROC curve
ROC curve (receiver operating characteristic): the false positive rate against the true positive rate, traced as the cut-off moves.
AUC
AUC: the area under the ROC curve. It equals the probability that a randomly chosen bad scores below a randomly chosen good, counting ties as half.
Gini coefficient
Gini coefficient: 2 × AUC − 1. 0 for a random score, 1 for a perfect one.
CAP curve
CAP curve (cumulative accuracy profile): the share of all borrowers declined, worst first, against the share of bads caught.
Accuracy ratio (AR)
Accuracy ratio (AR): the area between the model's CAP and the diagonal, divided by the same area for a perfect model. It equals the Gini coefficient.
Kolmogorov-Smirnov (KS) statistic
Kolmogorov-Smirnov (KS) statistic: the largest gap, over all cut-offs, between the share of bads and the share of goods scoring below the cut-off.
Discrimination
Discrimination: how well a score ranks bads below goods. Measured by AUC, Gini, AR and KS.
Calibration
Calibration: whether the predicted default probabilities match the default rates actually observed.
Basel Committee, Studies on the validation of internal rating systems (Working Paper 14): the measures validators use for rating systems, with their strengths and limits. Advanced.
Origins
James A. Hanley and Barbara J. McNeil, “The meaning and use of the area under a receiver operating characteristic (ROC) curve”, Radiology 143(1), 29–36, 1982.
Tom Fawcett, “An introduction to ROC analysis”, Pattern Recognition Letters 27(8), 861–874, 2006.
Bernd Engelmann, Evelyn Hayden and Dirk Tasche, “Measuring the discriminative power of rating systems”, Deutsche Bundesbank Discussion Paper Series 2, No 01/2003, 2003.
Lyn Thomas, Jonathan Crook and David Edelman, Credit Scoring and Its Applications, 2nd edition, SIAM, 2017 (chapter 8, measuring scorecard performance).