Rank correlation and distribution tests · RiskAtoms Academy

By the end you will be able to

1Rank a sample, with ties given their average rank.
2Compute Spearman's rank correlation as Pearson on ranks, or with 1 − 6Σd²/(n(n²−1)) when there are no ties, and say why it ignores outliers' size and monotone transforms.
3Build two empirical distribution functions and find the two-sample KS statistic as their largest vertical gap.
4Place a trading desk in the Basel P&L attribution test's green, amber or red zone from its Spearman and KS figures.

Why you need it

A bank that wants to use its own model for a trading desk's capital must show that the model's P&L looks like the desk's real P&L. Not with an ordinary correlation, but with two tests that look at order and at shape: Spearman's rank correlation and the Kolmogorov–Smirnov statistic.

This Core Path lesson uses this atom:

L4.3.07: Backtesting and the P&L attribution test
The P&L attribution test: Spearman correlation and the KS statistic of hypothetical against risk-theoretical P&L decide whether a desk keeps its model (MAR32.36–32.44).

You need Pearson correlation from M-C10: Covariance and correlation, empirical distribution functions from M-D02: Histograms and empirical distributions, and the idea of a test statistic from M-D04: Confidence intervals and hypothesis tests. Level 6 returns to both tools: rank correlations to measure dependence, and distribution tests in model validation (M6.7: Model risk, validation and machine learning).

The idea

Keep the order, drop the distances

Pearson correlation (M-C10: Covariance and correlation) measures how close points lie to a straight line, so one wild day can drag it anywhere. Replace each value by its rank, its place in the sorted list, and you keep only the order. Spearman's correlation is Pearson on those ranks: it asks whether the two series agree on which days were good and which were bad.

Pearson against Spearman, on twelve days

Move day 12, or bend the series

Day 12's RTPL, £k130

Bend the RTPL series

Pearson0.987

Spearman0.986

Day 12's ranks, HPL / RTPL12 / 12

Push day 12 up to 1,000: it stays the highest RTPL, so its rank stays 12 and Spearman does not move, while Pearson falls. Bending the series with a cube or an exponential keeps every rank, so Spearman is unchanged again. Drag day 12 below zero and both fall, because the day really is out of order now. But once it is the lowest RTPL its rank cannot fall further, so Spearman stops falling, while Pearson keeps reacting to the size of the miss.

With the twelve days as they are, Pearson is 0.987 and Spearman 0.986. Push day 12's RTPL to 900, still in the right place but far too big, and Pearson drops to 0.708 while Spearman stays at 0.986. Robustness is the point of ranks; the price is that Spearman cannot see a model that gets every size wrong in the same direction. For that you compare the two distributions, which is what the KS statistic does.

The rules

Ranks and ties.

Spearman's rank correlation. Pearson on the ranks, and a shortcut when nothing ties.

The KS statistic. Two empirical distribution functions, and the largest gap between them.

The P&L attribution zones. Basel puts the two numbers together.

The rules, for reference

\begin{gathered} \rho_S = \rho\big(R^{x}, R^{y}\big) \;=\; 1 - \frac{6\sum_i d_i^2}{n(n^2-1)} \ \text{(no ties)}, \qquad d_i = R^x_i - R^y_i \\ \hat F(x) = \frac{1}{n}\#\{i : x_i \le x\}, \qquad \text{KS} = \max_x \big|\hat F_{\text{H}}(x) - \hat F_{\text{R}}(x)\big| \\ \text{green: } \rho_S > 0.8 \text{ and KS} < 0.09; \quad \text{red: } \rho_S < 0.7 \text{ or KS} > 0.12; \quad \text{amber: otherwise} \end{gathered}

What the Basel text says. The P&L attribution test compares a desk's risk-theoretical P&L with its hypothetical P&L over 250 days. Each series is ranked, with “the lowest value in the HPL time series” given rank 1, and Spearman's correlation is computed from the two rank series (MAR32.36–32.38). Each empirical distribution function is 0.004, that is 1/250, times the number of observations at or below a value, and the KS metric is “the largest absolute difference observed between these two empirical cumulative distribution functions” (MAR32.39–32.41). Table 2 of MAR32.42 sets the zones above, and gives the KS thresholds' p-values: 0.264 at 0.09 and 0.055 at 0.12. A red desk must leave the internal models approach for the standardised approach (MAR32.43); an amber desk keeps its model but adds a capital surcharge (MAR32.44). MAR32 does not say how tied values are ranked; the bank's code averages them.

Try it

Pick one of RiskAtoms Bank's four trading desks: 250 days of HPL and RTPL from its illustrative PLA test. The rank scatter shows Spearman at work, the two staircases show KS, and the zone map shows the verdict. Then change the model: cut or grow its error, or scale its P&L, and see which metric reacts.

Try itThe P&L attribution test on the bank's desks

Desk

Risk model: One-factor curve: 2y = 0.6429 × 10y move.

Model error: RTPL = HPL + e × (desk's RTPL − HPL)e = 1

Scale the RTPL× 1

250 days

Spearman0.758

KS0.096

Pearson, for comparison0.8163

amber

Amber: not green (Spearman 0.758 is not above 0.8), but not red either. The desk keeps its model with a capital surcharge.

Ranks: each dot is a day. On the dashed line the two series put that day in the same place.

HPL RTPL KS gap

Largest gap at £194k: 62.0% of HPL days are at or below it, against 71.6% of RTPL days.

Hollow circles are the four desks as they are. Try: on Rates, cut the model error to e = 0.5 and watch it turn green. On Credit, scale the RTPL by 0.8: Spearman does not move, but KS rises past 0.09 and the desk turns amber, because a scaled series keeps its order but not its spread.

Worked examples

Examples 1 and 2 use six illustrative days in £ thousands. Example 3 is RiskAtoms Bank plc's illustrative PLA test from Module 4.3 (the UK internal models approach starts on 1 January 2028, and the bank has no model permission).

1. Spearman on six days

Step 1 of 3

1
The data and the ranks
Day HPL RTPL Rank HPL Rank RTPL d d²
1 120 100 5 5 0 0
2 −340 −290 1 1 0 0
3 55 70 4 4 0 0
4 −80 −20 2 3 −1 1
5 410 260 6 6 0 0
6 15 −30 3 2 1 1

Day	HPL	RTPL	Rank HPL	Rank RTPL	d	d²
1	120	100	5	5	0	0
2	−340	−290	1	1	0	0
3	55	70	4	4	0	0
4	−80	−20	2	3	−1	1
5	410	260	6	6	0	0
6	15	−30	3	2	1	1

2. KS on the same six days

Step 1 of 2

1
Both ECDFs at every observed value
x F̂ HPL F̂ RTPL Gap
−340 1/6 0/6 0.167
−290 1/6 1/6 0
−80 2/6 1/6 0.167
−30 2/6 2/6 0
−20 2/6 3/6 0.167
15 3/6 3/6 0
55 4/6 3/6 0.167
70 4/6 4/6 0
100 4/6 5/6 0.167
120 5/6 5/6 0
260 5/6 6/6 0.167
410 6/6 6/6 0

x	F̂ HPL	F̂ RTPL	Gap
−340	1/6	0/6	0.167
−290	1/6	1/6	0
−80	2/6	1/6	0.167
−30	2/6	2/6	0
−20	2/6	3/6	0.167
15	3/6	3/6	0
55	4/6	3/6	0.167
70	4/6	4/6	0
100	4/6	5/6	0.167
120	5/6	5/6	0
260	5/6	6/6	0.167
410	6/6	6/6	0

3. The bank's four desks

Step 1 of 4

1
The results
Desk Spearman KS Pearson Zone
Equities 1 0.008 0.9996 green
Rates 0.758 0.096 0.8163 amber
FX and commodities 0.3367 0.236 0.3637 red
Credit 1 0.008 1 green

Desk	Spearman	KS	Pearson	Zone
Equities	1	0.008	0.9996	green
Rates	0.758	0.096	0.8163	amber
FX and commodities	0.3367	0.236	0.3637	red
Credit	1	0.008	1	green

Common mistakes

Each line below is a tempting slip. Spot what is wrong, then check.

Reaction

Check what stuck

Question 1 · objective 1

Rank the values 4, −2, 7, 4, 1 (lowest = 1, ties averaged). What is the rank of the first value, 4?

Question 2 · objective 2

Over five days the HPL ranks are 1, 2, 3, 4, 5 and the RTPL ranks are 2, 1, 3, 5, 4. What is Spearman's ρ?

Question 3 · objective 3

Sample A is 1, 2, 3, 4 and sample B is 3, 4, 5, 6. What is the two-sample KS statistic?

Question 4 · objective 4

A desk's Spearman correlation is 0.85 and its KS statistic 0.10. Which PLA zone is it in?

AAmberBGreenCRedDIt depends on the backtesting result

Question 5 · objective 2

A desk's risk model gives exactly half the hypothetical P&L every day. What do the two PLA metrics say?

ASpearman is 1, but KS can be large: the order is right, the size is wrong.BBoth are perfect: the two series move together exactly.CSpearman is 0.5 and KS is 0.5.DSpearman falls, but KS is 0 because the shape is the same.

Glossary

Rank: A value's position when the sample is sorted from lowest to highest: the lowest has rank 1, the highest rank n.
Tie: Two or more equal values. They share the average of the positions they occupy, so 7 and 7 in places 3 and 4 both get 3.5.
Spearman's rank correlation: Spearman's rank correlation ρ_S: the Pearson correlation of the two series' ranks. It measures how well one series puts the days in the same order as the other.
Pearson correlation: Pearson correlation ρ: covariance divided by both standard deviations. It measures how close the points lie to a straight line.
Increasing (monotone) transformation: An increasing (monotone) transformation: a function g with g(a) < g(b) whenever a < b, such as x³ or eˣ.
Empirical distribution function (ECDF): Empirical distribution function F̂(x): the share of the sample at or below x. A staircase rising by 1/n at each value.
Kolmogorov–Smirnov (KS) statistic: Two-sample Kolmogorov–Smirnov statistic: the largest vertical gap between two empirical distribution functions.
Hypothetical P&L (HPL): Hypothetical P&L (HPL): the day's P&L from revaluing yesterday's end-of-day positions at today's market prices, with no new trades or fees.
Risk-theoretical P&L (RTPL): Risk-theoretical P&L (RTPL): the same day's P&L as the desk's risk model sees it, using only the risk factors in the model.
P&L attribution (PLA) test: P&L attribution test: compares HPL and RTPL over 250 days with Spearman's correlation and the KS statistic, and puts each desk in a green, amber or red zone.