Backtesting · RiskAtoms Academy

By the end you will be able to

1Backtest a VaR model: count exceptions over 250 days against hypothetical and actual P&L.
2Place the count in the Basel traffic light using the binomial probabilities, and give the multiplier it implies.
3Run Kupiec's proportion-of-failures test and read its result, including what it cannot detect.
4Explain the difference between hypothetical and actual P&L and why both are tested.

Why it matters

Over the last 250 trading days RiskAtoms Bank's Markets desk lost more than its £1.94m one-day 99% VaR on 5 days. A correct model would expect 2.5. Is the model broken, or was the desk unlucky? Backtesting turns that question into a probability, and the answer sets how much capital the bank holds and whether anyone trusts its risk numbers.

Core explanation

Holding a VaR model to account

Illustrative dataRiskAtoms Bank plc's Markets desk

The module's trading book: eight positions in four books (Equities, Rates, FX and commodities, Credit) driven by eight risk factors, with 750 days of seeded factor moves that include a stressed spell (days 480 to 539) and four jump days. Every position is revalued in full. The numbers are made up for teaching, in pounds, on one valuation day; they are not market data.

The VaR model is the delta-normal model of L3.2.09: Parametric VaR for a portfolio. The desk's positions are the same every day, so its VaR is £1.94m every day and the hypothetical P&L is the book's 750-day history. Actual P&L adds an illustrative stream of intraday and new-deal income (average £30k a day, standard deviation £120k).

What is compared with what

L2.5.06: Value at risk: the idea introduced the idea: a backtest sets each evening's VaR beside the next day's P&L. A day whose loss beats the VaR is an exception. Two choices make the test fair. First, the forecast must be the one made before the day, never revised with hindsight. Second, the P&L must match what the forecast was about. VaR describes yesterday's positions, so the cleanest comparison is hypothetical P&L. Actual P&L adds the day's trading and new deals: it is the number the business cares about, but a desk that earns a steady bid-ask spread can hide model errors in it. Supervisors want both.

How many exceptions is too many?

If the model is right, each day has a 1% chance of an exception, independently of the others. The count over 250 days is then binomial with n = 250 and p = 1% (M-C04: The binomial distribution): on average 2.5, but anything from 0 to 5 is common. The Basel Committee's traffic light uses exactly these probabilities. The amber zone (yellow in 1996) starts where the chance of seeing that many exceptions or fewer first reaches 95%; the red zone where it reaches 99.99%.

Exceptions	P(exactly)	P(this many or fewer)	Zone	1996 plus factor
0	8.11%	8.11%	green	0.00
1	20.47%	28.58%	green	0.00
2	25.74%	54.32%	green	0.00
3	21.49%	75.81%	green	0.00
4	13.41%	89.22%	green	0.00
5	6.66%	95.88%	yellow	0.40
6	2.75%	98.63%	yellow	0.50
7	0.97%	99.60%	yellow	0.65
8	0.30%	99.89%	yellow	0.75
9	0.08%	99.97%	yellow	0.85
10 or more	0.03%	100.00%	red	1.00

Under the 1996 framework the bank's market risk capital was its VaR times a multiplier of at least 3. The plus factor from the table was added to the 3, so a desk in the amber zone held up to 28% more capital and one in the red zone a third more. The amber zone is a grey area: a correct model lands there about one year in ten. Supervisors then ask why, and the bank's explanation matters.

Sort itWhy did the desk breach its VaR?

The 1996 framework sorted yellow-zone explanations into four kinds. Sort each one.

The corporate bond trade booked on day 512 never reached the risk system, so it was missing from the VaR.
The VaR maps all credit spreads onto one IG index, and the desk's bonds widened much more than the index.
The desk bought a large block of index futures at 10am and the market fell 2% by the close.
A central bank surprised the market with a 50bp cut, a move far outside recent experience.

A first statistical test

The traffic light only looks for too many exceptions. Paul Kupiec's proportion-of-failures test (1995) asks a sharper question: is the observed exception rate consistent with 1%? It is a likelihood-ratio test (M-D04: Confidence intervals and hypothesis tests), so it rejects too few exceptions as well as too many. With 250 days it accepts 1 to 6 exceptions at the 5% level. Zero exceptions gives a ratio of 5.03, above the 3.84 threshold: a model that is never breached is probably too cautious, which wastes capital and makes limits meaningless.

Every test with 250 observations has the same weakness: little power. A small Type I error (rejecting a good model) forces a large Type II error (keeping a bad one). Slide the true exception rate and watch:

How often does the traffic light catch a bad model?

True chance of an exception each day2.00% (the model claims 1%)

Green

43.9%

Amber

53.1%

Red

3.0%

A model that misses 2.00% of the time, not 1%, still lands in green 44% of the time: the test has little power over 250 days.

A model that really fails 2% of the time, twice as often as it claims, still shows green 44% of the time. Counting also ignores two things that matter. Clustered exceptions suggest the model reacts too slowly to changing volatility, and the size of each exception says how bad the tail is. Tests for both (Christoffersen's independence test, expected shortfall backtests) are in M6.2: Market risk modelling.

The formal version

The count. Let $I_t = 1$ if day $t$ is an exception and 0 otherwise. If the model is right, the $I_t$ are independent with $P(I_t = 1) = p$ , so the count $X = \sum_t I_t$ is binomial:

\begin{aligned} P(X = k) &= \binom{n}{k}\,p^k\,(1-p)^{n-k}, \qquad \mathbb{E}[X] = np = 2.5,\quad \operatorname{sd}(X) = \sqrt{np(1-p)} = 1.573 \\ P(X \le 4) &= \sum_{k=0}^{4}\binom{250}{k}\,0.01^k\,0.99^{250-k} = 89.22\%, \qquad P(X \ge 5) = 10.78\% \end{aligned}

Kupiec with the desk's numbers. $n = 250$ , $x = 5$ , $p_0 = 0.01$ , $\hat p = 5/250 = 0.020$ :

\begin{aligned} \ln L(p_0) &= 245\ln 0.99 + 5\ln 0.01 = -25.4882 \\ \ln L(\hat p) &= 245\ln 0.980 + 5\ln 0.020 = -24.5098 \\ \mathrm{LR} &= -2\,(\ln L(p_0) - \ln L(\hat p)) = 1.957 < 3.84, \qquad p\text{-value} = 16.2\% \end{aligned}

Kupiec's test does not reject the model, even though the traffic light shows amber. The two answer different questions: the traffic light is one-sided and set so that a correct model is rarely penalised; Kupiec's test is two-sided at 5%. Neither looks at when the exceptions happened.

Try it

The window starts on the most recent 250 days, the ones a supervisor would look at. Slide it back into the stressed spell and watch the delta-normal model turn red. Then switch to the 250-day historical model: its VaR jumps once the stressed days enter its own window, so it catches up, but only after the damage.

Try itBacktest the desk's VaR

VaR model

Window ends on day749 (days 500 to 749)

Bars: hypothetical P&L. Amber line: minus the VaR forecast for that day. Red bars: exceptions.

Exceptions in 250 days5

Expected2.5

P(this many or more) if the model is right10.78%

Basel zoneyellow

1996 multiplier3.40

Kupiec LR (reject above 3.84)1.96

p-value16.2%

Worked example

The desk's backtest, days 500 to 749

From P&L to a verdict

Step 1 of 6

1

List the exceptions

Compare each day's hypothetical P&L with the previous evening's VaR of £1.94m. Five days breach it:

Day	Hypothetical loss	Actual loss	VaR	Excess over VaR
505	£2.26m	£2.18m	£1.94m	£0.32m
513	£2.27m	£2.14m	£1.94m	£0.33m
515	£2.93m	£3.14m	£1.94m	£0.99m
520	£2.90m	£2.89m	£1.94m	£0.95m
524	£2.86m	£2.80m	£1.94m	£0.92m

Actual P&L gives the same 5 days here: the trading income is small next to a £2m VaR. The current Basel rule would count the larger of the two, still 5.

What the risk team writes

Delta-normal VaR: 5 exceptions in 250 days (amber), all in the stressed spell between days 505 and 524; Kupiec does not reject; clustering points to slow-reacting volatilities. Action: test a volatility-weighted or shorter-window model (M6.2: Market risk modelling), and report the exceptions to the model validation team.

Regulatory lens

The Basel Committee's January 1996 backtesting framework set the rules used above: 250 days, 99% one-day VaR, the three zones, and plus factors of 0.40 to 0.85 in the yellow (now amber) zone and 1.00 in the red, added to the multiplier of 3. In the current Basel framework (the Fundamental Review of the Trading Book) the zones are called green, amber and red and keep the same boundaries; the multiplier on the new expected shortfall-based capital runs from 1.50 to 2.00, banks must test against both hypothetical and actual P&L and count the larger number of exceptions, and each trading desk is backtested separately as well. That detail, with the P&L attribution test, is in M4.3: Market risk capital (FRTB) and CVA.

Spark Atoms

Spark AtomWhy not reject at the first exception?

The Basel Committee worked through the trade-off in its guidance: with a rule of rejecting any model that has even one exception in 250 days, an accurate 99% model would be rejected 91.9% of the time. Zones exist because no single cut-off keeps both kinds of error small.

Spark AtomA Fed economist's test

Paul Kupiec published the proportion-of-failures test in May 1995 as a Federal Reserve Board working paper, eight months before the Basel Committee issued its backtesting framework in January 1996. It is still a standard first statistical test for a VaR model.

Key takeaways

A backtest compares each evening's VaR with the next day's P&L; a loss larger than the VaR is an exception.
If a 99% model is right, exceptions in 250 days are Binomial(250, 1%): 2.5 on average, 0 to 4 for 89% of the time.
Basel traffic light: green 0–4, amber (yellow in 1996) 5–9, red 10+. In 1996 the multiplier of 3 rose by 0.40 to 1.00.
Kupiec's test, LR = −2 ln[L(p₀)/L(p̂)] against 3.84, rejects too few exceptions as well as too many.
Hypothetical P&L tests the model; actual P&L includes trading income. Clustered exceptions mean a slow model, even if the count passes.

Reaction

Check what stuck

Question 1 · objective 1

A bank backtests its 99% one-day VaR over 500 days. How many exceptions should it expect if the model is right?

Question 2 · objective 2

If a 99% VaR model is correct, what is the probability of exactly 2 exceptions in 250 days, in % (two decimals)?

%

Question 3 · objective 2

A desk has 7 exceptions in 250 days. Under the 1996 Basel framework, what is its multiplier?

A3.00 (green)B3.50 (amber)C3.65 (amber)D4.00 (red)

Question 4 · objective 3

Kupiec's test, 250 days, p = 1%, 3 exceptions. What is the likelihood ratio (two decimals)?

Question 5 · objective 4

A desk's VaR backtest uses actual P&L. The desk earns a steady bid-ask spread from clients every day. What does that do to the count?

ANothing: actual and hypothetical P&L always give the same countBIt tends to hide exceptions, because spread income cushions the market losses the VaR is meant to forecastCIt creates extra exceptionsDIt makes the test more accurate

Glossary

Backtest: Comparing a model's past forecasts with what actually happened, day by day, to see whether the model works.
Exception: A day on which the loss is larger than the VaR forecast made the evening before. Also called a breach.
Hypothetical P&L: The P&L the previous evening's positions would have made from the day's market moves alone: no new trades, no intraday trading, no fees.
Actual P&L: The P&L the desk really made that day, including intraday trading, new deals and (depending on the definition) fees.
Traffic light: The Basel Committee's backtesting zones for 250 days of 99% one-day VaR: green 0 to 4 exceptions, amber (called yellow in the 1996 framework) 5 to 9, red 10 or more.
Plus factor: In the 1996 framework, the amount added to the minimum multiplier of 3 on VaR capital, from 0 in the green zone to 1 in the red.
Proportion-of-failures test: Kupiec's proportion-of-failures test: a likelihood-ratio test of whether the observed exception rate matches the VaR's confidence level.
Type I error: Rejecting a model that is in fact correct.
Type II error: Accepting a model that is in fact wrong.
Exception clustering: Exceptions bunched together in time. A correct model's exceptions should be spread out at random.

Deep Dives & Origins

Deep Dives

P. Jorion, Value at Risk, 3rd ed., McGraw-Hill, 2007, the chapter on backtesting VaR. Intermediate.
P. Christoffersen, “Evaluating interval forecasts”, International Economic Review 39(4), 1998. Advanced.

Origins

Basel Committee on Banking Supervision, Supervisory framework for the use of “backtesting” in conjunction with the internal models approach to market risk capital requirements, January 1996.
OCC, Federal Reserve and FDIC, “Risk-Based Capital Standards: Market Risk”, final rule, Federal Register 61(174), 6 September 1996 (Table 1, multiplication factors).
Basel Committee on Banking Supervision, The Basel Framework, MAR99 Guidance on use of the internal models approach, version effective 1 January 2023.
P. Kupiec, “Techniques for verifying the accuracy of risk measurement models”, Journal of Derivatives 3(2), 1995, pp. 73–84 (first issued as Federal Reserve Board FEDS working paper 95-24, May 1995).

Watch this lesson

Video version

Prefer to watch? The same lesson as a 12-minute video will sit here, made from this page's script.