A risk model is only as good as its parameters, and the parameters come from data. Maximum likelihood is the standard way to get them: pick the values under which the data you saw were most probable. It works for a normal, a fat-tailed Student t, a GARCH model or an extreme value distribution alike, and it comes with its own error bars.
Five waiting times, in days, between large operational losses: 16.1, 15.7, 8.6, 57.1, 4.4. Suppose the waits are exponential with rate λ (events a day). Each candidate λ gives a density curve, and the density at each observation measures how plausible that observation is. Multiply the five heights: that is the likelihoodLikelihood L(θ): the probability (or density) of the observed data, read as a function of the parameter θ with the data held fixed.. Drag λ to make it as large as you can.
Waiting time in days. Each bar is the density at one observation.
Heights0.02 × 0.0208 × 0.0423 × 0.0003 × 0.0644
Likelihood L(λ), their product3.75 × 10⁻¹⁰
Log-likelihood ℓ(λ), the sum of logs−21.703
Average waiting time20.38 days
A big λ makes short waits likely but the 57.1-day wait nearly impossible; a small λ spreads the density so thin that every observation is improbable. The best compromise is λ = 0.0491, one over the average wait. That value is the maximum likelihood estimateMaximum likelihood estimate (MLE): the parameter value θ̂ that makes the observed data most probable under the model..
λ
Likelihood L(λ)
Log-likelihood ℓ(λ)
0.02
4.17 × 10⁻¹⁰
−21.598
0.04
17.38 × 10⁻¹⁰
−20.170
0.06
17.20 × 10⁻¹⁰
−20.181
0.08
9.44 × 10⁻¹⁰
−20.781
0.12
1.22 × 10⁻¹⁰
−22.829
The rules
Likelihood and log-likelihood. The data are fixed; the parameter moves.
Maximising it. Set the scoreScore: the slope ℓ′(θ) of the log-likelihood. It is zero at a smooth maximum. to zero, by hand when you can.
How precise it is. The observed informationObserved information: −ℓ″(θ̂), the curvature of the log-likelihood at its peak. A sharp peak means a precise estimate. gives the standard errorStandard error of an MLE: 1/√(observed information) for one parameter; with several, the square roots of the diagonal of the inverse information matrix..
Comparing models. The likelihood ratio testLikelihood ratio test: LR = 2[ℓ(θ̂) − ℓ(θ₀)], compared with a χ² distribution whose degrees of freedom are the number of restrictions. for nested modelsNested models: one is a special case of the other, obtained by fixing some of its parameters, so the likelihood ratio test applies.; AICAIC (Akaike information criterion): 2k − 2ℓ(θ̂) for a model with k parameters. Lower is better. and BICBIC (Bayesian or Schwarz information criterion): k ln n − 2ℓ(θ̂). It penalises parameters more than AIC once n is 8 or more. for any.
The formal version
ℓ(θ)θ^LRExp(λ)=i=1∑nlnf(xi∣θ),θ^=argθmaxℓ(θ),∇ℓ(θ^)=0≈N(θ,I(θ^)−1),I(θ^)=−∇2ℓ(θ^),SE(θ^j)=[I(θ^)−1]jj=2[ℓ(θ^)−ℓ(θ^0)]dχr2 under H0,AIC=2k−2ℓ(θ^),BIC=klnn−2ℓ(θ^):λ^=xˉ1,SE=nλ^;N(μ,σ2):μ^=xˉ,σ^2=n1i=1∑n(xi−xˉ)2,SE(μ^)=nσ^,SE(σ^)=2nσ^
x₁, …, xn are independent observations with density f(x | θ); θ may be a vector of k parameters; ∇ and ∇² are the gradient and the matrix of second derivatives; I(θ̂) is the observed information; θ̂₀ maximises ℓ under the null hypothesis H₀, which imposes r restrictions; the arrow marked d means convergence in distribution as n grows, under the usual regularity conditions (in particular, the true θ is not on the edge of the parameter space). Sources: Casella and Berger (2002), sections 7.2.2, 8.2.1, 10.1 and 10.3.1; Akaike (1974); Schwarz (1978).
Try it
Two data sets, both simulated with a seed: waiting times with a true rate of 0.05 a day, and daily returns with a true volatility of 1%. Drag the parameter and watch the score go to zero at the peak. Then change n: the curve sharpens and the standard error shrinks like 1/√n. Values whose LR exceeds 3.84 fall below the red line and are rejected.
Try itClimb the log-likelihood
Model and data
Observations n
ℓ(θ) − ℓ(θ̂) (solid), the parabola with the same curvature at the peak (dashed purple), the cut-off 1.92 below the peak (dashed red) and the true value used to simulate the data (dotted).
ℓ at your value−164.094
Slope ℓ′ (the score)538.93
Peak: the MLE0.0504
Curvature −ℓ″ at the peak15,776.8
Standard error 1/√(−ℓ″)0.008
95% interval, MLE ± 1.96 SE0.0347 to 0.066
95% likelihood ratio interval0.0363 to 0.0676
LR = 2[ℓ(θ̂) − ℓ(your value)]9.09
p-value, χ² with 1 df0.0026
Your value at 5%rejected
Worked examples
1. Loss arrivals: the exponential by hand
Step 1 of 4
1
The data
Ten waiting times in days between large operational losses (illustrative): 16.1, 15.7, 8.6, 57.1, 4.4, 36.1, 23.2, 21.8, 10.3, 9.5. They add up to 202.8 days, an average of 20.28.
2. Daily returns: the normal by hand
Step 1 of 4
1
The data
500 illustrative daily returns in %, from −4.78 to 4.09. Their sum is 12.62 and the sum of squared deviations from the mean is 519.51.
3. The same returns: a Student t, fitted numerically
Step 1 of 5
1
No closed form
The Student t has a location μ, a scale s and degrees of freedom ν that set how fat the tails are. Setting three derivatives to zero has no algebraic solution, so a simplex search (Nelder-Mead) climbs ℓ from fixed starting values μ = 0, s = 1, ν = 10, and stops when the simplex values agree to 1 part in 10¹⁴. It took 168 steps.
Common mistakes
Each line below is a tempting slip. Spot what is wrong, then check.
Reaction
Check what stuck
Question 1 · objective 2
Eight waiting times between loss events add up to 200 days. Assuming an exponential model, what is the maximum likelihood estimate of the rate λ, per day?
Question 2 · objective 3
For the same data, what is the standard error of λ̂? Give four decimals.
Question 3 · objective 3
A log-likelihood has second derivative −400 at its peak. What is the standard error of the estimate?
Question 4 · objective 4
A model with 3 parameters reaches a maximised log-likelihood of −700. What is its AIC?
Question 5 · objective 4
A likelihood ratio test of one restriction gives LR = 2.1. What do you conclude at the 5% level?
Glossary
Likelihood
Likelihood L(θ): the probability (or density) of the observed data, read as a function of the parameter θ with the data held fixed.
Log-likelihood
Log-likelihood ℓ(θ) = ln L(θ): a sum of log densities, maximised at the same θ as the likelihood.
Maximum likelihood estimate (MLE)
Maximum likelihood estimate (MLE): the parameter value θ̂ that makes the observed data most probable under the model.
Score
Score: the slope ℓ′(θ) of the log-likelihood. It is zero at a smooth maximum.
Observed information
Observed information: −ℓ″(θ̂), the curvature of the log-likelihood at its peak. A sharp peak means a precise estimate.
Standard error of an MLE
Standard error of an MLE: 1/√(observed information) for one parameter; with several, the square roots of the diagonal of the inverse information matrix.
Likelihood ratio test
Likelihood ratio test: LR = 2[ℓ(θ̂) − ℓ(θ₀)], compared with a χ² distribution whose degrees of freedom are the number of restrictions.
Nested models
Nested models: one is a special case of the other, obtained by fixing some of its parameters, so the likelihood ratio test applies.
AIC
AIC (Akaike information criterion): 2k − 2ℓ(θ̂) for a model with k parameters. Lower is better.
BIC
BIC (Bayesian or Schwarz information criterion): k ln n − 2ℓ(θ̂). It penalises parameters more than AIC once n is 8 or more.
Casella and Berger, Statistical Inference, chapter 10: why the MLE is consistent and asymptotically normal, and the asymptotic distribution of the likelihood ratio. Advanced.
Origins
George Casella and Roger L. Berger, Statistical Inference, 2nd edition, Duxbury, 2002 (sections 7.2.2 maximum likelihood estimators, 8.2.1 likelihood ratio tests, 10.1 and 10.3.1 asymptotics).
Hirotugu Akaike, “A new look at the statistical model identification”, IEEE Transactions on Automatic Control 19(6), 716–723, 1974.
Gideon Schwarz, “Estimating the dimension of a model”, The Annals of Statistics 6(2), 461–464, 1978.