Estimation error, shrinkage and Black-Litterman · RiskAtoms Academy

By the end you will be able to

1Quantify how uncertain estimated expected returns are, and why volatilities are much easier to estimate.
2Explain why a mean-variance optimiser magnifies small errors in its inputs into large swings in weights.
3Shrink a sample covariance matrix towards a constant-correlation target, and say when that helps.
4Build Black-Litterman expected returns from market weights and views, and read the resulting portfolio.

Why it matters

Give a mean-variance optimiser the module's expected returns and it suggests a sensible mix for RiskAtoms Group. Raise the government-bond forecast by half a percentage point, a change no economist could defend or refute, and it now wants 156% in government bonds and −59% in corporate bonds. Optimisers take their inputs literally. This lesson is about what to do when the inputs are guesses.

Core explanation

Optimising with noisy inputs

L3.1.07: Mean-variance optimisation and L3.1.08: The risk-free asset and the capital market line treated the expected returns $\mu$ and the covariance matrix $\Sigma$ as known. In practice both are estimated from data or set by judgement, and both carry estimation error.

Expected returns are hard to estimate

The average of $T$ years of returns has a standard error of $\sigma/\sqrt{T}$ (M-D04: Confidence intervals and hypothesis tests). For global equities, with volatility 16%:

Years of data	Standard error of the mean	95% interval around a true 7%
10	5.06%	−2.9% to 16.9%
30	2.92%	1.3% to 12.7%
100	1.60%	3.9% to 10.1%

To pin the expected return down to ±1% (one standard error) would take 256 years of data, from markets that did not exist and would not have stayed the same. Volatility is far easier: ten years of monthly data (120 observations) estimate a 16% volatility with a standard error of about 1.0%. The mean depends on how long you watch; the variance on how often.

The optimiser magnifies the errors

Mean-variance weights are $w = \Sigma^{-1}(\mu - r_f\mathbf{1})/\delta$ , with $\delta$ the investor's risk aversion. With the module's inputs and $\delta = 3.31$ they are the tangency portfolio of L3.1.08: The risk-free asset and the capital market line. Now add 0.5% to one expected return at a time:

Raise by 0.5%	Global eq.	EM eq.	Gov. bonds	Corp. bonds	Property
None (true inputs)	29%	18%	42%	13%	−3%
Global equities	47%	12%	55%	4%	−7%
Emerging-market equities	23%	26%	45%	9%	−3%
Government bonds	43%	22%	156%	−59%	−3%
Corporate bonds	20%	14%	−31%	88%	−6%
Property	25%	18%	41%	10%	5%

Government and corporate bonds are highly correlated (0.7), so the optimiser sees a long-short pair that is almost riskless and leans on it hard whenever their forecasts drift apart. Richard Michaud called optimisers error maximisers: they put the most money where the estimate is most flattering, which is often where it is most wrong. With means estimated from ten years of data, the swings are far larger:

History	Global eq.	EM eq.	Gov. bonds	Corp. bonds	Property
1: estimated return	1.1%	−3.8%	5.7%	6.2%	0.9%
1: weight	61%	−78%	63%	205%	−57%
2: estimated return	8.9%	0.6%	0.9%	2.1%	8.6%
2: weight	154%	−123%	−202%	64%	42%
3: estimated return	2.8%	1.1%	1.6%	3.2%	0.6%
3: weight	−32%	−21%	−340%	262%	−34%

Illustrative dataSimulated histories

Each history is 120 months drawn at random from the module's universe, so the true expected returns are 7.0%, 8.5%, 3.5%, 4.5%, 6.0% every time. The weights come from the estimated means and covariances.

Try itHow the optimiser reacts to its inputs

Inputs

Global equities: 7.0%0.0%

Emerging-market equities: 8.5%0.0%

Government bonds: 3.5%0.0%

Corporate bonds: 4.5%0.0%

Property: 6.0%0.0%

Optimal weights

Global eq.29.4%

EM eq.18.3%

Gov. bonds41.8%

Corp. bonds13.1%

Property−2.6%

Weights from the true inputs

Share of the portfolio moved from the true-input weights: 0%. Bars are cut off at −200% and +250%; the numbers are not.

There are three broad cures, often used together: constrain the weights (L3.1.07: Mean-variance optimisation), improve the covariance matrix, and replace the raw expected returns with something more stable. The rest of the lesson takes the last two.

Shrinking the covariance matrix

A covariance matrix for $N$ assets has $N(N+1)/2$ numbers to estimate: 15 for five asset classes, 325 for 25 shares, 125,250 for 500. With a few years of monthly data, many of them are mostly noise. Some sample correlations come out too high, others too low, and the optimiser builds on exactly those errors. A portfolio that looks very safe in sample turns out much riskier in reality.

Shrinkage blends the sample matrix $S$ with a simple target $F$ : $\hat\Sigma = \delta F + (1 - \delta) S$ . The target has few parameters, so little noise, but it is biased. Olivier Ledoit and Michael Wolf proposed the constant-correlation target: keep each sample variance, and replace every correlation by their average. They also gave a formula for the best $\delta$ from the data itself.

Illustrative data25 made-up shares

True covariances come from a single-index model (L3.1.11: Factor models and Fama-French): betas from 0.5 to 1.5, market volatility 16%, specific volatilities from 15% to 35%. 100 simulated histories at each length; each one is used to build the global minimum-variance portfolio, whose true volatility is then measured. The best possible, using the true matrix, is 11.5%.

History	Sample: true vol.	Sample: claimed	Shrunk: true vol.	Shrunk: claimed	Average δ*
3 years	20.8%	6.5%	13.6%	11.7%	0.83
5 years	15.0%	8.8%	13.1%	11.9%	0.70
10 years	12.8%	10.2%	12.4%	11.9%	0.50

With three years of data the sample-based portfolio claims a volatility of 6.5% and delivers 20.8%. Shrinkage cuts the true risk to 13.6% and makes the claim far more honest. With five years it still helps (15.0% against 13.1%); with ten, the gap is small and $\delta^*$ falls, as it should. Shrinkage is not free: on the module's five asset classes, whose correlations range from −0.10 to 0.75, the constant-correlation target is a poor description, and with ten years of data shrinkage makes the matrix very slightly worse on average (error 0.0123 against 0.0120). It pays most when there are many assets and short histories.

Try itShrink a covariance matrix

History

History 1

Shrinkage intensity δ0.76

The minimum-variance portfolio built from the estimate: its true volatility and the volatility the estimate claims

True volatilityVolatility the estimate claimsUsing the true matrix

Ledoit-Wolf δ*0.76

Average sample correlation r̄0.27

Best δ for this history0.50

True volatility at δ = 0.7613.1%

Claimed volatility11.5%

With no shrinkage (δ = 0)21.2%

With the true matrix11.5%

25 made-up shares, true covariances from a single-index model; 36 monthly returns, so the sample matrix estimates 325 numbers from 900 data points. Ledoit and Wolf choose δ to make the whole matrix as accurate as possible on average, not to minimise this one portfolio's risk, so the best δ for a given history can differ.

Black-Litterman: start from the market

Expected returns are the harder problem. Fischer Black and Robert Litterman's answer, developed at Goldman Sachs, is not to estimate them from history at all. Start from the expected returns that would make the market portfolio optimal, then adjust only where you have a view.

Step 1: reverse optimisation. If everyone holds the market and the market is optimal (the CAPM logic of L3.1.09: CAPM, beta and the security market line), the first-order condition $\mu - r_f\mathbf 1 = \delta\Sigma w_{mkt}$ can be read backwards. Given the market weights, it gives the market-implied excess returns $\Pi = \delta\Sigma w_{mkt}$ , with $\delta = (\mathbb E[R_m] - r_f)/\sigma_m^2$ . Feed $\Pi$ to the optimiser and it hands back the market portfolio: a sensible, diversified anchor.

Step 2: views. Each view is a portfolio (a row of $P$ ) and a forecast for it (an entry of $Q$ ), with an uncertainty (a diagonal entry of $\Omega$ ). “Property will beat cash by 2%” is an absolute view; “global equities will beat government bonds by 6%” a relative one.

Step 3: blend. The Black-Litterman expected returns are a precision-weighted average of $\Pi$ and the views. A confident view pulls hard; a vague one barely moves anything. Because the blend works through $\Sigma$ , a view on one asset also moves the assets correlated with it, and the optimal portfolio is the market portfolio plus a tilt towards each view.

Why it works

The market weights already encode a consensus about every asset. Black-Litterman asks you to state only where you disagree and how strongly, and leaves the rest of the portfolio where the market has it. The naive alternative, typing your views into the optimiser as raw expected returns, sends the weights flying for the reasons above.

Sort itA problem with the means or with the covariances?

Which input does each statement mainly concern?

Ten years of data give a 95% interval of about ±10% around equities' expected return.
Switching from monthly to daily data sharpens the estimate a lot.
A 500-share model has 125,250 numbers to estimate.
Black-Litterman replaces historical averages with market-implied returns.
Ledoit-Wolf pulls extreme sample correlations towards the average.

The formal version

The standard error of a mean, and why frequency does not help. Split $T$ years into $n = mT$ periods of length $1/m$ , with independent returns of mean $\mu/m$ and variance $\sigma^2/m$ . The annualised average is $m$ times the average period return:

\operatorname{var}(\hat\mu) = m^2 \cdot \frac{\sigma^2/m}{mT} = \frac{\sigma^2}{T} \qquad\Longrightarrow\qquad \operatorname{se}(\hat\mu) = \frac{\sigma}{\sqrt T}

The $m$ cancels: only the span $T$ matters. For the variance, the number of observations does matter: with normal returns, $\operatorname{se}(\hat\sigma) \approx \sigma/\sqrt{2n}$ .

Why the optimiser magnifies. The weights are linear in the expected returns, so a change $\Delta\mu$ moves them by $\Delta w = \Sigma^{-1}\Delta\mu/\delta$ (M-E03: Matrix inverse and linear systems). Raising government bonds by 0.5%:

\Delta w = \frac{1}{3.305}\,\Sigma^{-1}\!\begin{pmatrix} 0.000 \\ 0.000 \\ 0.005 \\ 0.000 \\ 0.000 \end{pmatrix} = \begin{pmatrix} 0.135 \\ 0.035 \\ 1.139 \\ -0.724 \\ -0.003 \end{pmatrix}

Write $\Sigma = \sum_k \lambda_k v_k v_k^\top$ with eigenvalues $\lambda_k$ and eigenvectors $v_k$ . Then $\Sigma^{-1} = \sum_k \lambda_k^{-1} v_k v_k^\top$ : the direction with the smallest eigenvalue is amplified most. Here it is $v_5 \approx \begin{pmatrix} 0.10 & 0.03 & 0.79 & -0.61 & 0.01 \end{pmatrix}^\top$ , mostly long government bonds and short corporate bonds, a portfolio with a volatility of only 3.0% (against 28.5% for the largest). Any error in the forecast spread between the two bonds is divided by that tiny variance.

Shrinkage: choosing δ. Measure the error of $\hat\Sigma = \delta F + (1-\delta)S$ by the expected squared distance from the truth, summed over all entries. Suppose for a moment that $F$ were fixed and $S$ unbiased ( $\mathbb E[s_{ij}] = \sigma_{ij}$ ). Then

\begin{aligned} L(\delta) &= \sum_{i,j}\mathbb E\big[(\delta f_{ij} + (1-\delta)s_{ij} - \sigma_{ij})^2\big] = \sum_{i,j}\Big[\delta^2(f_{ij}-\sigma_{ij})^2 + (1-\delta)^2\operatorname{var}(s_{ij})\Big] \\ L'(\delta) &= 2\delta\sum_{i,j}(f_{ij}-\sigma_{ij})^2 - 2(1-\delta)\sum_{i,j}\operatorname{var}(s_{ij}) = 0 \\ \delta^* &= \frac{\sum_{i,j}\operatorname{var}(s_{ij})}{\sum_{i,j}\operatorname{var}(s_{ij}) + \sum_{i,j}(f_{ij}-\sigma_{ij})^2} \end{aligned}

Shrink more when the sample is noisy (large variances, short history) and less when the target is far from the truth. Ledoit and Wolf estimate each piece from the data, allow for the target being estimated too, and get $\delta^* = \max\{0, \min\{1, \kappa/T\}\}$ with $\kappa = (\pi - \rho)/\gamma$ : $\pi$ sums the variances of the sample covariances, $\rho$ their covariances with the target, and $\gamma$ the squared distance between target and sample. The $1/T$ is why the intensity falls as the history grows. The constant-correlation target is $f_{ii} = s_{ii},\; f_{ij} = \bar r\sqrt{s_{ii}s_{jj}}$ , with $\bar r$ the average of the $N(N-1)/2$ sample correlations.

Black-Litterman, derived. Treat the unknown expected excess returns $\mu$ as uncertain. The prior says $\mu \sim N(\Pi, \tau\Sigma)$ ; the scalar τ says how far from equilibrium they might be. The views say $P\mu = Q + \varepsilon$ with $\varepsilon \sim N(0, \Omega)$ . The best combined estimate minimises the two squared surprises, each weighted by its precision:

\begin{aligned} J(\mu) &= (\mu - \Pi)^\top(\tau\Sigma)^{-1}(\mu - \Pi) + (P\mu - Q)^\top\Omega^{-1}(P\mu - Q) \\ \nabla J &= 2(\tau\Sigma)^{-1}(\mu - \Pi) + 2P^\top\Omega^{-1}(P\mu - Q) = 0 \\ \big[(\tau\Sigma)^{-1} + P^\top\Omega^{-1}P\big]\,\mu &= (\tau\Sigma)^{-1}\Pi + P^\top\Omega^{-1}Q \end{aligned}

That is a linear system for $\mu$ ; solving it gives the first form below. (It is also the mean of the Bayesian posterior.) The matrix inversion lemma turns it into the second form, which only needs a $K\times K$ inverse for $K$ views:

The last line follows by substituting $\mu_{BL} = \Pi + \tau\Sigma P^\top m$ , with $m = (P\tau\Sigma P^\top + \Omega)^{-1}(Q - P\Pi)$ , into $(\delta\Sigma)^{-1}\mu$ : the $\Pi$ part gives $w_{mkt}$ and the $\Sigma$ cancels in the rest.

Setting Ω with a confidence level

For a single view

p

, the posterior is

p^\top\mu_{BL} = p^\top\Pi + c\,(q - p^\top\Pi)

with

c = p^\top\tau\Sigma p / (p^\top\tau\Sigma p + \omega)

. This lesson sets

\omega = \frac{1-c}{c}\,p^\top\tau\Sigma p

, so

c

is the share of the way the view moves its portfolio from the market's figure. With

c = 0.5

,

\Omega = \operatorname{diag}(P\tau\Sigma P^\top)

, the common choice of He and Litterman. With several correlated views, each moves a little less or more than

c

.

Try it

Start with both views at 50% confidence. Push the property view down and watch property's weight fall, and equities and bonds shift a little too, through their correlations. Then raise a confidence to 95% and drop it to 5%. Compare the right-hand panel: the same views typed straight into the optimiser.

Try itBlend your views with the market

View 1: property beats cash by 2.0% a year

View2.0%

Confidence (weight on the view)50%

View 2: global equities beat government bonds by 6.0% a year

View6.0%

Confidence (weight on the view)50%

Expected excess return	Global eq.	Gov. bonds	Property
House assumptions	4.00%	0.50%	3.00%
Market-implied Π	4.54%	0.28%	4.19%
Black-Litterman	4.83%	0.06%	3.41%
Views plugged in	6.50%	0.50%	2.00%

Black-Litterman weights

Global eq.63.0%

Gov. bonds22.0%

Property0.6%

Market weights

Views plugged straight into the optimiser

Global eq.118.4%

Gov. bonds91.3%

Property−45.8%

Market weights

Black-Litterman: total in risky assets85.6%

Plugged in: total in risky assets164.0%

Weights are w = (δΣ)⁻¹μ with δ = 3.02, so they need not add up to 100%: the rest is cash, or borrowing if negative. With no views, Black-Litterman returns exactly the market weights.

Worked example

Black-Litterman with three asset classes

Illustrative dataThree asset classes

Three of the module's asset classes: global equities, government bonds and property, with the module's volatilities and correlations. Their market weights are taken as 50%, 35%, 15%, the market's expected excess return as 3% and $\tau = 0.05$ .

From market weights and two views to a portfolio

Step 1 of 8

1
Market variance and risk aversion
$\begin{aligned} \Sigma &= \begin{pmatrix} 0.02560 & -0.00096 & 0.01728 \\ -0.00096 & 0.00360 & 0.00108 \\ 0.01728 & 0.00108 & 0.03240 \end{pmatrix},\qquad w_{mkt} = \begin{pmatrix} 0.50 \\ 0.35 \\ 0.15 \end{pmatrix} \\[4pt] \Sigma w_{mkt} &= \begin{pmatrix} 0.01506 \\ 0.00094 \\ 0.01388 \end{pmatrix} \\[4pt] \sigma_m^2 &= w_{mkt}^\top\Sigma w_{mkt} = 0.00994 \quad (\sigma_m = 9.97\%) \\[4pt] \delta &= \frac{\mathbb E[R_m] - r_f}{\sigma_m^2} = \frac{0.03}{0.00994} = 3.018 \end{aligned}$

Regulatory lens · looking ahead

Estimation error is a model risk, and supervisors expect firms to manage it: the PRA's supervisory statement SS1/23 on model risk management for banks asks for model limitations and uncertainty in inputs to be identified and monitored. The same covariance problem returns in market risk, where VaR models estimate correlations between hundreds of risk factors from a year or two of data (M3.2: Market risk).

Spark Atoms

Spark AtomError maximisers

In 1989 Richard Michaud asked in the Financial Analysts Journal whether “optimized” portfolios were optimal, and answered that optimisers tend to maximise the errors in their inputs. He later proposed resampling: optimise on many simulated input sets and average the weights.

Spark AtomHoney, I shrunk

Ledoit and Wolf's 2004 paper on the constant-correlation target is titled “Honey, I Shrunk the Sample Covariance Matrix”. The idea itself goes back to Charles Stein, who showed in 1956 that when three or more means are estimated together, shrinking the estimates can beat estimating each on its own.

Key takeaways

The standard error of an estimated mean is σ/√T and depends only on the length of the history; volatilities are far easier to estimate.
Optimal weights are Σ⁻¹(μ − rf)/δ, so errors in μ are magnified along the low-risk long-short directions of Σ: optimisers are error maximisers.
Shrinkage, δF + (1 − δ)S, trades a little bias for much less noise. Ledoit-Wolf's constant-correlation target helps most with many assets and short histories.
Black-Litterman starts from Π = δΣw_mkt, the returns that make the market optimal, and blends in views (P, Q, Ω) by precision.
The Black-Litterman portfolio is the market portfolio plus a tilt along each view, sized by how far the view is from the market and how confident it is.

Reaction

Check what stuck

Question 1 · objective 1

An asset's annual returns have a volatility of 20%. What is the standard error of its average return estimated from 25 years of data, in %?

%

Question 2 · objective 1

You switch from monthly to daily returns over the same ten years. Which estimate improves a lot?

AThe expected returnBThe volatilityCBoth, equallyDNeither

Question 3 · objective 3

A sample covariance is 0.020 and the constant-correlation target puts it at 0.012. With shrinkage intensity δ = 0.3, what is the shrunk estimate? Give 4 decimal places.

Question 4 · objective 4

Two assets, market weights 60% and 40%, risk aversion δ = 2.5. Asset 1 has variance 0.04 and covariance 0.01 with asset 2. What is asset 1's market-implied excess return Π₁, in %?

%

Question 5 · objective 4

The market implies that a view portfolio earns 3%. You believe 5%, with confidence 50% in the lesson's sense (Ω = PτΣP′). What does Black-Litterman expect for that portfolio, in %?

%

Question 6 · objective 4

Why does Black-Litterman with no views hold exactly the market portfolio?

ABecause τ is smallBBecause Π was built by reverse optimisation from the market weights, so optimising on Π gives them backCBecause Ω is diagonalDBecause the market portfolio has the highest return

Glossary

Estimation error: The gap between an estimate made from limited data and the true value it is trying to measure.
Standard error: The typical size of the estimation error: the standard deviation of the estimate across all the samples you might have drawn.
Error maximisation: The tendency of an optimiser to load up on the assets whose expected returns were overestimated, and to short those underestimated.
Shrinkage: Replacing an estimate by a weighted average of it and a simpler, more stable target, trading a little bias for much less noise.
Shrinkage target: The simple, stable matrix a shrinkage estimator pulls the sample covariance matrix towards.
Constant-correlation target: A covariance matrix that keeps each asset's own variance but sets every correlation to the same average value.
Reverse optimisation: Working backwards from portfolio weights to the expected returns that would make them optimal: Π = δΣw.
Market-implied returns Π: The excess returns implied by the market portfolio, Π = δΣw_mkt: what investors must expect for the market weights to be optimal.
View: A forecast about one asset, or about the difference between assets, expressed as a row of P and a number in Q.
View uncertainty Ω: The variance of each view's error: small for a view you are sure of, large for a guess.
τ (tau): A small number that scales the uncertainty of the market-implied returns, τΣ; often between 0.01 and 0.1.

Deep Dives & Origins

Deep Dives

A. Meucci, Risk and Asset Allocation, Springer, 2005, the chapters on estimation risk and Bayesian allocation. Advanced.
T. Idzorek, “A Step-by-Step Guide to the Black-Litterman Model”, working paper, 2004. Intermediate.

Origins

R. O. Michaud, “The Markowitz Optimization Enigma: Is ‘Optimized’ Optimal?”, Financial Analysts Journal 45(1), 1989.
M. J. Best and R. R. Grauer, “On the Sensitivity of Mean-Variance-Efficient Portfolios to Changes in Asset Means: Some Analytical and Computational Results”, Review of Financial Studies 4(2), 1991.
V. K. Chopra and W. T. Ziemba, “The Effect of Errors in Means, Variances, and Covariances on Optimal Portfolio Choice”, Journal of Portfolio Management 19(2), 1993.
R. C. Merton, “On Estimating the Expected Return on the Market”, Journal of Financial Economics 8(4), 1980.
F. Black and R. Litterman, “Global Portfolio Optimization”, Financial Analysts Journal 48(5), 1992.
G. He and R. Litterman, “The Intuition Behind Black-Litterman Model Portfolios”, Goldman Sachs Investment Management Research, 1999.
O. Ledoit and M. Wolf, “Honey, I Shrunk the Sample Covariance Matrix”, Journal of Portfolio Management 30(4), 2004.
O. Ledoit and M. Wolf, “Improved Estimation of the Covariance Matrix of Stock Returns with an Application to Portfolio Selection”, Journal of Empirical Finance 10(5), 2003.
Prudential Regulation Authority, SS1/23 “Model risk management principles for banks”, 2023.

Watch this lesson

Video version

Prefer to watch? The same lesson as a 16-minute video will sit here, made from this page's script.