Confidence intervals and hypothesis tests · RiskAtoms Academy

By the end you will be able to

1Compute the standard error s/√n of a mean and a t confidence interval x̄ ± t* · s/√n, and say what 95% refers to.
2Run a hypothesis test: state H₀, compute the t statistic and the p-value, and reach a verdict at a level α.
3Test a regression slope with its standard error, and link a test to the matching interval.
4Explain why trying many strategies or variables produces false alarms, and correct for it.

Why you need it

Every estimate comes with noise. An interval says how much, and a test asks whether an estimate is further from zero than noise alone would put it.

These Core Path lessons use it:

L3.1.10: Using and testing CAPM
Testing CAPM asks whether intercepts are zero and slopes behave as predicted: t statistics on regression coefficients.
L3.1.15: Estimation error, shrinkage and Black-Litterman
Estimation error: the standard errors of estimated means are large, which is why shrinkage helps.

You need the sample mean and standard deviation from M-D01: Estimating mean and volatility, the normal distribution from M-C06: The normal distribution, sums of independent variables from M-C11: Combining random variables, and regression from M-D03: Linear regression.

The idea

An interval that is right 95% of the time

Here the true mean is known: it is 5. Draw a sample of 12 values, estimate the mean, and put a band around it. Do it 20 times. Most bands catch the true mean; a few, in red, miss.

Twenty samples, twenty intervals

True mean μ = 5, n = 12 each

Missed this time0 of 20

Intervals so far20

Missed so far0

Miss rate0%

Each line is x̄ ± t* · s/√n from one sample of 12; the dot is x̄. Keep pressing: the miss rate settles near 5%. With real data μ is unknown, so you cannot tell which intervals missed.

Each band is a 95% confidence interval. The 95% is a property of the method: over many samples, about 95 in 100 such intervals contain the true value. Its width comes from the standard error.

The rules

The standard error. How much an average moves from sample to sample.

The confidence interval. Built from the t distribution.

The hypothesis test. A null hypothesis, a t statistic and its p-value.

Testing a regression slope. The same recipe for b from M-D03: Linear regression.

Many tests at once. Why the best of many results is not as good as it looks: multiple testing.

The rules, for reference

\mathrm{SE}(\bar x) = \frac{s}{\sqrt n}, \qquad \bar x \pm t^*_{n-1}\,\mathrm{SE}, \qquad t = \frac{\bar x - \mu_0}{\mathrm{SE}}, \qquad p = P\big(|T_{n-1}| \ge |t|\big)

\mathrm{SE}(b) = \frac{s_e}{\sqrt{\sum (x_i - \bar x)^2}}, \;\; s_e = \sqrt{\tfrac{\mathrm{SSR}}{n-2}}, \qquad P(\text{any false alarm in } k) = 1 - (1 - \alpha)^k

Try it

Illustrative monthly returns, with a true mean you set. Is the sample mean far enough from zero to rule out luck? Start with a true mean of 0.5% and sd of 3%, and change the number of months. Then set the true mean to 0 and press “New sample” several times: how often is μ = 0 wrongly rejected?

Try itIs the mean different from zero?

Probability distribution calculator →

True mean μ (%)0.5

True sd σ (%)3

Months n60

Confidence level

The t distribution with 59 degrees of freedom: where t would fall if μ were 0. Shaded: the 5% of it beyond ±2.001.

Sample mean x̄0.69%

Sample sd s2.88%

Standard error s/√n0.37%

95% interval−0.05 to 1.44

t statistic1.868

p-value0.0667

Verdict at α = 5%Do not reject

0 lies inside the interval, and p is above α. Not rejecting is not proof that μ = 0.

Worked examples

1. An interval for a mean

Step 1 of 4

1
The set-up
Ten illustrative monthly returns, in %: 1.2, −0.8, 2.5, 0.4, −1.6, 3.1, 0.9, −0.3, 1.8, 0.6. Find a 95% interval for the true mean.

2. Is a regression slope zero?

Step 1 of 4

1
The set-up
The five points of M-D03: Linear regression worked example 1: b = 0.9, SSR = 1.9 and Σ(x − x̄)² = 10. Test H₀: true slope = 0 at α = 5%.

3. A backtest, and twenty backtests

Step 1 of 4

1
The set-up
A trading strategy is tested on 60 months of past data (a backtest). Its monthly returns have mean 0.5% and sd 3%. Is the mean return different from zero?

Common mistakes

Each line below is a tempting slip. Spot what is wrong, then check.

Reaction

Check what stuck

Question 1 · objective 1

A sample of 16 values has standard deviation 4. What is the standard error of the mean?

Question 2 · objective 1

x̄ = 10, SE = 1 and t* = 2. What is the upper end of the interval?

Question 3 · objective 2

x̄ = 1.2 with standard error 0.4. What is the t statistic for H₀: μ = 0?

Question 4 · objective 2

A test gives p = 0.20. What does that mean?

AIf H₀ were true, a t this far from zero would turn up in about 20% of samples.BThere is a 20% chance that H₀ is true.CThere is an 80% chance that H₀ is false.DThe effect is 20% of its standard error.

Question 5 · objective 4

Ten independent tests at α = 5%, all of true null hypotheses. What is the chance of at least one false alarm, in %?

Glossary

Standard error (SE): The standard deviation of an estimate across repeated samples, estimated from one sample. For a mean, s/√n.
t distribution: Student's t distribution: a bell like the normal with fatter tails, indexed by its degrees of freedom. It describes (x̄ − μ)/(s/√n) for normal data.
Confidence interval: A range built by a method that catches the true value in a stated share of samples, such as 95%: estimate ± t* × SE.
Null hypothesis (H₀): The null hypothesis H₀: the claim a test checks, such as μ = 0.
t statistic: (estimate − value under H₀) / standard error: how many standard errors the estimate is from H₀.
p-value: The probability, if H₀ were true, of a t statistic at least as far from zero as the one observed.
Significance level (α): The significance level α: the p-value below which H₀ is rejected, and so the false-alarm rate when H₀ is true.
Multiple testing: Running many tests at once. The chance that at least one gives a false alarm grows fast with the number of tests.
Bonferroni correction: Testing each of k hypotheses at α/k, so that the chance of any false alarm is at most α.