Math Atom · M-D03

Linear regression

10 minMath AtomStatistics

By the end you will be able to

  1. 1Write the linear model y = a + bx + e and read its intercept, slope, fitted values and residuals.
  2. 2Find the least-squares line, b = s_xy / s_x² and a = ȳ − b x̄, and explain why it minimises the sum of squared residuals.
  3. 3Measure the fit with R², and connect it to the correlation r.
  4. 4Write a regression in matrix form, XᵀXβ = Xᵀy, and spot when one point drives the answer.

Why you need it

A scatter of points hides a relationship. A regression line turns it into two numbers, a slope and an intercept, and a third, R², says how much of the scatter the line explains.

These Core Path lessons use it:

You need covariance and correlation from M-C10: Covariance and correlation, the sample mean and variance from M-D01: Estimating mean and volatility, and straight lines from M-A09: Functions and graphs. The derivation uses M-B04: Optimisation and Lagrange multipliers, and the matrix form M-E03: Matrix inverse and linear systems.

The idea

The line with the least squared error

Many lines pass close to a cloud of points. Which is best? Measure each point's vertical miss, square it, and draw the square. Move the line until the total shaded area is as small as you can make it, then press “Best fit”.

Fit a line by eye

Make the squares small
00336699
Your liney = 3 + 0.3x
Sum of squared residuals14.45
Smallest possible1.286

Your total area is 11.24 times the smallest possible.

The line with the smallest total area is the least-squares line. Each vertical miss is a residual. Squaring means one big miss costs more than several small ones, which is why a single stray point can tilt the line.

The rules

The model. A line, plus what it misses: the linear model.

Least squares. The best a and b come from setting two partial derivatives to zero, as in M-B04: Optimisation and Lagrange multipliers.

How good is the fit? Compare what the line leaves over with what there was to explain.

The matrix form. The same answer as a linear system, ready for more than one x.

The rules, for reference

yi=a+b xi+ei,b=∑(xi−xˉ)(yi−yˉ)∑(xi−xˉ)2=sxysx2=r sysx,a=yˉ−b xˉy_i = a + b\,x_i + e_i, \qquad b = \frac{\sum (x_i - \bar x)(y_i - \bar y)}{\sum (x_i - \bar x)^2} = \frac{s_{xy}}{s_x^2} = r\,\frac{s_y}{s_x}, \qquad a = \bar y - b\,\bar x
R2=1−∑ei2∑(yi−yˉ)2=r2,XTX β^=XTy  ⇒  β^=(XTX)−1XTyR^2 = 1 - \frac{\sum e_i^2}{\sum (y_i - \bar y)^2} = r^2, \qquad X^{\mathsf T}X\,\hat{\boldsymbol\beta} = X^{\mathsf T}\mathbf{y} \;\Rightarrow\; \hat{\boldsymbol\beta} = (X^{\mathsf T}X)^{-1}X^{\mathsf T}\mathbf{y}

Try it

Two years of illustrative monthly returns: a market index along the bottom and a fund up the side, built from a true line plus noise. Turn the noise to zero, then up to 6, and watch R². Press “New sample” a few times to see the fitted slope wobble. Then add one extreme month and see how far it moves the line.

Try itRegression explorer

Regression and correlation calculator →
Least-squares line True line
-20%-10%0%10%20%-12%-8%-4%0%4%8%12%market (x)fund (y)
Months n24
Slope b1.401
Intercept a0.322%
Correlation r0.964
R²0.929

The fitted slope wobbles around the true slope from sample to sample. More noise, bigger wobbles and a lower R².

Worked examples

1. A regression by hand

Step 1 of 5
  1. 1

    The set-up

    Five points: x = 1, 2, 3, 4, 5 and y = 2, 3, 5, 4, 6. The means are x̄ = 3 and ȳ = 4.

2. Beta and alpha from twelve months

Step 1 of 4
  1. 1

    The set-up

    Twelve illustrative monthly returns, in %, of a market index (x) and a fund (y):

    MonthMarket xᵢFund yᵢ
    11.4−1.2
    215.2
    3−2.7−6.1
    4−9.6−11.4
    5−3.2−6.4
    63.59
    71.12.4
    85.35.9
    90.2−0.2
    10−4.2−3.8
    112.52.6
    123.44.1

3. One mistyped value

Step 1 of 4
  1. 1

    The set-up

    The data of example 1, but the last y has been typed as 16 instead of 6: y = 2, 3, 5, 4, 16.

Common mistakes

Each line below is a tempting slip. Spot what is wrong, then check.

Reaction

Check what stuck

Question 1 · objective 2

The sample covariance of x and y is 6 and the sample variance of x is 4. What is the least-squares slope?

Question 2 · objective 2

x̄ = 2, ȳ = 7 and the slope is 1.5. What is the intercept?

Question 3 · objective 3

A regression has SSR = 3 and SST = 12. What is R²?

Question 4 · objective 1

A point lies below the fitted line. What is the sign of its residual?

Question 5 · objective 3

The correlation between x and y is 0.6. What is R² of the regression of y on x?

Glossary

Linear model
y = a + bx + e: a straight line in x plus a residual. a is the intercept, b the slope.
Fitted value (ŷ)
ŷ = a + bx: the point on the fitted line above x.
Residual (e)
e = y − ŷ: the vertical gap between a point and the line. Positive above the line, negative below.
Least squares
Least squares: choose a and b to make the sum of squared residuals as small as possible.
Sum of squared residuals (SSR)
Sum of squared residuals, Σe²: what the line leaves unexplained.
Total sum of squares (SST)
Total sum of squares, Σ(y − ȳ)²: the spread of y around its mean, before any line.
R²
R² = 1 − SSR/SST: the share of the spread of y that the line explains, from 0 to 1. With one x it equals r².
Normal equations
XᵀXβ = Xᵀy: the linear system whose solution is the least-squares estimate β̂.
Outlier
A point far from the pattern of the others. Because residuals are squared, it can pull the line a long way.

Back to top ↑