STAT 331 · Lecture 6 · Multiple Linear Regression: Intro and Estimation

# Lecture 6: Multiple Linear Regression: Intro and Estimation

This lecture writes multiple linear regression in matrix form, estimates \(\boldsymbol\beta\) and \(\sigma^2\), and finds the properties of the estimators (Lecture 6 · p.1–41).

Contents
01 · Recap: matrix calculus, random vectors, and the multivariate normal 02 · Data example and interpreting regression coefficients 03 · Estimating β: least squares and maximum likelihood 04 · Properties of the LS estimator, fitted values, and residuals 05 · Estimating σ², calculating the estimates, and practice

## 01 · Recap: matrix calculus, random vectors, and the multivariate normal

Plan · Slides p.1–7

Step 1: Take the derivative of a scalar with respect to a vector (Slides p.3).

Step 2: Find the expectation and variance of a random vector after a linear change (Slides p.4).

Step 3: Define the multivariate normal distribution (Slides p.5).

Step 4: Use four properties of the multivariate normal (Slides p.6).

Step 5: Write the regression model as \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\) (Slides p.7).

The slides give no data. The examples use small 2×2 or 3×3 matrices.

### 01.1 Notation Slides p.1–2

Slide p.1: title "Lecture 6: Multiple Linear Regression: Intro & Estimation". Slide p.2: section page "Recap". Added

Vector: a column of numbers, with a bold lower-case name, for example \(\boldsymbol{y}=(y_1,\dots,y_k)^T\).

Matrix: a rectangular table of numbers, with a bold upper-case name, for example \(\boldsymbol{A}\). "\(k\times k\)" means \(k\) rows and \(k\) columns.

Transpose \(^T\): changes rows into columns. \(\boldsymbol{a}^T\) is a row.

Scalar: one number, with a plain name, for example \(z\) or \(\sigma^2\).

Identity matrix \(\boldsymbol{I}\): 1 on the diagonal, 0 at all other positions. Zero vector \(\boldsymbol{0}\): 0 at all positions.

### 01.2 Matrix calculus Slides p.3

**What** Slides p.3

A partial derivative \(\partial z/\partial y_j\) is the rate of change of \(z\) when only \(y_j\) changes. A symmetric matrix has \(\boldsymbol{A}^T=\boldsymbol{A}\), that is, \(a_{ij}=a_{ji}\).

Matrix Calculus · Lecture 6 · p.3 Let \(z=f(y_1,\dots,y_k)\) and \(\boldsymbol{y}=\left[\begin{array}{c}y_1\\ \vdots\\ y_k\end{array}\right]\), then \[\frac{\partial z}{\partial \boldsymbol{y}}=\left[\begin{array}{c}\frac{\partial z}{\partial y_1}\\ \frac{\partial z}{\partial y_2}\\ \vdots\\ \frac{\partial z}{\partial y_k}\end{array}\right]\]

- If \(z=\boldsymbol{a}^T\boldsymbol{y}\), where \(\boldsymbol{a}=(a_1,\dots,a_k)^T\) is a vector, then \(\frac{\partial z}{\partial \boldsymbol{y}}=\boldsymbol{a}\)
- If \(z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}\) where \(\boldsymbol{A}\) is a \(k\times k\) matrix, then \(\frac{\partial z}{\partial \boldsymbol{y}}=\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\)
and if \(\boldsymbol{A}\) is symmetric, then: \(\frac{\partial z}{\partial \boldsymbol{y}}=2\boldsymbol{A}\boldsymbol{y}\)

Linear form \(\boldsymbol{a}^T\boldsymbol{y}=a_1y_1+\dots+a_ky_k\): each term is a constant times one entry. Quadratic form \(\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}\): each term is a constant times two entries.

**How** Added

How · Take the derivative with respect to a vector

- Find the type of \(z\): linear form or quadratic form.
- For a linear form, write \(\partial z/\partial\boldsymbol{y}=\boldsymbol{a}\).
- For a quadratic form, compare each \(a_{ij}\) with \(a_{ji}\). All equal: write \(2\boldsymbol{A}\boldsymbol{y}\). Else: write \(\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\).
- Write the result as a \(k\times1\) column.

**Self-check:** The result has length \(k\).

**Self-check:** Expand \(z\) and find each \(\partial z/\partial y_j\). The result must agree with step 2 or 3.

**Example** Added

Let \(k=2\), \(\boldsymbol{y}=(y_1,y_2)^T\), and use the point \(\boldsymbol{y}=(1,2)^T\).

Example (i): linear form, \(\boldsymbol{a}=(3,5)^T\).

1

Expand \(z\) How step 1

\[z=\boldsymbol{a}^T\boldsymbol{y}=3y_1+5y_2\]

2

Partial derivatives Self-check 2

\[\frac{\partial z}{\partial y_1}=3,\qquad \frac{\partial z}{\partial y_2}=5\]

3

Column How step 2

\[\frac{\partial z}{\partial\boldsymbol{y}}=\left[\begin{array}{c}3\\5\end{array}\right]=\boldsymbol{a}\]

Example (ii): symmetric quadratic form, \(\boldsymbol{A}=\left[\begin{array}{cc}2&1\\1&3\end{array}\right]\), \(a_{12}=a_{21}=1\).

1

Expand \(z\) How step 1

\[z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}=\left[\begin{array}{cc}y_1&y_2\end{array}\right]\left[\begin{array}{c}2y_1+y_2\\y_1+3y_2\end{array}\right]\]

\[=y_1(2y_1+y_2)+y_2(y_1+3y_2)\]

\[=2y_1^2+2y_1y_2+3y_2^2\]

2

Partial derivatives Self-check 2

\[\frac{\partial z}{\partial y_1}=4y_1+2y_2,\qquad \frac{\partial z}{\partial y_2}=2y_1+6y_2\]

3

Rule \(2\boldsymbol{A}\boldsymbol{y}\) How step 3

\[2\boldsymbol{A}\boldsymbol{y}=2\left[\begin{array}{c}2y_1+y_2\\y_1+3y_2\end{array}\right]=\left[\begin{array}{c}4y_1+2y_2\\2y_1+6y_2\end{array}\right]\]

Agrees with step 2.

4

Put in \(\boldsymbol{y}=(1,2)^T\)

\[z=2(1)^2+2(1)(2)+3(2)^2=2+4+12=18\]

\[\frac{\partial z}{\partial\boldsymbol{y}}=\left[\begin{array}{c}4(1)+2(2)\\2(1)+6(2)\end{array}\right]=\left[\begin{array}{c}8\\14\end{array}\right]\]

Example (iii): non-symmetric quadratic form, \(\boldsymbol{A}=\left[\begin{array}{cc}1&2\\0&3\end{array}\right]\), \(a_{12}=2\ne a_{21}=0\).

1

Expand \(z\) How step 1

\[z=\left[\begin{array}{cc}y_1&y_2\end{array}\right]\left[\begin{array}{c}y_1+2y_2\\3y_2\end{array}\right]\]

\[=y_1^2+2y_1y_2+3y_2^2\]

2

Partial derivatives Self-check 2

\[\frac{\partial z}{\partial y_1}=2y_1+2y_2,\qquad \frac{\partial z}{\partial y_2}=2y_1+6y_2\]

3

Rule \(\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\) How step 3

\[\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}=\left[\begin{array}{c}y_1+2y_2\\3y_2\end{array}\right]+\left[\begin{array}{c}y_1\\2y_1+3y_2\end{array}\right]\]

\[=\left[\begin{array}{c}2y_1+2y_2\\2y_1+6y_2\end{array}\right]\]

\(\boldsymbol{A}^T=\left[\begin{array}{cc}1&0\\2&3\end{array}\right]\). Agrees with step 2.

4

Put in \(\boldsymbol{y}=(1,2)^T\)

\[z=1^2+2(1)(2)+3(2)^2=1+4+12=17\]

\[\frac{\partial z}{\partial\boldsymbol{y}}=\left[\begin{array}{c}2(1)+2(2)\\2(1)+6(2)\end{array}\right]=\left[\begin{array}{c}6\\14\end{array}\right]\]

**Why** Added

Linear form:

1

\[z=\boldsymbol{a}^T\boldsymbol{y}=\sum_{i=1}^k a_iy_i\]

2

\[\frac{\partial z}{\partial y_j}=a_j\]

Only term \(i=j\) contains \(y_j\).

3

\[\frac{\partial z}{\partial\boldsymbol{y}}=(a_1,\dots,a_k)^T=\boldsymbol{a}\]

Definition on p.3. ∎

Quadratic form. \(a_{ij}\) is the entry of \(\boldsymbol{A}\) in row \(i\), column \(j\).

1

\[z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}=\sum_{i=1}^k\sum_{l=1}^k y_i\,a_{il}\,y_l\]

2

\[\frac{\partial z}{\partial y_j}=\sum_{l=1}^k a_{jl}y_l+\sum_{i=1}^k y_i a_{ij}\]

Product rule: \(y_j\) is in terms \(i=j\) and \(l=j\).

3

\[=(\boldsymbol{A}\boldsymbol{y})_j+\sum_{i=1}^k a_{ij}y_i\]

First sum: row \(j\) of \(\boldsymbol{A}\) times \(\boldsymbol{y}\).

4

\[=(\boldsymbol{A}\boldsymbol{y})_j+(\boldsymbol{A}^T\boldsymbol{y})_j\]

\(a_{ij}\) is entry \((j,i)\) of \(\boldsymbol{A}^T\).

5

\[\frac{\partial z}{\partial\boldsymbol{y}}=\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\]

6

\[=\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}\boldsymbol{y}=2\boldsymbol{A}\boldsymbol{y}\]

Symmetric: \(\boldsymbol{A}^T=\boldsymbol{A}\). ∎

Unit 03 uses these rules on \(S(\boldsymbol\beta)\).

### 01.3 Properties of random vectors Slides p.4

**What** Slides p.4

A random variable is a number whose value comes from a random result. A random vector has random variables as entries. The expectation \(E[\cdot]\) is the long-run average, taken entry by entry for a vector.

The variance of a random variable is its expected squared distance from its expectation. The covariance \(\mathrm{cov}(y_i,y_j)=E[(y_i-E[y_i])(y_j-E[y_j])]\) shows how much two variables change together. The covariance matrix \(\mathrm{Var}(\boldsymbol{y})\) has entry \((i,j)\) equal to \(\mathrm{cov}(y_i,y_j)\), and variances on the diagonal. Below, \(\boldsymbol{a}\), \(\boldsymbol{b}\), and \(\boldsymbol{A}\) are constants.

Properties of Random Vectors · Lecture 6 · p.4

- \(E[\boldsymbol{a}]=\boldsymbol{a}\)
- \(E[\boldsymbol{a}^T\boldsymbol{y}+\boldsymbol{b}]=\boldsymbol{a}^TE[\boldsymbol{y}]+\boldsymbol{b}\)
- \(E[\boldsymbol{A}\boldsymbol{y}]=\boldsymbol{A}E[\boldsymbol{y}]\)
- \(\mathrm{Var}(\boldsymbol{y})=E[(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{y}-E[\boldsymbol{y}])^T]\)
- \(\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})=\boldsymbol{a}^T\mathrm{Var}(\boldsymbol{y})\boldsymbol{a}\)
- \(\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})=\boldsymbol{A}\mathrm{Var}(\boldsymbol{y})\boldsymbol{A}^T\)

For the variance, \(\boldsymbol{A}\) goes on the left and \(\boldsymbol{A}^T\) on the right. \(\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})\) is a scalar. \(\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})\) is a matrix.

**How** Added

How · Expectation and variance after a linear change

- Write \(\boldsymbol{A}\) (\(m\times k\)), \(E[\boldsymbol{y}]\) (\(k\times1\)), and \(\mathrm{Var}(\boldsymbol{y})\) (\(k\times k\)).
- Calculate \(\boldsymbol{A}E[\boldsymbol{y}]\) (\(m\times1\)).
- Calculate \(\boldsymbol{A}\mathrm{Var}(\boldsymbol{y})\), then multiply on the right by \(\boldsymbol{A}^T\) (\(m\times m\)).

**Self-check:** The variance matrix is symmetric.

**Self-check:** The diagonal entries are \(\ge 0\).

**Example** Added

Let \(E[\boldsymbol{y}]=(10,20)^T\), \(\mathrm{Var}(\boldsymbol{y})=\boldsymbol{I}\) (2×2), \(\boldsymbol{A}=\left[\begin{array}{cc}1&2\\3&4\end{array}\right]\), and \(\boldsymbol{a}=(1,1)^T\).

1

Expectation How step 2

\[E[\boldsymbol{A}\boldsymbol{y}]=\boldsymbol{A}E[\boldsymbol{y}]=\left[\begin{array}{c}1(10)+2(20)\\3(10)+4(20)\end{array}\right]=\left[\begin{array}{c}50\\110\end{array}\right]\]

2

Variance How step 3

\[\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})=\boldsymbol{A}\,\boldsymbol{I}\,\boldsymbol{A}^T=\boldsymbol{A}\boldsymbol{A}^T\]

\[=\left[\begin{array}{cc}1&2\\3&4\end{array}\right]\left[\begin{array}{cc}1&3\\2&4\end{array}\right]\]

\[=\left[\begin{array}{cc}1+4&3+8\\3+8&9+16\end{array}\right]=\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\]

Symmetric, positive diagonal: both self-checks pass.

3

Scalar case

\[\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})=\boldsymbol{a}^T\boldsymbol{I}\boldsymbol{a}=1^2+1^2=2\]

\(\boldsymbol{a}^T\boldsymbol{y}=y_1+y_2\).

Order is important: \(\boldsymbol{A}^T\boldsymbol{A}=\left[\begin{array}{cc}10&14\\14&20\end{array}\right]\) is a different matrix.

**Why** Added

1

\[\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})=E[(\boldsymbol{A}\boldsymbol{y}-E[\boldsymbol{A}\boldsymbol{y}])(\boldsymbol{A}\boldsymbol{y}-E[\boldsymbol{A}\boldsymbol{y}])^T]\]

Definition of Var, p.4.

2

\[=E[(\boldsymbol{A}\boldsymbol{y}-\boldsymbol{A}E[\boldsymbol{y}])(\boldsymbol{A}\boldsymbol{y}-\boldsymbol{A}E[\boldsymbol{y}])^T]\]

\(E[\boldsymbol{A}\boldsymbol{y}]=\boldsymbol{A}E[\boldsymbol{y}]\).

3

\[=E[\boldsymbol{A}(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{A}(\boldsymbol{y}-E[\boldsymbol{y}]))^T]\]

4

\[=E[\boldsymbol{A}(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{y}-E[\boldsymbol{y}])^T\boldsymbol{A}^T]\]

\((\boldsymbol{B}\boldsymbol{C})^T=\boldsymbol{C}^T\boldsymbol{B}^T\).

5

\[=\boldsymbol{A}\,E[(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{y}-E[\boldsymbol{y}])^T]\,\boldsymbol{A}^T\]

Constant matrices move out of \(E\).

6

\[=\boldsymbol{A}\,\mathrm{Var}(\boldsymbol{y})\,\boldsymbol{A}^T\]

One row \(\boldsymbol{a}^T\) gives the scalar rule. ∎

Unit 04 uses this rule to find \(\mathrm{Var}[\hat{\boldsymbol\beta}]\).

### 01.4 Multivariate normal distribution Slides p.5

**What** Slides p.5

\(N(0,1)\) is the standard normal: mean 0, variance 1. "i.i.d." means independent and identically distributed. Two variables are independent if the value of one does not change the distribution of the other. The multivariate normal (MVN) is a distribution of a random vector, set fully by its mean \(\boldsymbol\mu\) and covariance matrix \(\boldsymbol\Sigma\).

The integral of the density \(f(\boldsymbol{y})\) over a region gives the probability that \(\boldsymbol{y}\) is in that region. The determinant \(|\boldsymbol\Sigma|\) is one number from a square matrix; for 2×2, \(\left|\begin{array}{cc}p&q\\r&s\end{array}\right|=ps-qr\). The inverse \(\boldsymbol\Sigma^{-1}\) satisfies \(\boldsymbol\Sigma\boldsymbol\Sigma^{-1}=\boldsymbol{I}\).

Multivariate Normal Distribution · Lecture 6 · p.5 Let \(\boldsymbol{z}=(z_1,\dots,z_n)^T\) be a random vector of i.i.d standard normal random variables, i.e. \(z_i\overset{iid}{\sim}N(0,1)\).
Then \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\) has multivariate normal distribution, i.e.: \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) where \(E[\boldsymbol{y}]=\boldsymbol\mu\) and \(\mathrm{Var}(\boldsymbol{y})=\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T\).
\(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\), if: \[f(\boldsymbol{y})=\frac{1}{(2\pi)^{\frac n2}|\boldsymbol\Sigma|^{\frac12}}\exp\left\{-\frac12(\boldsymbol{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\boldsymbol{y}-\boldsymbol\mu)\right\}\]

The slide gives two equivalent descriptions: a construction from \(n\) independent standard normals, and a density. Both use only \(\boldsymbol\mu\) (\(n\times1\)) and \(\boldsymbol\Sigma\) (\(n\times n\)).

**How** Added

How · Get the MVN parameters and the density

- Read \(\boldsymbol\mu\), the added constant vector.
- Calculate \(\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T\).
- Write \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\).
- For \(f(\boldsymbol{y})\), calculate \(|\boldsymbol\Sigma|\), then \(\boldsymbol\Sigma^{-1}\), then \((\boldsymbol{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\boldsymbol{y}-\boldsymbol\mu)\).
- Put these three values into the density formula.

**Self-check:** \(\boldsymbol\Sigma\) is symmetric and \(|\boldsymbol\Sigma|>0\). Else \(|\boldsymbol\Sigma|^{1/2}\) and \(\boldsymbol\Sigma^{-1}\) do not exist.

**Self-check:** At \(\boldsymbol{y}=\boldsymbol\mu\), the exponential term is \(e^0=1\).

**Example** Added

Use \(\boldsymbol{A}\) from Section 01.3, \(n=2\), \(\boldsymbol\mu=(10,20)^T\), and \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\).

1

Mean How step 1

\[\boldsymbol\mu=\left[\begin{array}{c}10\\20\end{array}\right]\]

2

Covariance matrix How step 2

\[\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T=\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\]

Same product as Section 01.3.

3

Distribution How step 3

\[\boldsymbol{y}\sim MVN\left(\left[\begin{array}{c}10\\20\end{array}\right],\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\right)\]

4

Determinant How step 4

\[|\boldsymbol\Sigma|=5(25)-11(11)=125-121=4\]

5

Density at \(\boldsymbol{y}=\boldsymbol\mu\) Self-check 2

\[f(\boldsymbol\mu)=\frac{1}{(2\pi)^{\frac22}\,4^{\frac12}}\exp\{0\}\]

\[=\frac{1}{2\pi\times2}=\frac{1}{4\pi}=0.07957747\]

Second example: \(\boldsymbol\mu=\boldsymbol{0}\), \(\boldsymbol\Sigma=\boldsymbol{I}\) (2×2), \(\boldsymbol{y}=(1,-1)^T\). Added

1

Determinant and inverse How step 4

\[|\boldsymbol{I}|=1(1)-0(0)=1,\qquad \boldsymbol{I}^{-1}=\boldsymbol{I}\]

2

Quadratic form How step 4

\[(\boldsymbol{y}-\boldsymbol{0})^T\boldsymbol{I}(\boldsymbol{y}-\boldsymbol{0})=1^2+(-1)^2=2\]

3

Density How step 5

\[f(\boldsymbol{y})=\frac{1}{(2\pi)^{1}\,1^{\frac12}}\exp\left\{-\frac12(2)\right\}\]

\[=\frac{e^{-1}}{2\pi}=\frac{0.36787944}{6.28318531}=0.05854983\]

4

Check with two one-dimensional densities

\[\frac{1}{\sqrt{2\pi}}e^{-\frac12(1)^2}\times\frac{1}{\sqrt{2\pi}}e^{-\frac12(-1)^2}=\frac{e^{-1}}{2\pi}=0.05854983\]

\(N(0,1)\) density: \(\frac{1}{\sqrt{2\pi}}e^{-y^2/2}\).

The cloud centers near \(\boldsymbol\mu=(10,20)\). It slopes up-right because \(\mathrm{cov}(y_1,y_2)=11>0\). It is taller than wide because \(\mathrm{Var}(y_2)=25>\mathrm{Var}(y_1)=5\).

[figure]
Figure 1-1. 300 simulated points \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\) with \(\boldsymbol{A}=\left[\begin{array}{cc}1&2\\3&4\end{array}\right]\) (blue). Orange cross: mean \(\boldsymbol\mu\).

**Why** Added

Each \(z_i\) has mean 0 and variance 1, and different \(z_i\) have covariance 0. Then \(E[\boldsymbol{z}]=\boldsymbol{0}\) and \(\mathrm{Var}(\boldsymbol{z})=\boldsymbol{I}\).

1

\[E[\boldsymbol{y}]=E[\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu]=\boldsymbol{A}E[\boldsymbol{z}]+\boldsymbol\mu\]

p.4 expectation rules.

2

\[=\boldsymbol{A}\boldsymbol{0}+\boldsymbol\mu=\boldsymbol\mu\]

3

\[\mathrm{Var}(\boldsymbol{y})=\mathrm{Var}(\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu)=\mathrm{Var}(\boldsymbol{A}\boldsymbol{z})\]

An added constant does not change variance.

4

\[=\boldsymbol{A}\,\mathrm{Var}(\boldsymbol{z})\,\boldsymbol{A}^T\]

p.4 variance rule.

5

\[=\boldsymbol{A}\,\boldsymbol{I}\,\boldsymbol{A}^T=\boldsymbol{A}\boldsymbol{A}^T=\boldsymbol\Sigma\]

∎

### 01.5 Properties of \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) Slides p.6

**What** Slides p.6

A marginal distribution is the distribution of some entries of a vector. A conditional distribution \(\boldsymbol{y}_1\mid\boldsymbol{y}_2\) is the distribution of \(\boldsymbol{y}_1\) when \(\boldsymbol{y}_2\) is known. \(\Sigma_{ij}=\mathrm{cov}(y_i,y_j)\) is entry \((i,j)\) of \(\boldsymbol\Sigma\).

Properties of \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 6 · p.6 Linearity: If \(\boldsymbol{u}=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\), then: \[\boldsymbol{u}\sim MVN(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d},\,\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T)\] Marginal Distribution: If \(\tilde{\boldsymbol{y}}=(y_1,\dots,y_m)^T\subset\boldsymbol{y}\) is a vector subset of \(\boldsymbol{y}\), then \(\tilde{\boldsymbol{y}}\) is MVN-distributed.
Conditional Distribution: If \(\boldsymbol{y}=(\boldsymbol{y}_1^T,\boldsymbol{y}_2^T)^T\), then \(\boldsymbol{y}_1^T\mid\boldsymbol{y}_2^T\) is MVN-distributed.
Independence: If \(\Sigma_{ij}=0\), then \(\boldsymbol{y}_i\) and \(\boldsymbol{y}_j\) are independent.

Linearity: \(\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\) stays MVN. Its mean and variance follow the p.4 rules.

Marginal: any set of entries is MVN. One entry is normal.

Conditional: when the lower block \(\boldsymbol{y}_2\) is known, the upper block \(\boldsymbol{y}_1\) is MVN. \(\boldsymbol{y}_1^T\mid\boldsymbol{y}_2^T\) means \(\boldsymbol{y}_1\mid\boldsymbol{y}_2\). The lecture does not give its parameters.

Independence: for MVN, covariance 0 gives independence.

**How** Added

A linear combination is a sum of entries, each multiplied by a constant.

How · Use linearity to find the distribution of a new variable

- Write \(\boldsymbol{u}=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\). For a subset, each row of \(\boldsymbol{C}\) has one 1 and all other entries 0.
- Mean: calculate \(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d}\).
- Variance: calculate \(\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T\).
- Write \(\boldsymbol{u}\sim MVN(\text{step 2},\text{step 3})\). For one entry, write \(N(\cdot,\cdot)\).

**Self-check:** Columns of \(\boldsymbol{C}\) = length of \(\boldsymbol{y}\). Length of \(\boldsymbol{d}\) = rows of \(\boldsymbol{C}\).

**Example** Added

Use \(\boldsymbol{y}\sim MVN\left(\left[\begin{array}{c}10\\20\end{array}\right],\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\right)\) from Section 01.4.

(i) Linearity: \(u=y_1+y_2\).

1

Write \(u=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\) How step 1

\[\boldsymbol{C}=\left[\begin{array}{cc}1&1\end{array}\right],\qquad \boldsymbol{d}=0\]

2

Mean How step 2

\[\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d}=1(10)+1(20)+0=30\]

3

Variance How step 3

\[\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T=\left[\begin{array}{cc}1&1\end{array}\right]\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\left[\begin{array}{c}1\\1\end{array}\right]\]

\[=\left[\begin{array}{cc}5+11&11+25\end{array}\right]\left[\begin{array}{c}1\\1\end{array}\right]\]

\[=\left[\begin{array}{cc}16&36\end{array}\right]\left[\begin{array}{c}1\\1\end{array}\right]=16+36=52\]

4

Distribution How step 4

\[u=y_1+y_2\sim N(30,\,52)\]

(ii) Marginal: \(\boldsymbol{C}=\left[\begin{array}{cc}1&0\end{array}\right]\), \(\boldsymbol{d}=0\). Mean \(1(10)+0(20)=10\), variance \(\Sigma_{11}=5\): \(y_1\sim N(10,5)\).

(iii) Independence: \(\boldsymbol\Sigma=\boldsymbol{I}\), \(\boldsymbol\mu=\boldsymbol{0}\), \(\Sigma_{12}=0\). The joint density is the product of two \(N(0,1)\) densities: both give 0.05854983 at \((1,-1)\) (Section 01.4).

**Why** Added

Let \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\), with \(\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T\).

1

\[\boldsymbol{u}=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}=\boldsymbol{C}(\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu)+\boldsymbol{d}\]

p.5 construction.

2

\[=(\boldsymbol{C}\boldsymbol{A})\boldsymbol{z}+(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d})\]

3

\[\Rightarrow\ \boldsymbol{u}\sim MVN\big(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d},\ (\boldsymbol{C}\boldsymbol{A})(\boldsymbol{C}\boldsymbol{A})^T\big)\]

Form "matrix × \(\boldsymbol{z}\) + vector": p.5 applies.

4

\[(\boldsymbol{C}\boldsymbol{A})(\boldsymbol{C}\boldsymbol{A})^T=\boldsymbol{C}\boldsymbol{A}\boldsymbol{A}^T\boldsymbol{C}^T\]

\((\boldsymbol{C}\boldsymbol{A})^T=\boldsymbol{A}^T\boldsymbol{C}^T\).

5

\[=\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T\]

∎ Marginal: \(\boldsymbol{C}\) selects entries.

The lecture states the conditional and independence properties without proof. Unit 04 uses linearity to get the distribution of \(\hat{\boldsymbol\beta}\).

### 01.6 Recap: multiple linear regression Slides p.7

**What** Slides p.7

Multiple linear regression (MLR) models an outcome \(y_i\) as an intercept, plus a linear combination of \(p\) covariates \(x_{i1},\dots,x_{ip}\), plus a random error \(\epsilon_i\). The regression coefficients \(\beta_0,\dots,\beta_p\) and the error variance \(\sigma^2\) are unknown constants. \(n\) is the number of observations.

Recap: Multiple Linear Regression · Lecture 6 · p.7 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_Px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] We can write this: \[\left[\begin{array}{c}y_1\\y_2\\ \vdots\\y_n\end{array}\right]=\left[\begin{array}{cccccc}1&x_{11}&x_{12}&\dots&x_{1p}\\1&x_{21}&x_{22}&\dots&x_{2p}\\ \vdots&\vdots&\vdots&\vdots&\vdots\\1&x_{n1}&x_{n2}&\dots&x_{np}\end{array}\right]\left[\begin{array}{c}\beta_0\\ \beta_1\\ \vdots\\ \beta_p\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\ \epsilon_2\\ \vdots\\ \epsilon_n\end{array}\right]\] Or more simply: \[\begin{array}{rl}&\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon,\qquad \boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\\ \iff&\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\end{array}\]

Notation note The slide writes \(\beta_P\) in the first line and \(\beta_p\) after. Both mean the last index \(p\).

Row \(i\) is one observation. Column 1 of \(\boldsymbol{X}\) is all 1s, for the intercept \(\beta_0\). Column \(j+1\) is covariate \(j\). \(\boldsymbol{X}\) is \(n\times(p+1)\); \(\boldsymbol\beta\) is \((p+1)\times1\).

"\(\epsilon_i\) i.i.d. \(N(0,\sigma^2)\)" equals \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\): means 0, variances \(\sigma^2\), covariances 0.

[figure]
Figure 1-2. Block sizes in \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\). Left of the dashed line: the column of 1s. Columns of \(\boldsymbol{X}\) = length \(p+1\) of \(\boldsymbol\beta\). \(\boldsymbol{X}\boldsymbol\beta\) is \(n\times1\).

**How** Added

How · Write \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\)

- For \(\boldsymbol{y}\), put \(y_1,\dots,y_n\) in one column.
- For \(\boldsymbol{X}\), write \((1,x_{i1},\dots,x_{ip})\) as row \(i\).
- For \(\boldsymbol\beta\), put \(\beta_0,\dots,\beta_p\) in one column.
- Check: row \(i\) of \(\boldsymbol{X}\) times \(\boldsymbol\beta\) equals \(\beta_0+\beta_1x_{i1}+\dots+\beta_px_{ip}\).

**Self-check:** \(\boldsymbol{X}\) is \(n\times(p+1)\), with a first column of 1s.

**Example** Added

Let \(n=3\), \(p=2\), covariates \((1,4)\), \((2,5)\), \((3,6)\). For the algebra only, let \(\boldsymbol\beta=(1,2,3)^T\).

1

Write \(\boldsymbol{X}\) How step 2

\[\boldsymbol{X}=\left[\begin{array}{ccc}1&1&4\\1&2&5\\1&3&6\end{array}\right]\quad(3\times3=n\times(p+1))\]

2

Calculate \(\boldsymbol{X}\boldsymbol\beta\) How step 4

\[\boldsymbol{X}\boldsymbol\beta=\left[\begin{array}{c}1+2(1)+3(4)\\1+2(2)+3(5)\\1+2(3)+3(6)\end{array}\right]=\left[\begin{array}{c}1+2+12\\1+4+15\\1+6+18\end{array}\right]=\left[\begin{array}{c}15\\20\\25\end{array}\right]\]

Then \(\boldsymbol{y}\sim MVN\big((15,20,25)^T,\ \sigma^2\boldsymbol{I}\big)\).

**Why** Added

Use linearity (Section 01.5) with \(\boldsymbol{C}=\boldsymbol{I}\) and constant \(\boldsymbol{d}=\boldsymbol{X}\boldsymbol\beta\).

1

\[\boldsymbol{y}=\boldsymbol{I}\boldsymbol\epsilon+\boldsymbol{X}\boldsymbol\beta\]

2

\[\boldsymbol{y}\sim MVN(\boldsymbol{I}\boldsymbol{0}+\boldsymbol{X}\boldsymbol\beta,\ \boldsymbol{I}(\sigma^2\boldsymbol{I})\boldsymbol{I}^T)\]

p.6 linearity.

3

\[\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\ \sigma^2\boldsymbol{I})\]

Reverse: \(\boldsymbol{d}=-\boldsymbol{X}\boldsymbol\beta\) gives \(\iff\). ∎

### 01.7 Practice Added

**Q1.** Let \(z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}\) with \(\boldsymbol{A}=\left[\begin{array}{cc}4&1\\1&2\end{array}\right]\). Find \(\frac{\partial z}{\partial\boldsymbol{y}}\) at \(\boldsymbol{y}=(1,-1)^T\). Check entry by entry.

Answer

1

\[\boldsymbol{A}^T=\boldsymbol{A}\ \Rightarrow\ \frac{\partial z}{\partial\boldsymbol{y}}=2\boldsymbol{A}\boldsymbol{y}=2\left[\begin{array}{c}4y_1+y_2\\y_1+2y_2\end{array}\right]=\left[\begin{array}{c}8y_1+2y_2\\2y_1+4y_2\end{array}\right]\]

2

\[z=4y_1^2+2y_1y_2+2y_2^2,\quad \frac{\partial z}{\partial y_1}=8y_1+2y_2,\quad \frac{\partial z}{\partial y_2}=2y_1+4y_2\]

Agrees with step 1.

3

\[\left.\frac{\partial z}{\partial\boldsymbol{y}}\right|_{(1,-1)}=\left[\begin{array}{c}8(1)+2(-1)\\2(1)+4(-1)\end{array}\right]=\left[\begin{array}{c}6\\-2\end{array}\right]\]

**Q2.** Let \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\), \(\boldsymbol\mu=(1,2)^T\), \(\boldsymbol\Sigma=\left[\begin{array}{cc}2&1\\1&3\end{array}\right]\). Find the distribution of \(u=y_1-y_2\).

Answer

1

\[u=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d},\quad \boldsymbol{C}=\left[\begin{array}{cc}1&-1\end{array}\right],\ \boldsymbol{d}=0\]

2

\[\boldsymbol{C}\boldsymbol\mu=1(1)+(-1)(2)=-1\]

3

\[\boldsymbol{C}\boldsymbol\Sigma=\left[\begin{array}{cc}2-1&1-3\end{array}\right]=\left[\begin{array}{cc}1&-2\end{array}\right]\]

4

\[\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T=\left[\begin{array}{cc}1&-2\end{array}\right]\left[\begin{array}{c}1\\-1\end{array}\right]=1+2=3\]

5

\[u\sim N(-1,\,3)\]

p.6 linearity.

**Q3.** An MLR has \(n=50\) and \(p=3\), with \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\). (a) Give the dimensions of \(\boldsymbol{X}\), \(\boldsymbol\beta\), and \(\mathrm{Var}(\boldsymbol{y})\). (b) Show \(E[\boldsymbol{y}]=\boldsymbol{X}\boldsymbol\beta\) with the random-vector properties.

Answer

(a) \(\boldsymbol{X}\): \(n\times(p+1)=50\times4\). \(\boldsymbol\beta\): \(4\times1\). \(\mathrm{Var}(\boldsymbol{y})=\sigma^2\boldsymbol{I}\): \(50\times50\).

(b)

1

\[E[\boldsymbol{y}]=E[\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon]\]

2

\[=\boldsymbol{X}\boldsymbol\beta+E[\boldsymbol\epsilon]\]

p.4: \(E[\boldsymbol{a}]=\boldsymbol{a}\) for constant \(\boldsymbol{X}\boldsymbol\beta\).

3

\[=\boldsymbol{X}\boldsymbol\beta+\boldsymbol{0}=\boldsymbol{X}\boldsymbol\beta\]

## 02 · Data example and interpreting regression coefficients

Plan · Slides p.8–16

Step 1 (Slides p.8–9): meet the data set satisfaction.csv.

Step 2 (Slides p.10–11): read the summary numbers and the scatter plots.

Step 3 (Slides p.12): list the questions of the lecture.

Step 4 (Slides p.13): write the model in scalar form and matrix form.

Step 5 (Slides p.14–16): interpret the intercept \(\beta_0\) and each slope \(\beta_j\).

Basic words · Added

Variable: one column of a data table.

Observation: one row of a data table. In this unit, one patient.

The subscript \(i\) gives the observation. The subscript \(j\) gives the covariate.

### 02.1 Title page Slides p.8

Slides p.8 Slide p.8 repeats the title "Lecture 6: Multiple Linear Regression: Intro & Estimation".

Slides p.1–7 are a recap. New material starts on Slides p.9.

### 02.2 Data example: satisfaction.csv Slides p.9

**What** Slides p.9

Data Example · Lecture 6 · p.9
satisfaction.csv contains data on \(n = 46\) hospital patients. The variables are:

- Satisfaction: degree of satisfaction with quality of care (higher values = more)
- Age: age of patient in years
- Severity: severity score for patient condition
- Stress: patient self reported degree of stress

| Variable | Role

| Satisfaction | outcome \(y_i\)

| Age | covariate 1, \(x_{i1}\)

| Severity | covariate 2, \(x_{i2}\)

| Stress | covariate 3, \(x_{i3}\)

\(n=46\) observations, \(p=3\) covariates.

### 02.3 Exploratory data analysis: first rows and summary numbers Slides p.10

Exploratory data analysis (EDA): a check of the range, center, and shape of the data before you make a model.

**What** Slides p.10

First 6 rows:

| Row | Satisfaction | Age | Severity | Stress

| 1 | 48 | 50 | 51 | 2.3

| 2 | 57 | 36 | 46 | 2.3

| 3 | 66 | 40 | 48 | 2.2

| 4 | 70 | 41 | 44 | 1.8

| 5 | 89 | 28 | 43 | 1.8

| 6 | 36 | 49 | 54 | 2.9

Summary numbers:

|  | Satisfaction | Age | Severity | Stress

| Minimum | 26.00 | 22.00 | 41.00 | 1.800

| 1st quartile | 48.25 | 31.25 | 48.00 | 2.100

| Median | 60.00 | 37.50 | 50.50 | 2.300

| Mean | 61.57 | 38.39 | 50.43 | 2.287

| 3rd quartile | 76.75 | 44.75 | 53.00 | 2.475

| Maximum | 92.00 | 55.00 | 62.00 | 2.900

Quartile: a value that divides the sorted data into four equal parts. About 25% of the data are below the 1st quartile, about 75% below the 3rd.

Median: the middle value of the sorted data.

Sample mean \(\bar y=\frac1n\sum_{i=1}^n y_i\).

**How** Added

Read the summary numbers of one variable

- Find the column of the variable.
- Read the minimum and maximum. They give the range.
- Read the median and mean. They give the center.
- Read the 1st and 3rd quartiles. The middle 50% of the data are between them.

**Self-check:** minimum ≤ 1st quartile ≤ median ≤ 3rd quartile ≤ maximum. The mean is in the range.

**Example · Read the two tables** Added

Patient 1: \(y_1=48\), \(x_{11}=50\), \(x_{12}=51\), \(x_{13}=2.3\).

| Variable | Range | Mean

| Satisfaction | 26 to 92 | 61.57 (median 60)

| Age | 22 to 55 years | 38.39

| Severity | 41 to 62 | 50.43

| Stress | 1.8 to 2.9 | 2.287

**Self-check:** For Age, 22 ≤ 31.25 ≤ 37.50 ≤ 44.75 ≤ 55, and 38.39 is between 22 and 55.

No covariate has the value 0. Section 02.7 uses this fact. ▲

### 02.4 Scatter plot matrix Slides p.11

**What** Slides p.11

Slides p.11 shows a scatter plot matrix of the four variables.

Scatter plot: one point for each observation. The horizontal position is one variable, the vertical position is another.

Scatter plot matrix (pairs plot): a grid with the scatter plot of each pair of variables. 4 variables give a 4×4 grid. The diagonal cells show only the names.

**How** Added

Find the scatter plot of one pair of variables

- Find variable A on the diagonal. Note its row.
- Find variable B on the diagonal. Note its column.
- Go to row A, column B. The vertical axis is A. The horizontal axis is B.
- Points from lower left to upper right show a positive association.
- Points from upper left to lower right show a negative association.

**Self-check:** Each axis covers the minimum to the maximum of its variable in the summary table.

**Example · Satisfaction against Age** Added

Row 1, column 2: vertical axis Satisfaction, horizontal axis Age.

[figure]
Figure 2-1. Satisfaction against Age, 46 patients. Orange: patient 1 (Age 50, Satisfaction 48).

The points go from upper left to lower right (step 5). Satisfaction and Age have a negative association.

The axes cover 26 to 92 and 22 to 55, as in the summary table.

This plot does not hold Severity and Stress fixed. Section 02.7 does. ▲

### 02.5 Motivating questions Slides p.12

**What** Slides p.12

Motivating Questions · Lecture 6 · p.12

- Is mean satisfaction associated with stress, conditional on age and severity?
- How does mean satisfaction differ for older patients vs younger patients with the same severity and stress scores?
- Given a patient's stress, age, and severity, can we predict their satisfaction?
- Later: is the (conditional) association between stress and satisfaction different for older patients than for younger patients?
- Many others...

Association: when one variable changes, the mean of another variable changes.

Conditional on: with the other variables held at fixed values.

Questions 1 and 2 hold other variables fixed. Figure 2-1 cannot do this. We need a model with all three covariates.

### 02.6 The multiple linear regression model Slides p.13

**What** Slides p.13

Multiple Linear Regression · Lecture 6 · p.13 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_Px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\quad i=1,\dots,n\]
Or
\[\boldsymbol{y}=\boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon},\quad \boldsymbol{\epsilon}\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\] \[\iff\] \[\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol{\beta},\sigma^2\boldsymbol{I})\]
Note: we assume \(p+1<n\)

This is the model of Section 01.6, with \(i=1,\dots,n\) added.

\(p+1<n\): there are fewer unknown coefficients than observations. It makes \(n-(p+1)\) positive. Unit 05 divides by it.

**How** Added

Write the model and \(\boldsymbol{X}\) for a data set

- Find the outcome and the covariates. Write \(n\) and \(p\).
- Write the scalar form with the variable names.
- Put \(n\) 1s in column 1 of \(\boldsymbol{X}\).
- Put covariate \(j\) in column \(j+1\).
- Check \(p+1<n\).

**Self-check:** \(\boldsymbol{X}\) is \(n\times(p+1)\). \(\boldsymbol{\beta}\) has \(p+1\) entries. Row \(i\) of \(\boldsymbol{X}\) matches row \(i\) of the data.

**Example · The model for satisfaction.csv** Added

**Step 1.** \(n=46\), \(p=3\).

**Step 2.**
\[\text{Satisfaction}_i=\beta_0+\beta_1\,\text{Age}_i+\beta_2\,\text{Severity}_i+\beta_3\,\text{Stress}_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\quad i=1,\dots,46\]
**Steps 3–4.** First three rows from Slides p.10:
\[\boldsymbol{X}=\left[\begin{array}{cccc}1&50&51&2.3\\1&36&46&2.3\\1&40&48&2.2\\\vdots&\vdots&\vdots&\vdots\end{array}\right]_{46\times4},\qquad \boldsymbol{\beta}=\left[\begin{array}{c}\beta_0\\\beta_1\\\beta_2\\\beta_3\end{array}\right]\]
**Step 5.** \(4<46\). The assumption is true.

**Self-check:** \(46\times4=n\times(p+1)\). Row 1 is patient 1: Age 50, Severity 51, Stress 2.3.

**Coefficient estimates used in this unit.** Estimate: a number calculated from the data in place of an unknown true value. A hat marks it, for example \(\hat\beta_1\).

Lecture 6 does not print the estimates. Lecture 7 · p.51 prints a 95% confidence interval for each coefficient. Confidence interval: a range of values for an unknown coefficient, calculated from the data.

The estimate is the midpoint of each interval:

1

\[\hat\beta_0=\frac{121.911727+195.0707761}{2}=\frac{316.9825031}{2}=158.4912516\approx158.491\]

2

\[\hat\beta_1=\frac{-1.575093+(-0.7081303)}{2}=\frac{-2.2832233}{2}=-1.1416117\approx-1.142\]

3

\[\hat\beta_2=\frac{-1.434831+0.5508228}{2}=\frac{-0.8840082}{2}=-0.4420041\approx-0.442\]

4

\[\hat\beta_3=\frac{-27.797859+0.8575324}{2}=\frac{-26.9403266}{2}=-13.4701633\approx-13.470\]

This unit uses \(\hat{\boldsymbol\beta}=(158.491,\,-1.142,\,-0.442,\,-13.470)^T\). Unit 03 calculates it. ▲

**Why** Added

The scalar form and the matrix form give the same model. Row \(i\) of \(\boldsymbol{X}\boldsymbol\beta\):

1

\[[\boldsymbol{X}\boldsymbol\beta]_i=\left[\begin{array}{cccc}1&x_{i1}&\cdots&x_{ip}\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]\]

2

\[=1\cdot\beta_0+x_{i1}\beta_1+\cdots+x_{ip}\beta_p\]

3

\[=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\]

Add \(\epsilon_i\) to get the scalar form. The form \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\) comes from linearity (Slides p.6, Section 01.6).

Check with patient 1, row \((1,50,51,2.3)\):

1

\[1(158.491)+50(-1.142)+51(-0.442)+2.3(-13.470)\]

2

\[=158.491-57.1-22.542-30.981\]

3

\[=47.868\]

This equals the scalar form \(\hat\beta_0+\hat\beta_1(50)+\hat\beta_2(51)+\hat\beta_3(2.3)\). ∎

### 02.7 Interpreting regression coefficients Slides p.14–16

Slides p.14–16 Slides p.14 and p.15 are the first parts of Slides p.16. This section follows p.16.

**What** Slides p.14–16

Interpreting Regression Coefficients · Lecture 6 · p.14–16
To interpret a parameter, isolate it!
\[E[y_i\,|\,x_{i1},\dots,x_{ip}]=\beta_0+\beta_1x_{i1}\cdots+\beta_Px_{ip}\]
What does \(\beta_0\) represent?
\[E[y_i\,|\,x_{i1}=\cdots=x_{ip}=0]=\beta_0+\beta_1(0)+\cdots+\beta_P(0)=\beta_0\]

- \(\implies\beta_0\) is the mean outcome when all covariates are set to 0
- May not be interpretable: e.g. what does it mean for age to be 0 in a sample of adults?

What does \(\beta_1\) represent?
\[E[y_i\,|\,x_{i1}=x_1,x_{i2}=x_2,\dots,x_{ip}=x_P]=\beta_0+\beta_1(x_1)+\beta_2(x_2)+\cdots+\beta_P(x_P)\] \[E[y_i\,|\,x_{i1}=(x_1+1),x_{i2}=x_2,\dots,x_{ip}=x_P]=\beta_0+\beta_1(x_1+1)+\cdots+\beta_P(x_P)\]

- \(E[y_i\,|\,x_{i1}=(x_1+1),x_{i2}=x_2,\dots,x_{ip}=x_P]-E[y_i\,|\,x_{i1}=x_1,x_{i2}=x_2,\dots,x_{ip}=x_P]=\beta_1\)
- More generally \(\beta_j\) represents the difference in mean outcome for a one-unit change in the \(j^{th}\) covariate, holding other covariates fixed.

Conditional expectation \(E[y_i\,|\,x_{i1},\dots,x_{ip}]\): the mean of \(y_i\) at the given covariate values. Read \(|\) as "given".

If 0 is outside the range of the data, \(\beta_0\) has no practical meaning.

"Holding other covariates fixed" is a necessary part of each slope interpretation.

**How** Added

Interpret one coefficient \(\beta_j\)

- Write \(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\).
- Intercept: put 0 for all covariates. The result is \(\beta_0\).
- Check in the summary table if 0 is in the range of each covariate.
- Slope: write the expression with covariate \(j\) at \(x_j\), then at \(x_j+1\).
- Give the other covariates the same values in both.
- Subtract. Only \(\beta_j\) remains.
- Write: a one-unit increase in covariate \(j\) changes the mean outcome by \(\beta_j\), other covariates held fixed.

**Self-check:** The sentence names the covariate and its unit, the direction and size, and "held fixed". With an estimate, it says "estimated mean outcome".

**Example · Interpret \(\hat\beta_0\) and \(\hat\beta_1\)** Added

**Step 1.**
\[\hat\beta_0+\hat\beta_1\,\text{Age}+\hat\beta_2\,\text{Severity}+\hat\beta_3\,\text{Stress}=158.491-1.142\,\text{Age}-0.442\,\text{Severity}-13.470\,\text{Stress}\]
**Step 2 (intercept).** Age = Severity = Stress = 0:

1

\[158.491-1.142(0)-0.442(0)-13.470(0)\]

2

\[=158.491=\hat\beta_0\]

**Step 3.** Minimum Age is 22, Severity 41, Stress 1.8 (Section 02.3). \(\hat\beta_0\) has no practical meaning. This is the second bullet of Slides p.15.

**Steps 4–5 (Age).** Hold Severity = 50, Stress = 2.3. Use Age 40, then 41.

1

\[\text{Age }40:\ 158.491-1.142(40)-0.442(50)-13.470(2.3)\]

2

\[=158.491-45.68-22.1-30.981\]

3

\[=59.73\]

1

\[\text{Age }41:\ 158.491-1.142(41)-0.442(50)-13.470(2.3)\]

2

\[=158.491-46.822-22.1-30.981\]

3

\[=58.588\]

**Step 6.**

1

\[58.588-59.73=-1.142=\hat\beta_1\]

\(-22.1\) and \(-30.981\) cancel.

**Step 7.** Each one-year increase in Age decreases the estimated mean satisfaction by 1.142, with Severity and Stress held fixed.

\(\hat\beta_2=-0.442\): each one-point increase in Severity decreases the estimated mean satisfaction by 0.442, with Age and Stress held fixed.

\(\hat\beta_3=-13.470\): each one-unit increase in Stress decreases the estimated mean satisfaction by 13.470, with Age and Severity held fixed.

Figure 2-2 holds Severity and Stress at their means, 50.43 and 2.287: \(158.491-0.442(50.43)-13.470(2.287)-1.142\,\text{Age}=105.395-1.142\,\text{Age}\) (3 decimals).

[figure]
Figure 2-2. Data (blue) and the estimated mean line (orange), slope \(\hat\beta_1=-1.142\). Green: Age 30 and 50. The line drops \(20\times1.142=22.84\).

Other fixed values, for example Severity = 60, move the line up or down. The slope stays \(-1.142\). ▲

**Why** Slides p.14–16

1

\[E[y_i\,|\,x_{i1},\dots,x_{ip}]=E[\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\,|\,x_{i1},\dots,x_{ip}]\]

Model, Slides p.13

2

\[=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+E[\epsilon_i]\]

Given covariates, first part is fixed

3

\[=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\]

\(E[\epsilon_i]=0\)

4

\[E[y_i\,|\,x_{i1}=\cdots=x_{ip}=0]=\beta_0+\beta_1(0)+\cdots+\beta_p(0)\]

5

\[=\beta_0\]

6

\[E[y_i\,|\,x_{ij}=x_j+1,\ \text{others fixed}]=\beta_0+\cdots+\beta_j(x_j+1)+\cdots+\beta_px_p\]

7

\[E[y_i\,|\,x_{ij}=x_j,\ \text{others fixed}]=\beta_0+\cdots+\beta_jx_j+\cdots+\beta_px_p\]

8

\[\text{line 6}-\text{line 7}=\beta_j(x_j+1)-\beta_jx_j\]

Other terms are equal and cancel

9

\[=\beta_jx_j+\beta_j-\beta_jx_j\]

10

\[=\beta_j\]

Line 8 needs equal values of the other covariates. The result does not depend on \(x_j\): Age 40 to 41 and Age 30 to 31 both give \(\beta_1\). ∎

### 02.8 Practice Added

Use \(\hat{\boldsymbol\beta}=(158.491,\,-1.142,\,-0.442,\,-13.470)^T\).

**Q1.** Interpret \(\hat\beta_3=-13.470\). Two patients have the same Age and Severity. Their Stress differs by 0.5. Find the difference in estimated mean satisfaction.

Answer

Each one-unit increase in Stress decreases the estimated mean satisfaction by 13.470, with Age and Severity held fixed.

\(0.5\times(-13.470)=-6.735\)

The patient with higher Stress has an estimated mean satisfaction 6.735 lower.

**Q2.** Two patients have Severity = 50 and Stress = 2.3. Their ages are 30 and 50. Find each estimated mean satisfaction and the difference. Write the difference with \(\hat\beta_1\).

Answer

1

\[\text{Age }30:\ 158.491-1.142(30)-0.442(50)-13.470(2.3)\]

2

\[=158.491-34.26-22.1-30.981\]

3

\[=71.15\]

1

\[\text{Age }50:\ 158.491-1.142(50)-0.442(50)-13.470(2.3)\]

2

\[=158.491-57.1-22.1-30.981\]

3

\[=48.31\]

Difference: \(48.31-71.15=-22.84\).

With \(\hat\beta_1\): \(20\times\hat\beta_1=20\times(-1.142)=-22.84\).

The Severity and Stress terms cancel. The older patient is 22.84 lower.

**Q3.** Patient 5 (Slides p.10) has Age 28, Severity 43, Stress 1.8. Write row 5 of \(\boldsymbol{X}\). Find the estimated mean satisfaction.

Answer

Row 5 of \(\boldsymbol{X}\): \((1,28,43,1.8)\).

1

\[1(158.491)+28(-1.142)+43(-0.442)+1.8(-13.470)\]

2

\[=158.491-31.976-19.006-24.246\]

3

\[=83.263\]

Estimate 83.263. Observed value 89.

## 03 · Estimating β: least squares and maximum likelihood

Plan · Slides p.17–22

Step 1: State the model (Slides p.17).

Step 2: Least squares: minimize \(S(\boldsymbol\beta)\) (Slides p.18–20).

Step 3: Solve a small data set by hand (Added).

Step 4: Maximum likelihood: link \(\ell\) to \(S(\boldsymbol\beta)\) (Slides p.21–22).

### 03.1 Title page: Estimating \(\boldsymbol\beta\) Slides p.17

Model: \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\). The errors are independent normal, each with variance \(\sigma^2\).

Estimator \(\hat{\boldsymbol\beta}\): a formula that gives a value for \(\boldsymbol\beta\) from the data.

The slides use the satisfaction data (\(n=46\), \(p=3\), Slides p.9). The example uses a smaller data set.

### 03.2 Least squares estimation Slides p.18–20

Slides p.18–19 build up p.20.

**What** Slides p.20

Least Squares Estimation · Lecture 6 · p.20 Want to minimize the sum of squares, i.e.: \[\begin{aligned} S(\boldsymbol\beta)&=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\\ &=\boldsymbol{y}^T\boldsymbol{y}-\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta-\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\\ &=\boldsymbol{y}^T\boldsymbol{y}-2\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta \end{aligned}\] \[\begin{aligned} \frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta&=0\\ (\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta&=\boldsymbol{X}^T\boldsymbol{y}\\ \hat{\boldsymbol\beta}&=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y} \end{aligned}\] assuming the columns of \(\boldsymbol{X}\) are linearly independent

Sum of squares: \(S(\boldsymbol\beta)=\sum_{i=1}^n\big(y_i-(\boldsymbol{X}\boldsymbol\beta)_i\big)^2\), one number.

Least squares (OLS, ordinary least squares): the estimate \(\hat{\boldsymbol\beta}\) is the \(\boldsymbol\beta\) with the smallest \(S(\boldsymbol\beta)\).

Linearly independent columns: no column of \(\boldsymbol{X}\) is a linear combination of the others.

Normal equations: \((\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\). With linearly independent columns, they have one solution.

**How** Added

Calculate \(\hat{\boldsymbol\beta}\) from data

- Write \(\boldsymbol{y}\) (\(n\times1\)) and \(\boldsymbol{X}\) (\(n\times(p+1)\), first column all 1s).
- Calculate \(\boldsymbol{X}^T\boldsymbol{X}\). Entry \((j,k)\) is \(\sum_i x_{ij}x_{ik}\).
- Calculate \(\boldsymbol{X}^T\boldsymbol{y}\). Entry \(j\) is \(\sum_i x_{ij}y_i\).
- Make sure that \(\det(\boldsymbol{X}^T\boldsymbol{X})\neq0\).
- Solve \((\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\).

**Self-check:** (1) \(\boldsymbol{X}^T\boldsymbol{X}\) is symmetric, and its entry \((1,1)\) is \(n\). (2) Entry 1 of \(\boldsymbol{X}^T\boldsymbol{y}\) is \(\sum_iy_i\). (3) \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}\) equals \(\boldsymbol{X}^T\boldsymbol{y}\).

**Example** Added

\(n=4\), one covariate (\(p=1\)), \(\boldsymbol\beta=(\beta_0,\beta_1)^T\).

Data: \(x=1,2,3,4\) and \(y=2,3,5,6\).

1

Write \(\boldsymbol{y}\) and \(\boldsymbol{X}\) How step 1

\[\boldsymbol{y}=\left[\begin{array}{c}2\\3\\5\\6\end{array}\right],\qquad \boldsymbol{X}=\left[\begin{array}{cc}1&1\\1&2\\1&3\\1&4\end{array}\right]\]

2

Calculate \(\boldsymbol{X}^T\boldsymbol{X}\) How step 2

\[(1,1):\ 1+1+1+1=4\]

\[(1,2)=(2,1):\ 1+2+3+4=10\]

\[(2,2):\ 1^2+2^2+3^2+4^2=1+4+9+16=30\]

\[\boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}4&10\\10&30\end{array}\right]\]

**Self-check:** symmetric; entry \((1,1)=4=n\).

3

Calculate \(\boldsymbol{X}^T\boldsymbol{y}\) How step 3

\[\text{entry }1:\ 2+3+5+6=16\]

\[\text{entry }2:\ 1(2)+2(3)+3(5)+4(6)=2+6+15+24=47\]

\[\boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}16\\47\end{array}\right]\]

**Self-check:** entry 1 is \(\sum_iy_i\).

4

Make sure that the inverse exists How step 4

\[\det(\boldsymbol{X}^T\boldsymbol{X})=4(30)-10(10)=120-100=20\neq0\]

The columns of \(\boldsymbol{X}\) are linearly independent.

5

Find \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\) How step 5

Rows \((a,b),(c,d)\) → \(\frac{1}{ad-bc}\) times rows \((d,-b),(-c,a)\).

\[(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\frac{1}{20}\left[\begin{array}{cc}30&-10\\-10&4\end{array}\right]\]

6

Calculate \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\) How step 5 · Slides p.20

\[\hat{\boldsymbol\beta}=\frac{1}{20}\left[\begin{array}{c}30(16)-10(47)\\-10(16)+4(47)\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\frac{1}{20}\left[\begin{array}{c}480-470\\-160+188\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\frac{1}{20}\left[\begin{array}{c}10\\28\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}\hat\beta_0\\\hat\beta_1\end{array}\right]=\left[\begin{array}{c}0.5\\1.4\end{array}\right]\]

**Basis:** Slides p.20. Step 4 makes the solution unique.

7

Put \(\hat{\boldsymbol\beta}\) back into the equations Self-check 3

\[\text{row }1:\ 4(0.5)+10(1.4)=2+14=16\]

\[\text{row }2:\ 10(0.5)+30(1.4)=5+42=47\]

**Self-check:** 16 and 47 equal \(\boldsymbol{X}^T\boldsymbol{y}\).

Model values \(0.5+1.4x\): 1.9, 3.3, 4.7, 6.1. Differences \(y_i-(\boldsymbol{X}\hat{\boldsymbol\beta})_i\): 0.1, −0.3, 0.3, −0.1.

\[S(\hat{\boldsymbol\beta})=0.1^2+(-0.3)^2+0.3^2+(-0.1)^2=0.01+0.09+0.09+0.01=0.2\]

Result: \(\hat{\boldsymbol\beta}=(0.5,\,1.4)^T\), \(S(\hat{\boldsymbol\beta})=0.2\). ▲

In the figure, \(\beta_0\) stays at 0.5 and only \(\beta_1\) changes. Then \(S=0.2+30(\beta_1-1.4)^2\), with \(30=\sum_ix_i^2\).

[figure]
Figure 3-1. \(x=1,2,3,4\), \(y=2,3,5,6\). Top: \(S\) is smallest at \(\hat\beta_1=1.4\). Bottom: the log-likelihood (Section 03.3) is largest at the same point.

At \(\beta_1=\hat\beta_1\pm0.1\), \(S=0.2+30(0.1)^2=0.5\).

**Why** Slides p.20

Rules from Unit 01, with \(\boldsymbol\beta\) in place of \(\boldsymbol{y}\):

Rule A: for a constant vector \(\boldsymbol{a}\), \(\dfrac{\partial(\boldsymbol{a}^T\boldsymbol{y})}{\partial \boldsymbol{y}}=\boldsymbol{a}\).

Rule B: for a constant symmetric matrix \(\boldsymbol{A}\), \(\dfrac{\partial(\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y})}{\partial \boldsymbol{y}}=2\boldsymbol{A}\boldsymbol{y}\).

1

\[S(\boldsymbol\beta)=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

2

\[S(\boldsymbol\beta)=(\boldsymbol{y}^T-\boldsymbol\beta^T\boldsymbol{X}^T)(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\((\boldsymbol{X}\boldsymbol\beta)^T=\boldsymbol\beta^T\boldsymbol{X}^T\).

3

\[S(\boldsymbol\beta)=\boldsymbol{y}^T\boldsymbol{y}-\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta-\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

4

\[\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}=(\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y})^T=\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta\]

A \(1\times1\) value equals its transpose.

5

\[S(\boldsymbol\beta)=\boldsymbol{y}^T\boldsymbol{y}-2\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Line 3 of Slides p.20.

6

\[\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta=(\boldsymbol{X}^T\boldsymbol{y})^T\boldsymbol\beta\]

Form \(\boldsymbol{a}^T\boldsymbol\beta\), \(\boldsymbol{a}=\boldsymbol{X}^T\boldsymbol{y}\).

7

\[\frac{\partial(\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta)}{\partial\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\]

Rule A.

8

\[(\boldsymbol{X}^T\boldsymbol{X})^T=\boldsymbol{X}^T(\boldsymbol{X}^T)^T=\boldsymbol{X}^T\boldsymbol{X}\]

Symmetric: Rule B applies.

9

\[\frac{\partial(\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta)}{\partial\boldsymbol\beta}=2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Rule B with \(\boldsymbol{A}=\boldsymbol{X}^T\boldsymbol{X}\).

10

\[\frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=0-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Steps 7 and 9.

11

\[-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=0\]

The derivative is 0 at the minimum.

12

\[2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=2\boldsymbol{X}^T\boldsymbol{y}\]

13

\[(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\]

14

\[\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

Multiply on the left by \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\). ∎

Check of step 11 on the example: \(-2(16,47)^T+2(16,47)^T=(0,0)^T\).

### 03.3 Maximum likelihood Slides p.21–22

Slide p.21 builds up p.22.

**What** Slides p.22

Maximum Likelihood · Lecture 6 · p.22 Under our assumptions, we have: \[\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\] \[\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})=\frac{1}{(2\pi)^{\frac n2}|\sigma^2\boldsymbol{I}|^{\frac12}}\exp\left\{-\frac12(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\sigma^2\boldsymbol{I})^{-1}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\right\}\] \[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\] Look familiar?
So \(\hat{\boldsymbol\beta}_{MLE}=\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\)
• I.e. MLE=OLS (when all our assumptions hold)

Likelihood \(\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})\): the MVN density of the observed data, as a function of \(\boldsymbol\beta\) and \(\sigma^2\).

Log-likelihood \(\ell\): \(\log\mathcal{L}\) (natural log). It has its maximum where \(\mathcal{L}\) has it.

Maximum likelihood estimator (MLE) \(\hat{\boldsymbol\beta}_{MLE}\): the \(\boldsymbol\beta\) with the largest \(\mathcal{L}\).

"Look familiar?": only the last term of \(\ell\) contains \(\boldsymbol\beta\), and it is \(-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\). The largest \(\ell\) occurs at the smallest \(S\). The assumptions are linearity, independence, normality, and equal variance.

**How** Added

Get \(\hat{\boldsymbol\beta}_{MLE}\) from the model

- Write \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\).
- Write \(\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})\). Take the log to get \(\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})\).
- Find the term of \(\ell\) with \(\boldsymbol\beta\): \(-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\).
- The factor \(-\frac{1}{2\sigma^2}\) is negative. Maximize \(\ell\) by minimizing \(S(\boldsymbol\beta)\).
- Use Section 03.2: \(\hat{\boldsymbol\beta}_{MLE}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).

**Self-check:** (1) A change of \(\sigma^2\) does not move the maximizing \(\boldsymbol\beta\). (2) \(\ell(\hat{\boldsymbol\beta},\sigma^2|\mathbf{Y})\) is larger than \(\ell\) at any other \(\boldsymbol\beta\).

**Example** Added

Data of Section 03.2 (\(n=4\)). \(\sigma^2\) is unknown. Fix \(\sigma^2=1\), then \(\sigma^2=0.5\). Unit 05 estimates \(\sigma^2\).

1

Write the three terms of \(\ell\) How step 2

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac{4}{2}\log(2\pi)-\frac{4}{2}\log\sigma^2-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\]

\(\frac n2=2\).

2

Calculate at \(\boldsymbol\beta=\hat{\boldsymbol\beta}\) and \(\sigma^2=1\) Slides p.22

\[-2\log(2\pi)=-3.675754\]

\[-2\log 1=0\]

\[-\tfrac{1}{2(1)}S(\hat{\boldsymbol\beta})=-\tfrac{0.2}{2}=-0.1\]

\[\ell(\hat{\boldsymbol\beta},1|\mathbf{Y})=-3.675754+0-0.1=-3.775754\]

3

Move \(\beta_1\) by \(\pm0.1\) and keep \(\beta_0=0.5\) Self-check 2

\[S=0.2+30(0.1)^2=0.5\]

\[\ell=-3.675754+0-\tfrac{0.5}{2}=-3.925754\]

\[-3.925754<-3.775754\]

Both sides; bottom curve of Figure 3-1.

4

Calculate again with \(\sigma^2=0.5\) Self-check 1

\[-2\log 0.5=1.386294\]

\[\ell(\hat{\boldsymbol\beta},0.5|\mathbf{Y})=-3.675754+1.386294-\tfrac{0.2}{1}=-2.489460\]

\[\beta_1=\hat\beta_1\pm0.1:\ \ell=-3.675754+1.386294-\tfrac{0.5}{1}=-2.789460\]

**Self-check:** \(\ell\) changes, but its maximum stays at \(\hat{\boldsymbol\beta}\). \(\sigma^2\) changes only the positive factor \(\frac{1}{2\sigma^2}\).

Result: \(\hat{\boldsymbol\beta}_{MLE}=(0.5,\,1.4)^T\), the OLS estimate. ▲

**Why** Slides p.22

1

\[\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})=(2\pi)^{-\frac n2}|\sigma^2\boldsymbol{I}|^{-\frac12}\exp\left\{-\frac12(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\sigma^2\boldsymbol{I})^{-1}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\right\}\]

MVN density.

2

\[|\sigma^2\boldsymbol{I}|=(\sigma^2)^n\]

Diagonal matrix: product of diagonal entries.

3

\[(\sigma^2\boldsymbol{I})^{-1}=\frac{1}{\sigma^2}\boldsymbol{I}\]

4

\[\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})=(2\pi)^{-\frac n2}(\sigma^2)^{-\frac n2}\exp\left\{-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\right\}\]

Put in steps 2 and 3.

5

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=\log\left[(2\pi)^{-\frac n2}\right]+\log\left[(\sigma^2)^{-\frac n2}\right]-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\(\log(abc)=\log a+\log b+\log c\) and \(\log e^{u}=u\).

6

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\(\log a^{k}=k\log a\). Slides p.22.

7

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\]

Definition of \(S(\boldsymbol\beta)\): "Look familiar?"

8

\[\frac{\partial\ell}{\partial\boldsymbol\beta}=-\frac{1}{2\sigma^2}\frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=-\frac{1}{2\sigma^2}\left(-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\right)\]

Section 03.2, Why step 10.

9

\[-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=0\]

Set to 0; multiply by \(-2\sigma^2\).

10

\[\hat{\boldsymbol\beta}_{MLE}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}=\hat{\boldsymbol\beta}\]

Section 03.2, Why steps 11–14. ∎

### 03.4 Practice Added

**Q1.** Use the data \(x=2,4,6\) and \(y=1,3,8\). Calculate \(\boldsymbol{X}^T\boldsymbol{X}\), \(\boldsymbol{X}^T\boldsymbol{y}\), and \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).

Answer

\(\boldsymbol{X}\) has rows \((1,2)\), \((1,4)\), \((1,6)\).

\(\boldsymbol{X}^T\boldsymbol{X}\): entry \((1,1)=3\); entry \((1,2)=2+4+6=12\); entry \((2,2)=4+16+36=56\).

\(\boldsymbol{X}^T\boldsymbol{y}\): entry \(1=1+3+8=12\); entry \(2=2(1)+4(3)+6(8)=2+12+48=62\).

\(\det(\boldsymbol{X}^T\boldsymbol{X})=3(56)-12(12)=168-144=24\).

\(\hat\beta_0=\frac{1}{24}\big(56(12)-12(62)\big)=\frac{1}{24}(672-744)=\frac{-72}{24}=-3\).

\(\hat\beta_1=\frac{1}{24}\big(-12(12)+3(62)\big)=\frac{1}{24}(-144+186)=\frac{42}{24}=1.75\).

Check row 1: \(3(-3)+12(1.75)=-9+21=12\). Check row 2: \(12(-3)+56(1.75)=-36+98=62\). \(\hat{\boldsymbol\beta}=(-3,\,1.75)^T\).

**Q2.** Intercept-only model \(y_i=\beta_0+\epsilon_i\): \(\boldsymbol{X}\) is an \(n\times1\) column of 1s. Use \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\) to find \(\hat\beta_0\). Calculate it for \(y=2,3,5,6\).

Answer

\(\boldsymbol{X}=(1,\dots,1)^T\).

\(\boldsymbol{X}^T\boldsymbol{X}=\sum_{i=1}^n1\cdot1=n\).

\(\boldsymbol{X}^T\boldsymbol{y}=\sum_{i=1}^ny_i\).

\(\hat\beta_0=n^{-1}\sum_{i=1}^ny_i=\bar y\), the sample mean.

Data: \(n=4\), \(\sum_iy_i=2+3+5+6=16\), \(\hat\beta_0=16/4=4\).

**Q3.** Use \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\). Why does the \(\boldsymbol\beta\) with the largest \(\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})\) not depend on \(\sigma^2\)? For \(n=4\), \(S(\hat{\boldsymbol\beta})=0.2\), calculate \(\ell(\hat{\boldsymbol\beta},2|\mathbf{Y})\).

Answer

\(\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\).

Only the third term contains \(\boldsymbol\beta\): the positive \(\frac{1}{2\sigma^2}\) times \(-S(\boldsymbol\beta)\).

For each \(\sigma^2>0\), the largest \(\ell\) occurs at the smallest \(S\), at \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).

Numbers: \(-2\log(2\pi)=-3.675754\), \(-2\log2=-1.386294\), \(-\frac{0.2}{2(2)}=-0.05\).

\(\ell(\hat{\boldsymbol\beta},2|\mathbf{Y})=-3.675754-1.386294-0.05=-5.112048\).

## 04 · Properties of the LS estimator, fitted values, and residuals

Plan · Slides p.23–32

Step 1: Show \(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\) (Slides p.23–25).

Step 2: Find the covariance matrix of \(\hat{\boldsymbol\beta}\) (Slides p.26–27).

Step 3: Write the distribution of \(\hat{\boldsymbol\beta}\) and of each \(\hat\beta_j\) (Slides p.28–29).

Step 4: Define the fitted values and the hat matrix \(\boldsymbol H\) (Slides p.30).

Step 5: Define the residuals and show \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\) (Slides p.31–32).

Facts from earlier units:

Model (Slides p.7, p.13): \(\boldsymbol y=\boldsymbol X\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol 0,\sigma^2\boldsymbol I)\). Equivalently, \(\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\).

\(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\) and \(\mathrm{Var}[\boldsymbol y]=\sigma^2\boldsymbol I\).

\(\boldsymbol X\) is a known \(n\times(p+1)\) matrix of constants.

For a constant matrix \(\boldsymbol C\) (Slides p.4): \(E[\boldsymbol C\boldsymbol y]=\boldsymbol C\,E[\boldsymbol y]\) and \(\mathrm{Var}[\boldsymbol C\boldsymbol y]=\boldsymbol C\,\mathrm{Var}[\boldsymbol y]\,\boldsymbol C^T\).

\(\mathrm{Var}[\cdot]\) and \(\mathrm{Var}(\cdot)\) have the same meaning.

Least squares estimator (Unit 03, Slides p.20): \(\hat{\boldsymbol\beta}=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\).

All examples use this data set Added: \(n=4\), \(p=1\), row \(i\) of \(\boldsymbol X\) is \((1,\ x_i)\).
\[x=(0,\ 1,\ 2,\ 3),\qquad \boldsymbol y=(1,\ 3,\ 2,\ 5)^T,\qquad \boldsymbol X=\left[\begin{array}{cc}1&0\\1&1\\1&2\\1&3\end{array}\right]\]

1

\[\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}1+1+1+1&0+1+2+3\\0+1+2+3&0+1+4+9\end{array}\right]=\left[\begin{array}{cc}4&6\\6&14\end{array}\right]\]

Entry \((j,k)\): column \(j\) dot column \(k\).

2

\[\det(\boldsymbol X^T\boldsymbol X)=4\cdot14-6\cdot6=56-36=20\]

3

\[(\boldsymbol X^T\boldsymbol X)^{-1}=\frac{1}{20}\left[\begin{array}{cc}14&-6\\-6&4\end{array}\right]=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\]

\(2\times2\) inverse: swap diagonal, negate off-diagonal, divide.

4

\[\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}1+3+2+5\\0\cdot1+1\cdot3+2\cdot2+3\cdot5\end{array}\right]=\left[\begin{array}{c}11\\22\end{array}\right]\]

5

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}0.7\cdot11-0.3\cdot22\\-0.3\cdot11+0.2\cdot22\end{array}\right]=\left[\begin{array}{c}7.7-6.6\\-3.3+4.4\end{array}\right]\]

6

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}1.1\\1.1\end{array}\right]\]

### 04.1 The mean of \(\hat{\boldsymbol\beta}\): unbiasedness Slides p.23–25

**What** Slides p.25

Properties of \(\hat{\boldsymbol\beta}\) · Lecture 6 · p.25 \[E[\hat{\boldsymbol\beta}]=E[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]=\boldsymbol\beta\]

\(\hat{\boldsymbol\beta}\) comes from the random \(\boldsymbol y\). Each new data set gives a different \(\hat{\boldsymbol\beta}\). Its mean over all data sets is exactly \(\boldsymbol\beta\). Unbiased: the mean of the estimator equals the parameter.

**How** Added

Test if an estimator \(\boldsymbol C\boldsymbol y\) is unbiased

- Write the estimator as \(\boldsymbol C\boldsymbol y\), with \(\boldsymbol C\) built only from \(\boldsymbol X\) and numbers.
- For \(\hat{\boldsymbol\beta}\), \(\boldsymbol C=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\).
- Move \(E\) inside: \(E[\boldsymbol C\boldsymbol y]=\boldsymbol C\,E[\boldsymbol y]\).
- Put in \(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\) to get \(\boldsymbol C\boldsymbol X\boldsymbol\beta\).
- Calculate \(\boldsymbol C\boldsymbol X\). If \(\boldsymbol C\boldsymbol X=\boldsymbol I\), then \(E[\boldsymbol C\boldsymbol y]=\boldsymbol\beta\).

**Self-check:** \(\boldsymbol C\) is \((p+1)\times n\). \(\boldsymbol C\boldsymbol X\) is \((p+1)\times(p+1)\), with 1 on the diagonal and 0 elsewhere.

**Example** Added

Steps 2 and 5 for the small data set:

1

\[\boldsymbol C=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\left[\begin{array}{cccc}1&1&1&1\\0&1&2&3\end{array}\right]\]

2

\[\boldsymbol C=\left[\begin{array}{cccc}0.7&0.7-0.3&0.7-0.6&0.7-0.9\\-0.3&-0.3+0.2&-0.3+0.4&-0.3+0.6\end{array}\right]\]

3

\[\boldsymbol C=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\-0.3&-0.1&0.1&0.3\end{array}\right]\]

4

\[\boldsymbol C\boldsymbol X=\left[\begin{array}{cc}0.7+0.4+0.1-0.2&0+0.4+0.2-0.6\\-0.3-0.1+0.1+0.3&0-0.1+0.2+0.9\end{array}\right]\]

5

\[\boldsymbol C\boldsymbol X=\left[\begin{array}{cc}1&0\\0&1\end{array}\right]=\boldsymbol I\]

\(E[\hat{\boldsymbol\beta}]=\boldsymbol I\boldsymbol\beta=\boldsymbol\beta\). Check: row 1 of \(\boldsymbol C\boldsymbol y\) is \(0.7\cdot1+0.4\cdot3+0.1\cdot2-0.2\cdot5=0.7+1.2+0.2-1.0=1.1\). ▲

**Why** Slides p.25

1

\[E[\hat{\boldsymbol\beta}]=E[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]\]

2

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T E[\boldsymbol y]\]

\(E[\boldsymbol C\boldsymbol y]=\boldsymbol C\,E[\boldsymbol y]\).

3

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol X\boldsymbol\beta)\]

Model: \(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\).

4

\[=(\boldsymbol X^T\boldsymbol X)^{-1}(\boldsymbol X^T\boldsymbol X)\boldsymbol\beta\]

5

\[=\boldsymbol\beta\]

∎

### 04.2 The covariance matrix of \(\hat{\boldsymbol\beta}\) Slides p.26–27

**What** Slides p.27

Properties of \(\hat{\boldsymbol\beta}\) · Lecture 6 · p.27 \[\mathrm{Var}[\hat{\boldsymbol\beta}]=\mathrm{Var}[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\]

\(\mathrm{Var}[\hat{\boldsymbol\beta}]\) is \((p+1)\times(p+1)\). Its diagonal entry for \(\beta_j\) is \(\mathrm{Var}[\hat\beta_j]\). \(\sigma^2\) is unknown. \((\boldsymbol X^T\boldsymbol X)^{-1}\) is known.

The slides write \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\) (p.29). Rows and columns follow the order \(\beta_0,\beta_1,\dots,\beta_p\). \(V_{jj}\) is the diagonal entry for \(\beta_j\).

**How** Added

Find \(\mathrm{Var}[\hat\beta_j]\)

- Make \(\boldsymbol X\), with 1 in each entry of column 1.
- Calculate \(\boldsymbol X^T\boldsymbol X\).
- Find \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\).
- Take \(V_{jj}\). The first diagonal entry is for \(\beta_0\).
- Write \(\mathrm{Var}[\hat\beta_j]=\sigma^2V_{jj}\).

**Self-check:** \(\boldsymbol V\) is \((p+1)\times(p+1)\) and symmetric. Each \(V_{jj}\) is larger than 0.

**Example** Added

Step 1–3: \(\boldsymbol V=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\), \(2\times2\) because \(p+1=2\).

Step 4: \(V_{00}=0.7\) (for \(\beta_0\)) and \(V_{11}=0.2\) (for \(\beta_1\)).

Step 5: \(\mathrm{Var}[\hat\beta_0]=0.7\,\sigma^2\) (intercept).

Step 5: \(\mathrm{Var}[\hat\beta_1]=0.2\,\sigma^2\) (slope).

\(\sigma^2\) is still unknown. Unit 05 estimates it and calculates the standard error. ▲

**Why** Slides p.27

The slides do lines 3 and 4 in one step Added.

1

\[\mathrm{Var}[\hat{\boldsymbol\beta}]=\mathrm{Var}[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]\]

2

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\,\mathrm{Var}[\boldsymbol y]\,((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T)^T\]

\(\mathrm{Var}[\boldsymbol C\boldsymbol y]=\boldsymbol C\,\mathrm{Var}[\boldsymbol y]\,\boldsymbol C^T\) (Slides p.4).

3

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\,\mathrm{Var}[\boldsymbol y]\,(\boldsymbol X^T)^T((\boldsymbol X^T\boldsymbol X)^{-1})^T\]

Added: \((\boldsymbol A\boldsymbol B)^T=\boldsymbol B^T\boldsymbol A^T\).

4

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\,\mathrm{Var}[\boldsymbol y]\,\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

Symmetric \(\boldsymbol X^T\boldsymbol X\) has a symmetric inverse.

5

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\sigma^2\boldsymbol I)\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

Model: \(\mathrm{Var}[\boldsymbol y]=\sigma^2\boldsymbol I\).

6

\[=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

7

\[=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\]

∎

### 04.3 The distribution of \(\hat{\boldsymbol\beta}\) Slides p.28–29

**What** Slides p.29

Properties of \(\hat{\boldsymbol\beta}\) · Lecture 6 · p.29 And \(\hat{\boldsymbol\beta}\) is MVN-distributed. Why? \[\Longrightarrow\ \hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\ \sigma^2(\boldsymbol X^T\boldsymbol X)^{-1})\] From property 2 of MVN distributions, this gives: \[\hat\beta_j\sim N(\beta_j,\ \sigma^2V_{jj})\] where \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\).

Line 1 combines the mean (04.1) and the covariance matrix (04.2). "Property 2" is the marginal property (Slides p.6). It gives one component \(\hat\beta_j\) a normal distribution.

**How** Added

Write the distribution of one \(\hat\beta_j\)

- Find \(j\): the intercept has \(j=0\), covariate \(k\) has \(j=k\).
- Find \(\boldsymbol V\) (How box of 04.2).
- Take \(V_{jj}\).
- Write \(\hat\beta_j\sim N(\beta_j,\ \sigma^2V_{jj})\). The mean is the parameter \(\beta_j\), not a number.

**Self-check:** The variance is \(\sigma^2\) times a positive number.

**Example** Added

Slope estimator \(\hat\beta_1\), small data set:

1

\[j=1\]

2

\[\boldsymbol V=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\]

3

\[V_{11}=0.2\]

4

\[\hat\beta_1\sim N(\beta_1,\ 0.2\,\sigma^2)\]

The value \(\hat\beta_1=1.1\) is one draw from this distribution. ▲

**Why** Added

Added · An answer to the "Why?" on p.29

The slides give no reason. The reason is the linearity property (Slides p.6):

Lecture 6 · p.6 Linearity: If \(\boldsymbol u=\boldsymbol C\boldsymbol y+\boldsymbol d\), then: \[\boldsymbol u\sim MVN(\boldsymbol C\boldsymbol\mu+\boldsymbol d,\ \boldsymbol C\boldsymbol\Sigma\boldsymbol C^T)\]

with \(\boldsymbol y\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\).

1

\[\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\ \sigma^2\boldsymbol I)\]

Model (p.7).

2

\[\hat{\boldsymbol\beta}=\boldsymbol C\boldsymbol y+\boldsymbol 0,\qquad \boldsymbol C=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

\(\boldsymbol C\) is constant; \(\boldsymbol d=\boldsymbol 0\).

3

\[\hat{\boldsymbol\beta}\sim MVN(\boldsymbol C\boldsymbol X\boldsymbol\beta,\ \boldsymbol C(\sigma^2\boldsymbol I)\boldsymbol C^T)\]

Linearity.

4

\[\boldsymbol C\boldsymbol X\boldsymbol\beta=\boldsymbol\beta\]

Why of 04.1, lines 3–5.

5

\[\boldsymbol C(\sigma^2\boldsymbol I)\boldsymbol C^T=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\]

Why of 04.2, lines 2–7.

6

\[\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\ \sigma^2(\boldsymbol X^T\boldsymbol X)^{-1})\]

Line 1 of p.29. ∎

Line 2 of p.29: the marginal property (p.6) takes entry \(j\) of the mean and diagonal entry \(j\) of \(\sigma^2\boldsymbol V\).

### 04.4 Fitted values and the hat matrix \(\boldsymbol H\) Slides p.30

**What** Slides p.30

Fitted values · Lecture 6 · p.30 \[\hat{\boldsymbol y}=\boldsymbol X\hat{\boldsymbol\beta}=\boldsymbol H\boldsymbol y\] where \(\boldsymbol H=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\) is called the Hat matrix.

Fitted value: the estimated regression equation at the covariates of observation \(i\), \(\hat y_i=\hat\beta_0+\hat\beta_1x_{i1}+\cdots+\hat\beta_px_{ip}\). Hat matrix \(\boldsymbol H\): the \(n\times n\) matrix that changes \(\boldsymbol y\) into \(\hat{\boldsymbol y}\). \(\boldsymbol H\) depends only on \(\boldsymbol X\).

**How** Added

Calculate the fitted values

- Calculate \(\hat{\boldsymbol\beta}\).
- Take row \(i\) of \(\boldsymbol X\), \((1,x_{i1},\dots,x_{ip})\).
- Multiply it with \(\hat{\boldsymbol\beta}\) term by term and add, to get \(\hat y_i\).
- Do Steps 2 and 3 for each \(i=1,\dots,n\).
- Alternative: calculate \(\boldsymbol H=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\), then \(\boldsymbol H\boldsymbol y\).

**Self-check:** \(\boldsymbol H\) is \(n\times n\) and symmetric. Steps 4 and 5 give the same \(\hat{\boldsymbol y}\).

**Example** Added

Step 1: \(\hat\beta_0=1.1\), \(\hat\beta_1=1.1\).

Steps 2–3, \(i=1\): \(\hat y_1=1.1\cdot1+1.1\cdot0=1.1+0=1.1\).

Steps 2–3, \(i=2\): \(\hat y_2=1.1\cdot1+1.1\cdot1=1.1+1.1=2.2\).

Steps 2–3, \(i=3\): \(\hat y_3=1.1\cdot1+1.1\cdot2=1.1+2.2=3.3\).

Steps 2–3, \(i=4\): \(\hat y_4=1.1\cdot1+1.1\cdot3=1.1+3.3=4.4\).

Step 5: \(\boldsymbol H=\boldsymbol X\boldsymbol C\), with \(\boldsymbol C\) from 04.1. Row \(i\) of \(\boldsymbol H\) is \((\text{row 1 of }\boldsymbol C)+x_i\cdot(\text{row 2 of }\boldsymbol C)\).

1

\[\boldsymbol H=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\0.7-0.3&0.4-0.1&0.1+0.1&-0.2+0.3\\0.7-0.6&0.4-0.2&0.1+0.2&-0.2+0.6\\0.7-0.9&0.4-0.3&0.1+0.3&-0.2+0.9\end{array}\right]\]

\(x_i=0,1,2,3\).

2

\[\boldsymbol H=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\0.4&0.3&0.2&0.1\\0.1&0.2&0.3&0.4\\-0.2&0.1&0.4&0.7\end{array}\right]\]

3

\[\boldsymbol H\boldsymbol y=\left[\begin{array}{c}0.7\cdot1+0.4\cdot3+0.1\cdot2-0.2\cdot5\\0.4\cdot1+0.3\cdot3+0.2\cdot2+0.1\cdot5\\0.1\cdot1+0.2\cdot3+0.3\cdot2+0.4\cdot5\\-0.2\cdot1+0.1\cdot3+0.4\cdot2+0.7\cdot5\end{array}\right]\]

4

\[=\left[\begin{array}{c}0.7+1.2+0.2-1.0\\0.4+0.9+0.4+0.5\\0.1+0.6+0.6+2.0\\-0.2+0.3+0.8+3.5\end{array}\right]\]

5

\[=\left[\begin{array}{c}1.1\\2.2\\3.3\\4.4\end{array}\right]\]

Same as Steps 2–4. ▲

**Why** Slides p.30

1

\[\hat{\boldsymbol y}=\boldsymbol X\hat{\boldsymbol\beta}\]

2

\[=\boldsymbol X\left[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\right]\]

3

\[=\left[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\right]\boldsymbol y\]

Matrix multiplication is associative.

4

\[=\boldsymbol H\boldsymbol y\]

∎

### 04.5 Residuals and \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\) Slides p.31–32

**What** Slides p.32

Residuals · Lecture 6 · p.32 \[\boldsymbol e=\boldsymbol y-\hat{\boldsymbol y}=(\boldsymbol I-\boldsymbol H)\boldsymbol y.\] Note: \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). Why?

Residual: observed value minus fitted value, \(e_i=y_i-\hat y_i\). \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\) gives one equation for each column of \(\boldsymbol X\):

Column of ones: \(\sum_{i=1}^n e_i=0\).

Column of covariate \(k\): \(\sum_{i=1}^n x_{ik}e_i=0\).

Orthogonal: two vectors with dot product 0 (sum of products of matching entries). \(\boldsymbol e\) is orthogonal to each column of \(\boldsymbol X\).

**How** Added

Calculate the residuals and check \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\)

- Calculate \(\hat{\boldsymbol y}\) (How box of 04.4).
- Calculate \(e_i=y_i-\hat y_i\) for each \(i\).
- Calculate \(\sum e_i\).
- For each covariate \(k\), calculate \(\sum x_{ik}e_i\).

**Self-check:** With exact arithmetic, each of the \(p+1\) entries is exactly 0.

**Example** Added

Step 1: \(\hat{\boldsymbol y}=(1.1,\ 2.2,\ 3.3,\ 4.4)^T\).

Step 2: \(e_1=1-1.1=-0.1\).

Step 2: \(e_2=3-2.2=0.8\).

Step 2: \(e_3=2-3.3=-1.3\).

Step 2: \(e_4=5-4.4=0.6\).

Step 3: \(\sum e_i=-0.1+0.8-1.3+0.6=0\).

Step 4: \(\sum x_ie_i=0\cdot(-0.1)+1\cdot0.8+2\cdot(-1.3)+3\cdot0.6=0+0.8-2.6+1.8=0\). ▲

[figure]
Figure 4-1. Small data set (\(n=4\)): points \((x_i,y_i)\) in green, fitted line \(\hat y=1.1+1.1x\) in blue, residuals in orange. Segment length is \(|e_i|\); points above the line have \(e_i>0\). The residuals add to 0.

**Why** Slides p.31–32

Part 1 is Slides p.31. Part 2 answers the "Why?" on p.32. The slides do lines 2 and 3 of Part 2 in one step Added.

1

\[\boldsymbol e=\boldsymbol y-\hat{\boldsymbol y}\]

2

\[=\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta}\]

3

\[=\boldsymbol y-\boldsymbol H\boldsymbol y\]

04.4.

4

\[=(\boldsymbol I-\boldsymbol H)\boldsymbol y\]

∎

1

\[\boldsymbol X^T\boldsymbol e=\boldsymbol X^T(\boldsymbol y-\boldsymbol H\boldsymbol y)\]

Line 3 of Part 1.

2

\[=\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\boldsymbol H\boldsymbol y\]

Added.

3

\[=\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\]

4

\[=\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\boldsymbol y\]

5

\[=\boldsymbol 0\]

∎

Summary of this unit

\(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\). \(\mathrm{Var}[\hat{\boldsymbol\beta}]=\sigma^2\boldsymbol V\). \(\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\). \(\hat{\boldsymbol y}=\boldsymbol H\boldsymbol y\). \(\boldsymbol e=(\boldsymbol I-\boldsymbol H)\boldsymbol y\), \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). Unit 05 estimates \(\sigma^2\).

### 04.6 Practice Added

**Q1.** Data: \(n=3\), \(x=(1,\ 2,\ 3)\), model \(y_i=\beta_0+\beta_1x_i+\epsilon_i\). Find \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\). Write the distribution of \(\hat\beta_1\) in terms of \(\sigma^2\).

Answer

\(\boldsymbol X\) has rows \((1,1)\), \((1,2)\), \((1,3)\).

\(\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}1+1+1&1+2+3\\1+2+3&1+4+9\end{array}\right]=\left[\begin{array}{cc}3&6\\6&14\end{array}\right]\).

\(\det(\boldsymbol X^T\boldsymbol X)=3\cdot14-6\cdot6=42-36=6\).

\(\boldsymbol V=\frac16\left[\begin{array}{cc}14&-6\\-6&3\end{array}\right]=\left[\begin{array}{cc}7/3&-1\\-1&0.5\end{array}\right]\).

\(V_{11}=0.5\). By p.29, \(\hat\beta_1\sim N(\beta_1,\ 0.5\,\sigma^2)\).

**Q2.** Use the data of Q1 with \(\boldsymbol y=(2,\ 2,\ 5)^T\). Calculate \(\hat{\boldsymbol\beta}\), the fitted values, and the residuals. Check \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\).

Answer

\(\boldsymbol X^T\boldsymbol y=(2+2+5,\ 1\cdot2+2\cdot2+3\cdot5)^T=(9,\ 21)^T\).

\(\hat\beta_0=\frac16(14\cdot9-6\cdot21)=\frac16(126-126)=0\).

\(\hat\beta_1=\frac16(-6\cdot9+3\cdot21)=\frac16(-54+63)=\frac96=1.5\).

\(\hat y_1=0+1.5\cdot1=1.5\). \(\hat y_2=0+1.5\cdot2=3\). \(\hat y_3=0+1.5\cdot3=4.5\).

\(e_1=2-1.5=0.5\). \(e_2=2-3=-1\). \(e_3=5-4.5=0.5\).

\(\sum e_i=0.5-1+0.5=0\).

\(\sum x_ie_i=1\cdot0.5+2\cdot(-1)+3\cdot0.5=0.5-2+1.5=0\).

**Q3.** Show \(\boldsymbol X^T\hat{\boldsymbol y}=\boldsymbol X^T\boldsymbol y\). Check row 1 with the small data set.

Answer

1

\[\boldsymbol X^T\boldsymbol e=\boldsymbol 0\]

p.32.

2

\[\boldsymbol X^T(\boldsymbol y-\hat{\boldsymbol y})=\boldsymbol 0\]

3

\[\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\hat{\boldsymbol y}=\boldsymbol 0\]

4

\[\boldsymbol X^T\hat{\boldsymbol y}=\boldsymbol X^T\boldsymbol y\]

∎

Row 1 (column of ones): \(\sum\hat y_i=\sum y_i\). \(\sum\hat y_i=1.1+2.2+3.3+4.4=11\). \(\sum y_i=1+3+2+5=11\).

## 05 · Estimating σ², calculating the estimates, and practice

Plan · Slides p.33–41

Step 1: Find the MLE \(\hat\sigma^2_{MLE}=\frac1n\boldsymbol{e}^T\boldsymbol{e}\) (Slides p.33–35).

Step 2: Change the divisor to \(n-(p+1)\) to get the unbiased \(\hat\sigma^2\) (Slides p.36).

Step 3: Calculate \(\hat{\boldsymbol\beta}\) from a system of linear equations (Slides p.37).

Step 4: Calculate the fitted values, residuals, \(\hat\sigma^2\), and standard errors (Slides p.38).

Step 5: Do the two practice questions for simple linear regression (Slides p.39–41).

Model (Slides p.13): \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\). \(\boldsymbol{X}\) is \(n\times(p+1)\). \(p\) is the number of covariates.

The error variance \(\sigma^2\) is the variance of each \(\epsilon_i\). It is the last unknown parameter.

Least squares estimate (Unit 03): \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\). It is also the MLE.

Residuals (Unit 04): \(\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}\). The residual sum of squares is \(\boldsymbol{e}^T\boldsymbol{e}=\sum_{i=1}^n e_i^2\). Slides p.32: \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\).

**The small data set of this unit** Added

The slides do not print the fit for the satisfaction data (\(n=46\), \(p=3\)). This unit uses a small data set that you can do by hand.

\(n=5\), \(p=1\). Covariate: \(x=(1,\,2,\,3,\,4,\,5)\). Outcome: \(y=(2,\,4,\,5,\,4,\,5)\).
\[\boldsymbol{X}=\left[\begin{array}{cc}1&1\\1&2\\1&3\\1&4\\1&5\end{array}\right],\qquad \boldsymbol{y}=\left[\begin{array}{c}2\\4\\5\\4\\5\end{array}\right]\]
Section 5.3 calculates \(\hat{\boldsymbol\beta}=(2.2,\,0.6)^T\) from these data.

### 05.1 MLE of \(\sigma^2\) Slides p.33–35

**What** Slides p.35

MLE of σ² · Lecture 6 · p.35 \[\ell(\boldsymbol\beta,\sigma^2\mid\boldsymbol{y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\] \[\frac{\partial\ell(\boldsymbol\beta,\sigma^2\mid\boldsymbol{y})}{\partial\sigma^2}=-\frac{n}{2\sigma^2}+\frac{1}{2(\sigma^2)^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)=0\] \[(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)=n\sigma^2\] \[\Longrightarrow\ \hat\sigma^2_{MLE}=\frac1n(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})^T(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})\]

\(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{e}\), and \(\hat\sigma^2_{MLE}=\frac1n\boldsymbol{e}^T\boldsymbol{e}\): the mean of the squared residuals.

\(\partial\ell/\partial\sigma^2\) holds \(\boldsymbol\beta\) constant. Slides p.34 show the same page without the derivative.

**How** Added

Calculate \(\hat\sigma^2_{MLE}\)

- Calculate the fitted values \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\).
- Calculate the residuals: \(e_i=y_i-\hat y_i\).
- Add the squared residuals to get \(\boldsymbol{e}^T\boldsymbol{e}\).
- Divide by \(n\).

**Self-check:** \(\boldsymbol{e}^T\boldsymbol{e}\ge0\) and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) (Slides p.32). If not, \(\hat{\boldsymbol\beta}\) is wrong.

**Example** Added

Use \(\hat y=2.2+0.6x\) from Section 05.3.

1

Fitted values How step 1

\[\hat y_1=2.2+0.6(1)=2.8,\quad \hat y_2=2.2+0.6(2)=3.4,\quad \hat y_3=2.2+0.6(3)=4.0\]

\[\hat y_4=2.2+0.6(4)=4.6,\quad \hat y_5=2.2+0.6(5)=5.2\]

2

Residuals How step 2

\[e_1=2-2.8=-0.8,\quad e_2=4-3.4=0.6,\quad e_3=5-4.0=1.0\]

\[e_4=4-4.6=-0.6,\quad e_5=5-5.2=-0.2\]

3

Residual sum of squares How step 3

\[\boldsymbol{e}^T\boldsymbol{e}=(-0.8)^2+0.6^2+1.0^2+(-0.6)^2+(-0.2)^2\]

\[\boldsymbol{e}^T\boldsymbol{e}=0.64+0.36+1.00+0.36+0.04=2.4\]

4

Divide by \(n\) How step 4

\[\hat\sigma^2_{MLE}=\frac{2.4}{5}=0.48\]

Check: \(\sum e_i=-0.8+0.6+1.0-0.6-0.2=0\), \(\sum x_ie_i=-0.8+1.2+3.0-2.4-1.0=0\).

[figure]
Figure 5-1. Data and fitted line \(\hat y=2.2+0.6x\) (green). Each vertical segment is a residual \(e_i\). The squared lengths add to \(\boldsymbol{e}^T\boldsymbol{e}=2.4\). Purple circle: \((\bar x,\bar y)\).

**Why** Slides p.35

Let \(Q=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\). \(Q\) does not contain \(\sigma^2\).

1

\[\ell=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}Q\]

Line 1 of p.35.

2

\[\ell=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{Q}{2}(\sigma^2)^{-1}\]

3

\[\frac{\partial\ell}{\partial\sigma^2}=0-\frac n2\cdot\frac{1}{\sigma^2}-\frac Q2\cdot\left(-(\sigma^2)^{-2}\right)\]

With \(u=\sigma^2\): \(\frac{d}{du}\log u=\frac1u\), \(\frac{d}{du}u^{-1}=-u^{-2}\).

4

\[\frac{\partial\ell}{\partial\sigma^2}=-\frac{n}{2\sigma^2}+\frac{Q}{2(\sigma^2)^2}\]

Line 2 of p.35.

5

\[-\frac{n}{2\sigma^2}+\frac{Q}{2(\sigma^2)^2}=0\]

6

\[-n\sigma^2+Q=0\]

Multiply by \(2(\sigma^2)^2\).

7

\[Q=n\sigma^2\]

Line 3 of p.35.

8

\[\sigma^2=\frac1n Q=\frac1n(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

9

\[\hat\sigma^2_{MLE}=\frac1n(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})^T(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})=\frac1n\boldsymbol{e}^T\boldsymbol{e}\]

\(\hat{\boldsymbol\beta}\) minimizes \(Q\) for every \(\sigma^2\). ∎

### 05.2 Unbiased estimate of \(\sigma^2\) Slides p.36

**What** Slides p.36

The bias of an estimator \(\hat\theta\) is \(E[\hat\theta]-\theta\).

Unbiased estimate of σ² · Lecture 6 · p.36 The MLE of \(\sigma^2\) is \(\frac1n\boldsymbol{e}^T\boldsymbol{e}\), where \(\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}\).
Instead we usually use \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol{e}^T\boldsymbol{e}\). Why?

Only the divisor changes. \(n-(p+1)=n-p-1\) is the residual degrees of freedom: the number of residuals that can change freely. On this page, \(\hat\sigma^2\) is this unbiased estimate.

**How** Added

Calculate \(\hat\sigma^2\)

- Calculate \(\boldsymbol{e}^T\boldsymbol{e}\) (Section 05.1, steps 1–3).
- Count the coefficients: \(p+1\), with the intercept.
- Calculate the degrees of freedom: \(n-p-1\).
- Divide: \(\hat\sigma^2=\boldsymbol{e}^T\boldsymbol{e}/(n-p-1)\).

**Self-check:** \(\hat\sigma^2/\hat\sigma^2_{MLE}=n/(n-p-1)>1\).

**Example** Added

1

Residual sum of squares How step 1

\[\boldsymbol{e}^T\boldsymbol{e}=2.4\]

Section 05.1, step 3.

2

Count the coefficients How step 2

\[p=1,\qquad p+1=2\]

3

Degrees of freedom How step 3

\[n-p-1=5-1-1=3\]

4

Divide How step 4

\[\hat\sigma^2=\frac{2.4}{3}=0.8\]

Check: \(0.8/0.48=5/3=n/(n-p-1)\). \(\hat\sigma=\sqrt{0.8}=0.8944\).

**Why** Added

Slides p.36 give no proof.

1

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\]

p.32. \(\boldsymbol{X}^T\) has \(p+1\) rows.

2

\[\sum_{i=1}^n x_{ij}e_i=0,\qquad j=0,1,\dots,p\]

\(p+1\) linear constraints on \(e_1,\dots,e_n\).

3

\[\text{number of free residuals}=n-(p+1)\]

The constraints fix the other \(p+1\) residuals.

4

\[\hat\sigma^2=\frac{\boldsymbol{e}^T\boldsymbol{e}}{n-(p+1)}\]

Gives \(E[\hat\sigma^2]=\sigma^2\) (proof below). ∎

In the example, \(\sum e_i=0\) and \(\sum x_ie_i=0\) fix \(e_4\) and \(e_5\) from \(e_1,e_2,e_3\). Only \(5-2=3\) residuals are free.

Optional — not on the slides · Proof that \(E[\boldsymbol{e}^T\boldsymbol{e}]=(n-p-1)\sigma^2\)

The trace \(\mathrm{tr}(\boldsymbol{A})\) is the sum of the diagonal entries of a square matrix. \(\mathrm{tr}(\boldsymbol{AB})=\mathrm{tr}(\boldsymbol{BA})\). Use \(\boldsymbol{H}=\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\) (p.30) and \(\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}\) (p.31). \(\boldsymbol{I}-\boldsymbol{H}\) is symmetric and idempotent: it times itself gives itself.

1

\[\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}=(\boldsymbol{I}-\boldsymbol{H})(\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon)\]

2

\[\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\]

\(\boldsymbol{H}\boldsymbol{X}=\boldsymbol{X}\) gives \((\boldsymbol{I}-\boldsymbol{H})\boldsymbol{X}=\boldsymbol{0}\).

3

\[\boldsymbol{e}^T\boldsymbol{e}=\boldsymbol\epsilon^T(\boldsymbol{I}-\boldsymbol{H})^T(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon=\boldsymbol\epsilon^T(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\]

Symmetric and idempotent.

4

\[\boldsymbol{e}^T\boldsymbol{e}=\mathrm{tr}\!\left(\boldsymbol\epsilon^T(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\right)=\mathrm{tr}\!\left((\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\boldsymbol\epsilon^T\right)\]

A number is its own trace. Then \(\mathrm{tr}(\boldsymbol{AB})=\mathrm{tr}(\boldsymbol{BA})\).

5

\[E[\boldsymbol{e}^T\boldsymbol{e}]=\mathrm{tr}\!\left((\boldsymbol{I}-\boldsymbol{H})E[\boldsymbol\epsilon\boldsymbol\epsilon^T]\right)=\mathrm{tr}\!\left((\boldsymbol{I}-\boldsymbol{H})\sigma^2\boldsymbol{I}\right)=\sigma^2\,\mathrm{tr}(\boldsymbol{I}-\boldsymbol{H})\]

\(E[\boldsymbol\epsilon\boldsymbol\epsilon^T]=\mathrm{Var}[\boldsymbol\epsilon]=\sigma^2\boldsymbol{I}\).

6

\[\mathrm{tr}(\boldsymbol{H})=\mathrm{tr}\!\left(\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\right)=\mathrm{tr}\!\left((\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{X}\right)=\mathrm{tr}(\boldsymbol{I}_{p+1})=p+1\]

7

\[E[\boldsymbol{e}^T\boldsymbol{e}]=\sigma^2(n-(p+1))\quad\Longrightarrow\quad E\!\left[\tfrac{1}{n-(p+1)}\boldsymbol{e}^T\boldsymbol{e}\right]=\sigma^2\]

\(E[\hat\sigma^2_{MLE}]=\frac{n-p-1}{n}\sigma^2\): too small. ∎

### 05.3 Calculating \(\hat{\boldsymbol\beta}\) as a system of equations Slides p.37

**What** Slides p.37

The code on Slides p.37 does these steps:

It makes \(\boldsymbol{X}\): a column of ones, then one column for each covariate.

It calculates \(\boldsymbol{X}^T\boldsymbol{X}\) and \(\boldsymbol{X}^T\boldsymbol{y}\).

It solves the system \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\) for \(\hat{\boldsymbol\beta}\).

The slide marks as worse: calculate \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\), then multiply by \(\boldsymbol{X}^T\boldsymbol{y}\). Both give the same \(\hat{\boldsymbol\beta}\).

In a system of linear equations, each unknown has only the first power.

**How** Added

Calculate \(\hat{\boldsymbol\beta}\) from the system \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\)

- Make \(\boldsymbol{X}\) and \(\boldsymbol{y}\).
- Calculate \(\boldsymbol{X}^T\boldsymbol{X}\). Entry \((j,k)\) is \(\sum_i\) (column \(j\) entry)(column \(k\) entry).
- Calculate \(\boldsymbol{X}^T\boldsymbol{y}\). Entry \(j\) is \(\sum_i\) (column \(j\) entry)\(y_i\).
- Write the \(p+1\) equations.
- Solve by elimination.

**Self-check:** \(\boldsymbol{X}^T\boldsymbol{X}\) is symmetric, \((p+1)\times(p+1)\), with top-left entry \(n\). \(\hat{\boldsymbol\beta}\) satisfies each equation.

**Example** Added

1

Sums for \(\boldsymbol{X}^T\boldsymbol{X}\) How step 2

\[n=5,\quad \sum x_i=1+2+3+4+5=15,\quad \sum x_i^2=1+4+9+16+25=55\]

\[\boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}5&15\\15&55\end{array}\right]\]

2

Sums for \(\boldsymbol{X}^T\boldsymbol{y}\) How step 3

\[\sum y_i=2+4+5+4+5=20,\quad \sum x_iy_i=2+8+15+16+25=66\]

\[\boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}20\\66\end{array}\right]\]

3

The system How step 4

\[5\hat\beta_0+15\hat\beta_1=20\]

\[15\hat\beta_0+55\hat\beta_1=66\]

4

Elimination How step 5

\[15\hat\beta_0+45\hat\beta_1=60\]

First equation times 3.

5

\[(55-45)\hat\beta_1=66-60\]

Second equation minus step 4.

6

\[10\hat\beta_1=6\quad\Longrightarrow\quad \hat\beta_1=0.6\]

7

\[5\hat\beta_0+15(0.6)=20\quad\Longrightarrow\quad 5\hat\beta_0=20-9=11\quad\Longrightarrow\quad\hat\beta_0=2.2\]

8

Self-check

\[15(2.2)+55(0.6)=33+33=66\]

\(\hat{\boldsymbol\beta}=(2.2,\,0.6)^T\). ▲

**Why** Added

1

\[\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

Slides p.20.

2

\[(\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

3

\[(\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\]

A system \(\boldsymbol{A}\boldsymbol{b}=\boldsymbol{c}\). ∎

### 05.4 Calculating \(\hat\sigma^2\) and the standard errors Slides p.38

**What** Slides p.38

The code on Slides p.38 does these steps:

\(n\) = number of rows of \(\boldsymbol{X}\). \(p\) = number of columns minus 1.

It calculates \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\) and \(\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}\).

It calculates \(\hat\sigma^2=\sum e_i^2/(n-p-1)\).

It calculates \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\) and the standard errors \(\hat\sigma\sqrt{V_{jj}}\).

The standard error (SE) of an estimator estimates its standard deviation, with \(\hat\sigma\) in place of \(\sigma\). Slides p.27–29: \(\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1})\) and \(\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\), \(j=0,1,\dots,p\).

**How** Added

Calculate \(\mathrm{SE}(\hat\beta_j)\)

- Calculate \(\hat\sigma^2\) (Section 05.2).
- Calculate \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\). Read \(V_{jj}\).
- Multiply: \(\hat\sigma^2V_{jj}\) estimates \(\mathrm{Var}[\hat\beta_j]\).
- Take the square root: \(\mathrm{SE}(\hat\beta_j)=\hat\sigma\sqrt{V_{jj}}\).

**Self-check:** Each \(V_{jj}>0\). \(\boldsymbol{V}\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{I}\).

**Example** Added

Use \(\boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}5&15\\15&55\end{array}\right]\) (Section 5.3) and the \(2\times2\) inverse:
\[\left[\begin{array}{cc}a&b\\c&d\end{array}\right]^{-1}=\frac{1}{ad-bc}\left[\begin{array}{cc}d&-b\\-c&a\end{array}\right]\]

1

\(\hat\sigma^2\) How step 1

\[\hat\sigma^2=\frac{2.4}{5-1-1}=\frac{2.4}{3}=0.8\]

2

Determinant How step 2

\[ad-bc=5(55)-15(15)=275-225=50\]

3

\(\boldsymbol{V}\) How step 2

\[\boldsymbol{V}=\frac{1}{50}\left[\begin{array}{cc}55&-15\\-15&5\end{array}\right]=\left[\begin{array}{cc}1.1&-0.3\\-0.3&0.1\end{array}\right]\]

Check: \(1.1(5)-0.3(15)=5.5-4.5=1\).

4

Multiply How step 3

\[\hat\sigma^2V_{00}=0.8(1.1)=0.88,\qquad \hat\sigma^2V_{11}=0.8(0.1)=0.08\]

5

Square root How step 4

\[\mathrm{SE}(\hat\beta_0)=\sqrt{0.88}=0.9381,\qquad \mathrm{SE}(\hat\beta_1)=\sqrt{0.08}=0.2828\]

3 degrees of freedom. Lecture 7 uses these. ▲

**Why** Added

1

\[\mathrm{Var}[\hat{\boldsymbol\beta}]=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\sigma^2\boldsymbol{V}\]

p.27.

2

\[\mathrm{Var}[\hat\beta_j]=\sigma^2V_{jj}\]

Diagonal entry \(j\) (p.29).

3

\[\mathrm{sd}(\hat\beta_j)=\sqrt{\sigma^2V_{jj}}\]

4

\[\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\]

Put in \(\hat\sigma^2\) for unknown \(\sigma^2\). ∎

### 05.5 Practice Slides p.39–41

All questions use \(p=1\): simple linear regression (SLR). Lecture 2 notation: \(\bar x=\frac1n\sum x_i\), \(\bar y=\frac1n\sum y_i\), \(S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2\), \(S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)\).

Least squares estimates · Lecture 2 \[\hat\beta_1=\frac{S_{xy}}{S_{xx}},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.\]
\[\boldsymbol{X}=\left[\begin{array}{cc}1&x_1\\1&x_2\\\vdots&\vdots\\1&x_n\end{array}\right],\qquad \boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}n&\sum x_i\\\sum x_i&\sum x_i^2\end{array}\right],\qquad \boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}\sum y_i\\\sum x_iy_i\end{array}\right].\]
Five-point data: \(\bar x=15/5=3\), \(\bar y=20/5=4\).

\(S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=4+1+0+1+4=10\).

\(S_{xy}=(-2)(-2)+(-1)(0)+(0)(1)+(1)(0)+(2)(1)=4+0+0+0+2=6\).

**Q1** Slides p.40

Practice Q1 · Lecture 6 · p.40 In the case of \(p=1\) covariate, show that the matrix formula for \(\hat{\boldsymbol\beta}\) derived for multiple linear regression yields the familiar \(\hat\beta_0\) and \(\hat\beta_1\) from simple linear regression.

Answer Q1

Plan: invert \(\boldsymbol{X}^T\boldsymbol{X}\), multiply by \(\boldsymbol{X}^T\boldsymbol{y}\), write the sums with \(S_{xx}\) and \(S_{xy}\). First, two identities:

a

\[S_{xx}=\sum(x_i-\bar x)^2=\sum x_i^2-2\bar x\sum x_i+n\bar x^2=\sum x_i^2-2n\bar x^2+n\bar x^2=\sum x_i^2-n\bar x^2\]

\(\sum x_i=n\bar x\).

b

\[S_{xy}=\sum(x_i-\bar x)(y_i-\bar y)=\sum x_iy_i-\bar y\sum x_i-\bar x\sum y_i+n\bar x\bar y=\sum x_iy_i-n\bar x\bar y\]

Use the \(2\times2\) inverse rule of Section 05.4.

1

\[\det(\boldsymbol{X}^T\boldsymbol{X})=n\sum x_i^2-\left(\sum x_i\right)^2=n\sum x_i^2-n^2\bar x^2=n\left(\sum x_i^2-n\bar x^2\right)=nS_{xx}\]

Identity a.

2

\[(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\frac{1}{nS_{xx}}\left[\begin{array}{cc}\sum x_i^2&-\sum x_i\\-\sum x_i&n\end{array}\right]\]

3

\[\hat{\boldsymbol\beta}=\frac{1}{nS_{xx}}\left[\begin{array}{cc}\sum x_i^2&-\sum x_i\\-\sum x_i&n\end{array}\right]\left[\begin{array}{c}\sum y_i\\\sum x_iy_i\end{array}\right]\]

4

\[\hat{\boldsymbol\beta}=\frac{1}{nS_{xx}}\left[\begin{array}{c}\sum x_i^2\sum y_i-\sum x_i\sum x_iy_i\\-\sum x_i\sum y_i+n\sum x_iy_i\end{array}\right]\]

5

Row 2: \(\hat\beta_1\)

\[\hat\beta_1=\frac{n\sum x_iy_i-n\bar x\cdot n\bar y}{nS_{xx}}=\frac{\sum x_iy_i-n\bar x\bar y}{S_{xx}}=\frac{S_{xy}}{S_{xx}}\]

\(\sum x_i=n\bar x\), \(\sum y_i=n\bar y\), then identity b.

6

Row 1: \(\hat\beta_0\)

\[\hat\beta_0=\frac{n\bar y\sum x_i^2-n\bar x\sum x_iy_i}{nS_{xx}}=\frac{\bar y\sum x_i^2-\bar x\sum x_iy_i}{S_{xx}}\]

7

\[\hat\beta_0=\frac{\bar y\left(S_{xx}+n\bar x^2\right)-\bar x\left(S_{xy}+n\bar x\bar y\right)}{S_{xx}}\]

Identities a and b in reverse.

8

\[\hat\beta_0=\frac{\bar yS_{xx}+n\bar x^2\bar y-\bar xS_{xy}-n\bar x^2\bar y}{S_{xx}}=\frac{\bar yS_{xx}-\bar xS_{xy}}{S_{xx}}\]

9

\[\hat\beta_0=\bar y-\frac{S_{xy}}{S_{xx}}\bar x=\bar y-\hat\beta_1\bar x\]

The Lecture 2 formula. ∎

**Numerical check (five-point data).** \(n=5\), \(\sum x_i=15\), \(\sum x_i^2=55\), \(\sum y_i=20\), \(\sum x_iy_i=66\).

Determinant: \(5(55)-15^2=275-225=50=nS_{xx}=5(10)\).

Row 2: \(\hat\beta_1=\frac{5(66)-15(20)}{50}=\frac{330-300}{50}=\frac{30}{50}=0.6\).

Row 1: \(\hat\beta_0=\frac{55(20)-15(66)}{50}=\frac{1100-990}{50}=\frac{110}{50}=2.2\).

SLR: \(\hat\beta_1=6/10=0.6\), \(\hat\beta_0=4-0.6(3)=4-1.8=2.2\).

Elimination in Section 5.3 gave the same \(\hat{\boldsymbol\beta}=(2.2,\,0.6)^T\). ▲

**Q2** Slides p.41

Practice Q2 · Lecture 6 · p.41 In the case of \(p=1\) covariate, find the covariance between \(\hat\beta_0\) and \(\hat\beta_1\) using the above variance covariance matrix for \(\hat{\boldsymbol\beta}\).

Answer Q2

\(\mathrm{cov}(U,W)=E[(U-E[U])(W-E[W])]\). The "above" matrix is \(\mathrm{Var}[\hat{\boldsymbol\beta}]=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}\) (p.27). Its off-diagonal entry is \(\mathrm{cov}(\hat\beta_0,\hat\beta_1)\).

1

\[\mathrm{Var}[\hat{\boldsymbol\beta}]=\left[\begin{array}{cc}\mathrm{Var}(\hat\beta_0)&\mathrm{cov}(\hat\beta_0,\hat\beta_1)\\\mathrm{cov}(\hat\beta_1,\hat\beta_0)&\mathrm{Var}(\hat\beta_1)\end{array}\right]=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}\]

2

\[\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\frac{\sigma^2}{nS_{xx}}\left[\begin{array}{cc}\sum x_i^2&-\sum x_i\\-\sum x_i&n\end{array}\right]\]

Q1, step 2.

3

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=\frac{\sigma^2}{nS_{xx}}\left(-\sum x_i\right)\]

Row 1, column 2.

4

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=\frac{\sigma^2}{nS_{xx}}\left(-n\bar x\right)\]

5

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=-\frac{\sigma^2\bar x}{S_{xx}}\]

Negative when \(\bar x>0\). ∎

The fitted line goes through \((\bar x,\bar y)\) (Figure 5-1). A larger \(\hat\beta_1\) gives a smaller \(\hat\beta_0=\bar y-\hat\beta_1\bar x\).

**Numerical check (five-point data).** Use \(\hat\sigma^2=0.8\) for \(\sigma^2\).

Formula: \(-\frac{0.8(3)}{10}=-\frac{2.4}{10}=-0.24\).

Matrix: \(V_{01}=-0.3\) (Section 05.4), \(\hat\sigma^2V_{01}=0.8(-0.3)=-0.24\). ▲

**Q3** Added

Data: \(n=4\), \(x=(0,1,2,3)\), \(y=(1,3,2,6)\). Fitted line: \(\hat y=0.9+1.4x\). (a) Calculate \(\boldsymbol{e}^T\boldsymbol{e}\), \(\hat\sigma^2_{MLE}\), and \(\hat\sigma^2\). (b) Calculate \(S_{xx}\) and \(\mathrm{SE}(\hat\beta_1)=\sqrt{\hat\sigma^2V_{11}}\). (c) Calculate \(\mathrm{cov}(\hat\beta_0,\hat\beta_1)\) with \(\hat\sigma^2\) for \(\sigma^2\).

Answer Q3

1

(a) Fitted values 05.1 How step 1

\[\hat y_1=0.9,\quad \hat y_2=0.9+1.4=2.3,\quad \hat y_3=0.9+2.8=3.7,\quad \hat y_4=0.9+4.2=5.1\]

2

(a) Residuals 05.1 How step 2

\[e_1=1-0.9=0.1,\quad e_2=3-2.3=0.7,\quad e_3=2-3.7=-1.7,\quad e_4=6-5.1=0.9\]

Check: \(\sum e_i=0.1+0.7-1.7+0.9=0\), \(\sum x_ie_i=0+0.7-3.4+2.7=0\).

3

(a) Residual sum of squares 05.1 How step 3

\[\boldsymbol{e}^T\boldsymbol{e}=0.01+0.49+2.89+0.81=4.2\]

4

(a) MLE 05.1 How step 4

\[\hat\sigma^2_{MLE}=\frac{4.2}{4}=1.05\]

5

(a) Unbiased estimate 05.2 How

\[n-p-1=4-1-1=2,\qquad \hat\sigma^2=\frac{4.2}{2}=2.1\]

Check: \(2.1/1.05=2=4/2=n/(n-p-1)\).

6

(b) \(S_{xx}\) and \(V_{11}\) Q1 step 2

\[\bar x=\frac{6}{4}=1.5,\qquad S_{xx}=2.25+0.25+0.25+2.25=5\]

\[V_{11}=\frac{n}{nS_{xx}}=\frac{1}{S_{xx}}=\frac15=0.2\]

Bottom-right entry of the Q1 inverse.

7

(b) SE 05.4 How steps 3–4

\[\hat\sigma^2V_{11}=2.1(0.2)=0.42,\qquad \mathrm{SE}(\hat\beta_1)=\sqrt{0.42}=0.6481\]

SLR form: \(\hat\sigma/\sqrt{S_{xx}}=\sqrt{2.1}/\sqrt{5}=1.4491/2.2361=0.6481\).

8

(c) Covariance Q2 step 5

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=-\frac{\hat\sigma^2\bar x}{S_{xx}}=-\frac{2.1(1.5)}{5}=-\frac{3.15}{5}=-0.63\]

Negative because \(\bar x=1.5>0\). ▲