Lecture 6: Multiple Linear Regression: Intro and Estimation

Unit 01Recap: matrix calculus, random vectors, and the multivariate normal

Plan · Slides p.1–7

Step 1: Take the derivative of a scalar with respect to a vector (Slides p.3).

Step 2: Find the expectation and variance of a random vector after a linear change (Slides p.4).

Step 3: Define the multivariate normal distribution (Slides p.5).

Step 4: Use four properties of the multivariate normal (Slides p.6).

Step 5: Write the regression model as \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\) (Slides p.7).

The slides give no data. The examples use small 2×2 or 3×3 matrices.

01.1Notation Slides p.1–2

Slide p.1: title "Lecture 6: Multiple Linear Regression: Intro & Estimation". Slide p.2: section page "Recap". Added

Vector: a column of numbers, with a bold lower-case name, for example \(\boldsymbol{y}=(y_1,\dots,y_k)^T\).

Matrix: a rectangular table of numbers, with a bold upper-case name, for example \(\boldsymbol{A}\). "\(k\times k\)" means \(k\) rows and \(k\) columns.

Transpose \(^T\): changes rows into columns. \(\boldsymbol{a}^T\) is a row.

Scalar: one number, with a plain name, for example \(z\) or \(\sigma^2\).

Identity matrix \(\boldsymbol{I}\): 1 on the diagonal, 0 at all other positions. Zero vector \(\boldsymbol{0}\): 0 at all positions.

01.2Matrix calculus Slides p.3

What Slides p.3

A partial derivative \(\partial z/\partial y_j\) is the rate of change of \(z\) when only \(y_j\) changes. A symmetric matrix has \(\boldsymbol{A}^T=\boldsymbol{A}\), that is, \(a_{ij}=a_{ji}\).

Matrix Calculus · Lecture 6 · p.3 Let \(z=f(y_1,\dots,y_k)\) and \(\boldsymbol{y}=\left[\begin{array}{c}y_1\\ \vdots\\ y_k\end{array}\right]\), then \[\frac{\partial z}{\partial \boldsymbol{y}}=\left[\begin{array}{c}\frac{\partial z}{\partial y_1}\\ \frac{\partial z}{\partial y_2}\\ \vdots\\ \frac{\partial z}{\partial y_k}\end{array}\right]\]

If \(z=\boldsymbol{a}^T\boldsymbol{y}\), where \(\boldsymbol{a}=(a_1,\dots,a_k)^T\) is a vector, then \(\frac{\partial z}{\partial \boldsymbol{y}}=\boldsymbol{a}\)
If \(z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}\) where \(\boldsymbol{A}\) is a \(k\times k\) matrix, then \(\frac{\partial z}{\partial \boldsymbol{y}}=\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\)
and if \(\boldsymbol{A}\) is symmetric, then: \(\frac{\partial z}{\partial \boldsymbol{y}}=2\boldsymbol{A}\boldsymbol{y}\)

Linear form \(\boldsymbol{a}^T\boldsymbol{y}=a_1y_1+\dots+a_ky_k\): each term is a constant times one entry. Quadratic form \(\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}\): each term is a constant times two entries.

How Added

How · Take the derivative with respect to a vector

Find the type of \(z\): linear form or quadratic form.
For a linear form, write \(\partial z/\partial\boldsymbol{y}=\boldsymbol{a}\).
For a quadratic form, compare each \(a_{ij}\) with \(a_{ji}\). All equal: write \(2\boldsymbol{A}\boldsymbol{y}\). Else: write \(\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\).
Write the result as a \(k\times1\) column.

Self-check: The result has length \(k\).

Self-check: Expand \(z\) and find each \(\partial z/\partial y_j\). The result must agree with step 2 or 3.

Example Added

Let \(k=2\), \(\boldsymbol{y}=(y_1,y_2)^T\), and use the point \(\boldsymbol{y}=(1,2)^T\).

Example (i): linear form, \(\boldsymbol{a}=(3,5)^T\).

Expand \(z\) How step 1

\[z=\boldsymbol{a}^T\boldsymbol{y}=3y_1+5y_2\]

Partial derivatives Self-check 2

\[\frac{\partial z}{\partial y_1}=3,\qquad \frac{\partial z}{\partial y_2}=5\]

Column How step 2

\[\frac{\partial z}{\partial\boldsymbol{y}}=\left[\begin{array}{c}3\\5\end{array}\right]=\boldsymbol{a}\]

Example (ii): symmetric quadratic form, \(\boldsymbol{A}=\left[\begin{array}{cc}2&1\\1&3\end{array}\right]\), \(a_{12}=a_{21}=1\).

Expand \(z\) How step 1

\[z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}=\left[\begin{array}{cc}y_1&y_2\end{array}\right]\left[\begin{array}{c}2y_1+y_2\\y_1+3y_2\end{array}\right]\]

\[=y_1(2y_1+y_2)+y_2(y_1+3y_2)\]

\[=2y_1^2+2y_1y_2+3y_2^2\]

Partial derivatives Self-check 2

\[\frac{\partial z}{\partial y_1}=4y_1+2y_2,\qquad \frac{\partial z}{\partial y_2}=2y_1+6y_2\]

Rule \(2\boldsymbol{A}\boldsymbol{y}\) How step 3

\[2\boldsymbol{A}\boldsymbol{y}=2\left[\begin{array}{c}2y_1+y_2\\y_1+3y_2\end{array}\right]=\left[\begin{array}{c}4y_1+2y_2\\2y_1+6y_2\end{array}\right]\]

Agrees with step 2.

Put in \(\boldsymbol{y}=(1,2)^T\)

\[z=2(1)^2+2(1)(2)+3(2)^2=2+4+12=18\]

\[\frac{\partial z}{\partial\boldsymbol{y}}=\left[\begin{array}{c}4(1)+2(2)\\2(1)+6(2)\end{array}\right]=\left[\begin{array}{c}8\\14\end{array}\right]\]

Example (iii): non-symmetric quadratic form, \(\boldsymbol{A}=\left[\begin{array}{cc}1&2\\0&3\end{array}\right]\), \(a_{12}=2\ne a_{21}=0\).

Expand \(z\) How step 1

\[z=\left[\begin{array}{cc}y_1&y_2\end{array}\right]\left[\begin{array}{c}y_1+2y_2\\3y_2\end{array}\right]\]

\[=y_1^2+2y_1y_2+3y_2^2\]

Partial derivatives Self-check 2

\[\frac{\partial z}{\partial y_1}=2y_1+2y_2,\qquad \frac{\partial z}{\partial y_2}=2y_1+6y_2\]

Rule \(\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\) How step 3

\[\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}=\left[\begin{array}{c}y_1+2y_2\\3y_2\end{array}\right]+\left[\begin{array}{c}y_1\\2y_1+3y_2\end{array}\right]\]

\[=\left[\begin{array}{c}2y_1+2y_2\\2y_1+6y_2\end{array}\right]\]

\(\boldsymbol{A}^T=\left[\begin{array}{cc}1&0\\2&3\end{array}\right]\). Agrees with step 2.

Put in \(\boldsymbol{y}=(1,2)^T\)

\[z=1^2+2(1)(2)+3(2)^2=1+4+12=17\]

\[\frac{\partial z}{\partial\boldsymbol{y}}=\left[\begin{array}{c}2(1)+2(2)\\2(1)+6(2)\end{array}\right]=\left[\begin{array}{c}6\\14\end{array}\right]\]

Why Added

Linear form:

\[z=\boldsymbol{a}^T\boldsymbol{y}=\sum_{i=1}^k a_iy_i\]

\[\frac{\partial z}{\partial y_j}=a_j\]

Only term \(i=j\) contains \(y_j\).

\[\frac{\partial z}{\partial\boldsymbol{y}}=(a_1,\dots,a_k)^T=\boldsymbol{a}\]

Definition on p.3. ∎

Quadratic form. \(a_{ij}\) is the entry of \(\boldsymbol{A}\) in row \(i\), column \(j\).

\[z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}=\sum_{i=1}^k\sum_{l=1}^k y_i\,a_{il}\,y_l\]

\[\frac{\partial z}{\partial y_j}=\sum_{l=1}^k a_{jl}y_l+\sum_{i=1}^k y_i a_{ij}\]

Product rule: \(y_j\) is in terms \(i=j\) and \(l=j\).

\[=(\boldsymbol{A}\boldsymbol{y})_j+\sum_{i=1}^k a_{ij}y_i\]

First sum: row \(j\) of \(\boldsymbol{A}\) times \(\boldsymbol{y}\).

\[=(\boldsymbol{A}\boldsymbol{y})_j+(\boldsymbol{A}^T\boldsymbol{y})_j\]

\(a_{ij}\) is entry \((j,i)\) of \(\boldsymbol{A}^T\).

\[\frac{\partial z}{\partial\boldsymbol{y}}=\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}^T\boldsymbol{y}\]

\[=\boldsymbol{A}\boldsymbol{y}+\boldsymbol{A}\boldsymbol{y}=2\boldsymbol{A}\boldsymbol{y}\]

Symmetric: \(\boldsymbol{A}^T=\boldsymbol{A}\). ∎

Unit 03 uses these rules on \(S(\boldsymbol\beta)\).

01.3Properties of random vectors Slides p.4

What Slides p.4

A random variable is a number whose value comes from a random result. A random vector has random variables as entries. The expectation \(E[\cdot]\) is the long-run average, taken entry by entry for a vector.

The variance of a random variable is its expected squared distance from its expectation. The covariance \(\mathrm{cov}(y_i,y_j)=E[(y_i-E[y_i])(y_j-E[y_j])]\) shows how much two variables change together. The covariance matrix \(\mathrm{Var}(\boldsymbol{y})\) has entry \((i,j)\) equal to \(\mathrm{cov}(y_i,y_j)\), and variances on the diagonal. Below, \(\boldsymbol{a}\), \(\boldsymbol{b}\), and \(\boldsymbol{A}\) are constants.

Properties of Random Vectors · Lecture 6 · p.4

\(E[\boldsymbol{a}]=\boldsymbol{a}\)
\(E[\boldsymbol{a}^T\boldsymbol{y}+\boldsymbol{b}]=\boldsymbol{a}^TE[\boldsymbol{y}]+\boldsymbol{b}\)
\(E[\boldsymbol{A}\boldsymbol{y}]=\boldsymbol{A}E[\boldsymbol{y}]\)
\(\mathrm{Var}(\boldsymbol{y})=E[(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{y}-E[\boldsymbol{y}])^T]\)
\(\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})=\boldsymbol{a}^T\mathrm{Var}(\boldsymbol{y})\boldsymbol{a}\)
\(\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})=\boldsymbol{A}\mathrm{Var}(\boldsymbol{y})\boldsymbol{A}^T\)

For the variance, \(\boldsymbol{A}\) goes on the left and \(\boldsymbol{A}^T\) on the right. \(\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})\) is a scalar. \(\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})\) is a matrix.

How Added

How · Expectation and variance after a linear change

Write \(\boldsymbol{A}\) (\(m\times k\)), \(E[\boldsymbol{y}]\) (\(k\times1\)), and \(\mathrm{Var}(\boldsymbol{y})\) (\(k\times k\)).
Calculate \(\boldsymbol{A}E[\boldsymbol{y}]\) (\(m\times1\)).
Calculate \(\boldsymbol{A}\mathrm{Var}(\boldsymbol{y})\), then multiply on the right by \(\boldsymbol{A}^T\) (\(m\times m\)).

Self-check: The variance matrix is symmetric.

Self-check: The diagonal entries are \(\ge 0\).

Example Added

Let \(E[\boldsymbol{y}]=(10,20)^T\), \(\mathrm{Var}(\boldsymbol{y})=\boldsymbol{I}\) (2×2), \(\boldsymbol{A}=\left[\begin{array}{cc}1&2\\3&4\end{array}\right]\), and \(\boldsymbol{a}=(1,1)^T\).

Expectation How step 2

\[E[\boldsymbol{A}\boldsymbol{y}]=\boldsymbol{A}E[\boldsymbol{y}]=\left[\begin{array}{c}1(10)+2(20)\\3(10)+4(20)\end{array}\right]=\left[\begin{array}{c}50\\110\end{array}\right]\]

Variance How step 3

\[\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})=\boldsymbol{A}\,\boldsymbol{I}\,\boldsymbol{A}^T=\boldsymbol{A}\boldsymbol{A}^T\]

\[=\left[\begin{array}{cc}1&2\\3&4\end{array}\right]\left[\begin{array}{cc}1&3\\2&4\end{array}\right]\]

\[=\left[\begin{array}{cc}1+4&3+8\\3+8&9+16\end{array}\right]=\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\]

Symmetric, positive diagonal: both self-checks pass.

Scalar case

\[\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})=\boldsymbol{a}^T\boldsymbol{I}\boldsymbol{a}=1^2+1^2=2\]

\(\boldsymbol{a}^T\boldsymbol{y}=y_1+y_2\).

Order is important: \(\boldsymbol{A}^T\boldsymbol{A}=\left[\begin{array}{cc}10&14\\14&20\end{array}\right]\) is a different matrix.

Why Added

\[\mathrm{Var}(\boldsymbol{A}\boldsymbol{y})=E[(\boldsymbol{A}\boldsymbol{y}-E[\boldsymbol{A}\boldsymbol{y}])(\boldsymbol{A}\boldsymbol{y}-E[\boldsymbol{A}\boldsymbol{y}])^T]\]

Definition of Var, p.4.

\[=E[(\boldsymbol{A}\boldsymbol{y}-\boldsymbol{A}E[\boldsymbol{y}])(\boldsymbol{A}\boldsymbol{y}-\boldsymbol{A}E[\boldsymbol{y}])^T]\]

\(E[\boldsymbol{A}\boldsymbol{y}]=\boldsymbol{A}E[\boldsymbol{y}]\).

\[=E[\boldsymbol{A}(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{A}(\boldsymbol{y}-E[\boldsymbol{y}]))^T]\]

\[=E[\boldsymbol{A}(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{y}-E[\boldsymbol{y}])^T\boldsymbol{A}^T]\]

\((\boldsymbol{B}\boldsymbol{C})^T=\boldsymbol{C}^T\boldsymbol{B}^T\).

\[=\boldsymbol{A}\,E[(\boldsymbol{y}-E[\boldsymbol{y}])(\boldsymbol{y}-E[\boldsymbol{y}])^T]\,\boldsymbol{A}^T\]

Constant matrices move out of \(E\).

\[=\boldsymbol{A}\,\mathrm{Var}(\boldsymbol{y})\,\boldsymbol{A}^T\]

One row \(\boldsymbol{a}^T\) gives the scalar rule. ∎

Unit 04 uses this rule to find \(\mathrm{Var}[\hat{\boldsymbol\beta}]\).

01.4Multivariate normal distribution Slides p.5

What Slides p.5

\(N(0,1)\) is the standard normal: mean 0, variance 1. "i.i.d." means independent and identically distributed. Two variables are independent if the value of one does not change the distribution of the other. The multivariate normal (MVN) is a distribution of a random vector, set fully by its mean \(\boldsymbol\mu\) and covariance matrix \(\boldsymbol\Sigma\).

The integral of the density \(f(\boldsymbol{y})\) over a region gives the probability that \(\boldsymbol{y}\) is in that region. The determinant \(|\boldsymbol\Sigma|\) is one number from a square matrix; for 2×2, \(\left|\begin{array}{cc}p&q\\r&s\end{array}\right|=ps-qr\). The inverse \(\boldsymbol\Sigma^{-1}\) satisfies \(\boldsymbol\Sigma\boldsymbol\Sigma^{-1}=\boldsymbol{I}\).

Multivariate Normal Distribution · Lecture 6 · p.5 Let \(\boldsymbol{z}=(z_1,\dots,z_n)^T\) be a random vector of i.i.d standard normal random variables, i.e. \(z_i\overset{iid}{\sim}N(0,1)\).
Then \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\) has multivariate normal distribution, i.e.: \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) where \(E[\boldsymbol{y}]=\boldsymbol\mu\) and \(\mathrm{Var}(\boldsymbol{y})=\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T\).
\(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\), if: \[f(\boldsymbol{y})=\frac{1}{(2\pi)^{\frac n2}|\boldsymbol\Sigma|^{\frac12}}\exp\left\{-\frac12(\boldsymbol{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\boldsymbol{y}-\boldsymbol\mu)\right\}\]

The slide gives two equivalent descriptions: a construction from \(n\) independent standard normals, and a density. Both use only \(\boldsymbol\mu\) (\(n\times1\)) and \(\boldsymbol\Sigma\) (\(n\times n\)).

How Added

How · Get the MVN parameters and the density

Read \(\boldsymbol\mu\), the added constant vector.
Calculate \(\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T\).
Write \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\).
For \(f(\boldsymbol{y})\), calculate \(|\boldsymbol\Sigma|\), then \(\boldsymbol\Sigma^{-1}\), then \((\boldsymbol{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\boldsymbol{y}-\boldsymbol\mu)\).
Put these three values into the density formula.

Self-check: \(\boldsymbol\Sigma\) is symmetric and \(|\boldsymbol\Sigma|>0\). Else \(|\boldsymbol\Sigma|^{1/2}\) and \(\boldsymbol\Sigma^{-1}\) do not exist.

Self-check: At \(\boldsymbol{y}=\boldsymbol\mu\), the exponential term is \(e^0=1\).

Example Added

Use \(\boldsymbol{A}\) from Section 01.3, \(n=2\), \(\boldsymbol\mu=(10,20)^T\), and \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\).

Mean How step 1

\[\boldsymbol\mu=\left[\begin{array}{c}10\\20\end{array}\right]\]

Covariance matrix How step 2

\[\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T=\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\]

Same product as Section 01.3.

Distribution How step 3

\[\boldsymbol{y}\sim MVN\left(\left[\begin{array}{c}10\\20\end{array}\right],\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\right)\]

Determinant How step 4

\[|\boldsymbol\Sigma|=5(25)-11(11)=125-121=4\]

Density at \(\boldsymbol{y}=\boldsymbol\mu\) Self-check 2

\[f(\boldsymbol\mu)=\frac{1}{(2\pi)^{\frac22}\,4^{\frac12}}\exp\{0\}\]

\[=\frac{1}{2\pi\times2}=\frac{1}{4\pi}=0.07957747\]

Second example: \(\boldsymbol\mu=\boldsymbol{0}\), \(\boldsymbol\Sigma=\boldsymbol{I}\) (2×2), \(\boldsymbol{y}=(1,-1)^T\). Added

Determinant and inverse How step 4

\[|\boldsymbol{I}|=1(1)-0(0)=1,\qquad \boldsymbol{I}^{-1}=\boldsymbol{I}\]

Quadratic form How step 4

\[(\boldsymbol{y}-\boldsymbol{0})^T\boldsymbol{I}(\boldsymbol{y}-\boldsymbol{0})=1^2+(-1)^2=2\]

Density How step 5

\[f(\boldsymbol{y})=\frac{1}{(2\pi)^{1}\,1^{\frac12}}\exp\left\{-\frac12(2)\right\}\]

\[=\frac{e^{-1}}{2\pi}=\frac{0.36787944}{6.28318531}=0.05854983\]

Check with two one-dimensional densities

\[\frac{1}{\sqrt{2\pi}}e^{-\frac12(1)^2}\times\frac{1}{\sqrt{2\pi}}e^{-\frac12(-1)^2}=\frac{e^{-1}}{2\pi}=0.05854983\]

\(N(0,1)\) density: \(\frac{1}{\sqrt{2\pi}}e^{-y^2/2}\).

The cloud centers near \(\boldsymbol\mu=(10,20)\). It slopes up-right because \(\mathrm{cov}(y_1,y_2)=11>0\). It is taller than wide because \(\mathrm{Var}(y_2)=25>\mathrm{Var}(y_1)=5\).

Figure 1-1. 300 simulated points \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\) with \(\boldsymbol{A}=\left[\begin{array}{cc}1&2\\3&4\end{array}\right]\) (blue). Orange cross: mean \(\boldsymbol\mu\).

Why Added

Each \(z_i\) has mean 0 and variance 1, and different \(z_i\) have covariance 0. Then \(E[\boldsymbol{z}]=\boldsymbol{0}\) and \(\mathrm{Var}(\boldsymbol{z})=\boldsymbol{I}\).

\[E[\boldsymbol{y}]=E[\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu]=\boldsymbol{A}E[\boldsymbol{z}]+\boldsymbol\mu\]

p.4 expectation rules.

\[=\boldsymbol{A}\boldsymbol{0}+\boldsymbol\mu=\boldsymbol\mu\]

\[\mathrm{Var}(\boldsymbol{y})=\mathrm{Var}(\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu)=\mathrm{Var}(\boldsymbol{A}\boldsymbol{z})\]

An added constant does not change variance.

\[=\boldsymbol{A}\,\mathrm{Var}(\boldsymbol{z})\,\boldsymbol{A}^T\]

p.4 variance rule.

\[=\boldsymbol{A}\,\boldsymbol{I}\,\boldsymbol{A}^T=\boldsymbol{A}\boldsymbol{A}^T=\boldsymbol\Sigma\]

∎

01.5Properties of \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) Slides p.6

What Slides p.6

A marginal distribution is the distribution of some entries of a vector. A conditional distribution \(\boldsymbol{y}_1\mid\boldsymbol{y}_2\) is the distribution of \(\boldsymbol{y}_1\) when \(\boldsymbol{y}_2\) is known. \(\Sigma_{ij}=\mathrm{cov}(y_i,y_j)\) is entry \((i,j)\) of \(\boldsymbol\Sigma\).

Properties of \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 6 · p.6 Linearity: If \(\boldsymbol{u}=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\), then: \[\boldsymbol{u}\sim MVN(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d},\,\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T)\] Marginal Distribution: If \(\tilde{\boldsymbol{y}}=(y_1,\dots,y_m)^T\subset\boldsymbol{y}\) is a vector subset of \(\boldsymbol{y}\), then \(\tilde{\boldsymbol{y}}\) is MVN-distributed.
Conditional Distribution: If \(\boldsymbol{y}=(\boldsymbol{y}_1^T,\boldsymbol{y}_2^T)^T\), then \(\boldsymbol{y}_1^T\mid\boldsymbol{y}_2^T\) is MVN-distributed.
Independence: If \(\Sigma_{ij}=0\), then \(\boldsymbol{y}_i\) and \(\boldsymbol{y}_j\) are independent.

Linearity: \(\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\) stays MVN. Its mean and variance follow the p.4 rules.

Marginal: any set of entries is MVN. One entry is normal.

Conditional: when the lower block \(\boldsymbol{y}_2\) is known, the upper block \(\boldsymbol{y}_1\) is MVN. \(\boldsymbol{y}_1^T\mid\boldsymbol{y}_2^T\) means \(\boldsymbol{y}_1\mid\boldsymbol{y}_2\). The lecture does not give its parameters.

Independence: for MVN, covariance 0 gives independence.

How Added

A linear combination is a sum of entries, each multiplied by a constant.

How · Use linearity to find the distribution of a new variable

Write \(\boldsymbol{u}=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\). For a subset, each row of \(\boldsymbol{C}\) has one 1 and all other entries 0.
Mean: calculate \(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d}\).
Variance: calculate \(\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T\).
Write \(\boldsymbol{u}\sim MVN(\text{step 2},\text{step 3})\). For one entry, write \(N(\cdot,\cdot)\).

Self-check: Columns of \(\boldsymbol{C}\) = length of \(\boldsymbol{y}\). Length of \(\boldsymbol{d}\) = rows of \(\boldsymbol{C}\).

Example Added

Use \(\boldsymbol{y}\sim MVN\left(\left[\begin{array}{c}10\\20\end{array}\right],\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\right)\) from Section 01.4.

(i) Linearity: \(u=y_1+y_2\).

Write \(u=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}\) How step 1

\[\boldsymbol{C}=\left[\begin{array}{cc}1&1\end{array}\right],\qquad \boldsymbol{d}=0\]

Mean How step 2

\[\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d}=1(10)+1(20)+0=30\]

Variance How step 3

\[\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T=\left[\begin{array}{cc}1&1\end{array}\right]\left[\begin{array}{cc}5&11\\11&25\end{array}\right]\left[\begin{array}{c}1\\1\end{array}\right]\]

\[=\left[\begin{array}{cc}5+11&11+25\end{array}\right]\left[\begin{array}{c}1\\1\end{array}\right]\]

\[=\left[\begin{array}{cc}16&36\end{array}\right]\left[\begin{array}{c}1\\1\end{array}\right]=16+36=52\]

Distribution How step 4

\[u=y_1+y_2\sim N(30,\,52)\]

(ii) Marginal: \(\boldsymbol{C}=\left[\begin{array}{cc}1&0\end{array}\right]\), \(\boldsymbol{d}=0\). Mean \(1(10)+0(20)=10\), variance \(\Sigma_{11}=5\): \(y_1\sim N(10,5)\).

(iii) Independence: \(\boldsymbol\Sigma=\boldsymbol{I}\), \(\boldsymbol\mu=\boldsymbol{0}\), \(\Sigma_{12}=0\). The joint density is the product of two \(N(0,1)\) densities: both give 0.05854983 at \((1,-1)\) (Section 01.4).

Why Added

Let \(\boldsymbol{y}=\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu\), with \(\boldsymbol\Sigma=\boldsymbol{A}\boldsymbol{A}^T\).

\[\boldsymbol{u}=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d}=\boldsymbol{C}(\boldsymbol{A}\boldsymbol{z}+\boldsymbol\mu)+\boldsymbol{d}\]

p.5 construction.

\[=(\boldsymbol{C}\boldsymbol{A})\boldsymbol{z}+(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d})\]

\[\Rightarrow\ \boldsymbol{u}\sim MVN\big(\boldsymbol{C}\boldsymbol\mu+\boldsymbol{d},\ (\boldsymbol{C}\boldsymbol{A})(\boldsymbol{C}\boldsymbol{A})^T\big)\]

Form "matrix × \(\boldsymbol{z}\) + vector": p.5 applies.

\[(\boldsymbol{C}\boldsymbol{A})(\boldsymbol{C}\boldsymbol{A})^T=\boldsymbol{C}\boldsymbol{A}\boldsymbol{A}^T\boldsymbol{C}^T\]

\((\boldsymbol{C}\boldsymbol{A})^T=\boldsymbol{A}^T\boldsymbol{C}^T\).

\[=\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T\]

∎ Marginal: \(\boldsymbol{C}\) selects entries.

The lecture states the conditional and independence properties without proof. Unit 04 uses linearity to get the distribution of \(\hat{\boldsymbol\beta}\).

01.6Recap: multiple linear regression Slides p.7

What Slides p.7

Multiple linear regression (MLR) models an outcome \(y_i\) as an intercept, plus a linear combination of \(p\) covariates \(x_{i1},\dots,x_{ip}\), plus a random error \(\epsilon_i\). The regression coefficients \(\beta_0,\dots,\beta_p\) and the error variance \(\sigma^2\) are unknown constants. \(n\) is the number of observations.

Recap: Multiple Linear Regression · Lecture 6 · p.7 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_Px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] We can write this: \[\left[\begin{array}{c}y_1\\y_2\\ \vdots\\y_n\end{array}\right]=\left[\begin{array}{cccccc}1&x_{11}&x_{12}&\dots&x_{1p}\\1&x_{21}&x_{22}&\dots&x_{2p}\\ \vdots&\vdots&\vdots&\vdots&\vdots\\1&x_{n1}&x_{n2}&\dots&x_{np}\end{array}\right]\left[\begin{array}{c}\beta_0\\ \beta_1\\ \vdots\\ \beta_p\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\ \epsilon_2\\ \vdots\\ \epsilon_n\end{array}\right]\] Or more simply: \[\begin{array}{rl}&\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon,\qquad \boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\\ \iff&\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\end{array}\]

Notation note The slide writes \(\beta_P\) in the first line and \(\beta_p\) after. Both mean the last index \(p\).

Row \(i\) is one observation. Column 1 of \(\boldsymbol{X}\) is all 1s, for the intercept \(\beta_0\). Column \(j+1\) is covariate \(j\). \(\boldsymbol{X}\) is \(n\times(p+1)\); \(\boldsymbol\beta\) is \((p+1)\times1\).

"\(\epsilon_i\) i.i.d. \(N(0,\sigma^2)\)" equals \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\): means 0, variances \(\sigma^2\), covariances 0.

Figure 1-2. Block sizes in \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\). Left of the dashed line: the column of 1s. Columns of \(\boldsymbol{X}\) = length \(p+1\) of \(\boldsymbol\beta\). \(\boldsymbol{X}\boldsymbol\beta\) is \(n\times1\).

How Added

How · Write \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\)

For \(\boldsymbol{y}\), put \(y_1,\dots,y_n\) in one column.
For \(\boldsymbol{X}\), write \((1,x_{i1},\dots,x_{ip})\) as row \(i\).
For \(\boldsymbol\beta\), put \(\beta_0,\dots,\beta_p\) in one column.
Check: row \(i\) of \(\boldsymbol{X}\) times \(\boldsymbol\beta\) equals \(\beta_0+\beta_1x_{i1}+\dots+\beta_px_{ip}\).

Self-check: \(\boldsymbol{X}\) is \(n\times(p+1)\), with a first column of 1s.

Example Added

Let \(n=3\), \(p=2\), covariates \((1,4)\), \((2,5)\), \((3,6)\). For the algebra only, let \(\boldsymbol\beta=(1,2,3)^T\).

Write \(\boldsymbol{X}\) How step 2

\[\boldsymbol{X}=\left[\begin{array}{ccc}1&1&4\\1&2&5\\1&3&6\end{array}\right]\quad(3\times3=n\times(p+1))\]

Calculate \(\boldsymbol{X}\boldsymbol\beta\) How step 4

\[\boldsymbol{X}\boldsymbol\beta=\left[\begin{array}{c}1+2(1)+3(4)\\1+2(2)+3(5)\\1+2(3)+3(6)\end{array}\right]=\left[\begin{array}{c}1+2+12\\1+4+15\\1+6+18\end{array}\right]=\left[\begin{array}{c}15\\20\\25\end{array}\right]\]

Then \(\boldsymbol{y}\sim MVN\big((15,20,25)^T,\ \sigma^2\boldsymbol{I}\big)\).

Why Added

Use linearity (Section 01.5) with \(\boldsymbol{C}=\boldsymbol{I}\) and constant \(\boldsymbol{d}=\boldsymbol{X}\boldsymbol\beta\).

\[\boldsymbol{y}=\boldsymbol{I}\boldsymbol\epsilon+\boldsymbol{X}\boldsymbol\beta\]

\[\boldsymbol{y}\sim MVN(\boldsymbol{I}\boldsymbol{0}+\boldsymbol{X}\boldsymbol\beta,\ \boldsymbol{I}(\sigma^2\boldsymbol{I})\boldsymbol{I}^T)\]

p.6 linearity.

\[\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\ \sigma^2\boldsymbol{I})\]

Reverse: \(\boldsymbol{d}=-\boldsymbol{X}\boldsymbol\beta\) gives \(\iff\). ∎

01.7Practice Added

Q1. Let \(z=\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y}\) with \(\boldsymbol{A}=\left[\begin{array}{cc}4&1\\1&2\end{array}\right]\). Find \(\frac{\partial z}{\partial\boldsymbol{y}}\) at \(\boldsymbol{y}=(1,-1)^T\). Check entry by entry.

Answer

\[\boldsymbol{A}^T=\boldsymbol{A}\ \Rightarrow\ \frac{\partial z}{\partial\boldsymbol{y}}=2\boldsymbol{A}\boldsymbol{y}=2\left[\begin{array}{c}4y_1+y_2\\y_1+2y_2\end{array}\right]=\left[\begin{array}{c}8y_1+2y_2\\2y_1+4y_2\end{array}\right]\]

\[z=4y_1^2+2y_1y_2+2y_2^2,\quad \frac{\partial z}{\partial y_1}=8y_1+2y_2,\quad \frac{\partial z}{\partial y_2}=2y_1+4y_2\]

Agrees with step 1.

\[\left.\frac{\partial z}{\partial\boldsymbol{y}}\right|_{(1,-1)}=\left[\begin{array}{c}8(1)+2(-1)\\2(1)+4(-1)\end{array}\right]=\left[\begin{array}{c}6\\-2\end{array}\right]\]

Q2. Let \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\), \(\boldsymbol\mu=(1,2)^T\), \(\boldsymbol\Sigma=\left[\begin{array}{cc}2&1\\1&3\end{array}\right]\). Find the distribution of \(u=y_1-y_2\).

Answer

\[u=\boldsymbol{C}\boldsymbol{y}+\boldsymbol{d},\quad \boldsymbol{C}=\left[\begin{array}{cc}1&-1\end{array}\right],\ \boldsymbol{d}=0\]

\[\boldsymbol{C}\boldsymbol\mu=1(1)+(-1)(2)=-1\]

\[\boldsymbol{C}\boldsymbol\Sigma=\left[\begin{array}{cc}2-1&1-3\end{array}\right]=\left[\begin{array}{cc}1&-2\end{array}\right]\]

\[\boldsymbol{C}\boldsymbol\Sigma\boldsymbol{C}^T=\left[\begin{array}{cc}1&-2\end{array}\right]\left[\begin{array}{c}1\\-1\end{array}\right]=1+2=3\]

\[u\sim N(-1,\,3)\]

p.6 linearity.

Q3. An MLR has \(n=50\) and \(p=3\), with \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\). (a) Give the dimensions of \(\boldsymbol{X}\), \(\boldsymbol\beta\), and \(\mathrm{Var}(\boldsymbol{y})\). (b) Show \(E[\boldsymbol{y}]=\boldsymbol{X}\boldsymbol\beta\) with the random-vector properties.

Answer

(a) \(\boldsymbol{X}\): \(n\times(p+1)=50\times4\). \(\boldsymbol\beta\): \(4\times1\). \(\mathrm{Var}(\boldsymbol{y})=\sigma^2\boldsymbol{I}\): \(50\times50\).

(b)

\[E[\boldsymbol{y}]=E[\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon]\]

\[=\boldsymbol{X}\boldsymbol\beta+E[\boldsymbol\epsilon]\]

p.4: \(E[\boldsymbol{a}]=\boldsymbol{a}\) for constant \(\boldsymbol{X}\boldsymbol\beta\).

\[=\boldsymbol{X}\boldsymbol\beta+\boldsymbol{0}=\boldsymbol{X}\boldsymbol\beta\]

Unit 02Data example and interpreting regression coefficients

Plan · Slides p.8–16

Step 1 (Slides p.8–9): meet the data set satisfaction.csv.

Step 2 (Slides p.10–11): read the summary numbers and the scatter plots.

Step 3 (Slides p.12): list the questions of the lecture.

Step 4 (Slides p.13): write the model in scalar form and matrix form.

Step 5 (Slides p.14–16): interpret the intercept \(\beta_0\) and each slope \(\beta_j\).

Basic words · Added

Variable: one column of a data table.

Observation: one row of a data table. In this unit, one patient.

The subscript \(i\) gives the observation. The subscript \(j\) gives the covariate.

02.1Title page Slides p.8

Slides p.8 Slide p.8 repeats the title "Lecture 6: Multiple Linear Regression: Intro & Estimation".

Slides p.1–7 are a recap. New material starts on Slides p.9.

02.2Data example: satisfaction.csv Slides p.9

What Slides p.9

Data Example · Lecture 6 · p.9

satisfaction.csv contains data on \(n = 46\) hospital patients. The variables are:

Satisfaction: degree of satisfaction with quality of care (higher values = more)
Age: age of patient in years
Severity: severity score for patient condition
Stress: patient self reported degree of stress

Variable	Role
Satisfaction	outcome \(y_i\)
Age	covariate 1, \(x_{i1}\)
Severity	covariate 2, \(x_{i2}\)
Stress	covariate 3, \(x_{i3}\)

\(n=46\) observations, \(p=3\) covariates.

02.3Exploratory data analysis: first rows and summary numbers Slides p.10

Exploratory data analysis (EDA): a check of the range, center, and shape of the data before you make a model.

What Slides p.10

First 6 rows:

Row	Satisfaction	Age	Severity	Stress
1	48	50	51	2.3
2	57	36	46	2.3
3	66	40	48	2.2
4	70	41	44	1.8
5	89	28	43	1.8
6	36	49	54	2.9

Summary numbers:

	Satisfaction	Age	Severity	Stress
Minimum	26.00	22.00	41.00	1.800
1st quartile	48.25	31.25	48.00	2.100
Median	60.00	37.50	50.50	2.300
Mean	61.57	38.39	50.43	2.287
3rd quartile	76.75	44.75	53.00	2.475
Maximum	92.00	55.00	62.00	2.900

Quartile: a value that divides the sorted data into four equal parts. About 25% of the data are below the 1st quartile, about 75% below the 3rd.

Median: the middle value of the sorted data.

Sample mean \(\bar y=\frac1n\sum_{i=1}^n y_i\).

How Added

Read the summary numbers of one variable

Find the column of the variable.
Read the minimum and maximum. They give the range.
Read the median and mean. They give the center.
Read the 1st and 3rd quartiles. The middle 50% of the data are between them.

Self-check: minimum ≤ 1st quartile ≤ median ≤ 3rd quartile ≤ maximum. The mean is in the range.

Example · Read the two tables Added

Patient 1: \(y_1=48\), \(x_{11}=50\), \(x_{12}=51\), \(x_{13}=2.3\).

Variable	Range	Mean
Satisfaction	26 to 92	61.57 (median 60)
Age	22 to 55 years	38.39
Severity	41 to 62	50.43
Stress	1.8 to 2.9	2.287

Self-check: For Age, 22 ≤ 31.25 ≤ 37.50 ≤ 44.75 ≤ 55, and 38.39 is between 22 and 55.

No covariate has the value 0. Section 02.7 uses this fact. ▲

02.4Scatter plot matrix Slides p.11

What Slides p.11

Slides p.11 shows a scatter plot matrix of the four variables.

Scatter plot: one point for each observation. The horizontal position is one variable, the vertical position is another.

Scatter plot matrix (pairs plot): a grid with the scatter plot of each pair of variables. 4 variables give a 4×4 grid. The diagonal cells show only the names.

How Added

Find the scatter plot of one pair of variables

Find variable A on the diagonal. Note its row.
Find variable B on the diagonal. Note its column.
Go to row A, column B. The vertical axis is A. The horizontal axis is B.
Points from lower left to upper right show a positive association.
Points from upper left to lower right show a negative association.

Self-check: Each axis covers the minimum to the maximum of its variable in the summary table.

Example · Satisfaction against Age Added

Row 1, column 2: vertical axis Satisfaction, horizontal axis Age.

Figure 2-1. Satisfaction against Age, 46 patients. Orange: patient 1 (Age 50, Satisfaction 48).

The points go from upper left to lower right (step 5). Satisfaction and Age have a negative association.

The axes cover 26 to 92 and 22 to 55, as in the summary table.

This plot does not hold Severity and Stress fixed. Section 02.7 does. ▲

02.5Motivating questions Slides p.12

What Slides p.12

Motivating Questions · Lecture 6 · p.12

Is mean satisfaction associated with stress, conditional on age and severity?
How does mean satisfaction differ for older patients vs younger patients with the same severity and stress scores?
Given a patient's stress, age, and severity, can we predict their satisfaction?
Later: is the (conditional) association between stress and satisfaction different for older patients than for younger patients?
Many others...

Association: when one variable changes, the mean of another variable changes.

Conditional on: with the other variables held at fixed values.

Questions 1 and 2 hold other variables fixed. Figure 2-1 cannot do this. We need a model with all three covariates.

02.6The multiple linear regression model Slides p.13

What Slides p.13

Multiple Linear Regression · Lecture 6 · p.13 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_Px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\quad i=1,\dots,n\]

\[\boldsymbol{y}=\boldsymbol{X}\boldsymbol{\beta}+\boldsymbol{\epsilon},\quad \boldsymbol{\epsilon}\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\] \[\iff\] \[\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol{\beta},\sigma^2\boldsymbol{I})\]

Note: we assume \(p+1<n\)

This is the model of Section 01.6, with \(i=1,\dots,n\) added.

\(p+1<n\): there are fewer unknown coefficients than observations. It makes \(n-(p+1)\) positive. Unit 05 divides by it.

How Added

Write the model and \(\boldsymbol{X}\) for a data set

Find the outcome and the covariates. Write \(n\) and \(p\).
Write the scalar form with the variable names.
Put \(n\) 1s in column 1 of \(\boldsymbol{X}\).
Put covariate \(j\) in column \(j+1\).
Check \(p+1<n\).

Self-check: \(\boldsymbol{X}\) is \(n\times(p+1)\). \(\boldsymbol{\beta}\) has \(p+1\) entries. Row \(i\) of \(\boldsymbol{X}\) matches row \(i\) of the data.

Example · The model for satisfaction.csv Added

Step 1. \(n=46\), \(p=3\).

Step 2.

\[\text{Satisfaction}_i=\beta_0+\beta_1\,\text{Age}_i+\beta_2\,\text{Severity}_i+\beta_3\,\text{Stress}_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\quad i=1,\dots,46\]

Steps 3–4. First three rows from Slides p.10:

\[\boldsymbol{X}=\left[\begin{array}{cccc}1&50&51&2.3\\1&36&46&2.3\\1&40&48&2.2\\\vdots&\vdots&\vdots&\vdots\end{array}\right]_{46\times4},\qquad \boldsymbol{\beta}=\left[\begin{array}{c}\beta_0\\\beta_1\\\beta_2\\\beta_3\end{array}\right]\]

Step 5. \(4<46\). The assumption is true.

Self-check: \(46\times4=n\times(p+1)\). Row 1 is patient 1: Age 50, Severity 51, Stress 2.3.

Coefficient estimates used in this unit. Estimate: a number calculated from the data in place of an unknown true value. A hat marks it, for example \(\hat\beta_1\).

Lecture 6 does not print the estimates. Lecture 7 · p.51 prints a 95% confidence interval for each coefficient. Confidence interval: a range of values for an unknown coefficient, calculated from the data.

The estimate is the midpoint of each interval:

\[\hat\beta_0=\frac{121.911727+195.0707761}{2}=\frac{316.9825031}{2}=158.4912516\approx158.491\]

\[\hat\beta_1=\frac{-1.575093+(-0.7081303)}{2}=\frac{-2.2832233}{2}=-1.1416117\approx-1.142\]

\[\hat\beta_2=\frac{-1.434831+0.5508228}{2}=\frac{-0.8840082}{2}=-0.4420041\approx-0.442\]

\[\hat\beta_3=\frac{-27.797859+0.8575324}{2}=\frac{-26.9403266}{2}=-13.4701633\approx-13.470\]

This unit uses \(\hat{\boldsymbol\beta}=(158.491,\,-1.142,\,-0.442,\,-13.470)^T\). Unit 03 calculates it. ▲

Why Added

The scalar form and the matrix form give the same model. Row \(i\) of \(\boldsymbol{X}\boldsymbol\beta\):

\[[\boldsymbol{X}\boldsymbol\beta]_i=\left[\begin{array}{cccc}1&x_{i1}&\cdots&x_{ip}\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]\]

\[=1\cdot\beta_0+x_{i1}\beta_1+\cdots+x_{ip}\beta_p\]

\[=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\]

Add \(\epsilon_i\) to get the scalar form. The form \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\) comes from linearity (Slides p.6, Section 01.6).

Check with patient 1, row \((1,50,51,2.3)\):

\[1(158.491)+50(-1.142)+51(-0.442)+2.3(-13.470)\]

\[=158.491-57.1-22.542-30.981\]

\[=47.868\]

This equals the scalar form \(\hat\beta_0+\hat\beta_1(50)+\hat\beta_2(51)+\hat\beta_3(2.3)\). ∎

02.7Interpreting regression coefficients Slides p.14–16

Slides p.14–16 Slides p.14 and p.15 are the first parts of Slides p.16. This section follows p.16.

What Slides p.14–16

Interpreting Regression Coefficients · Lecture 6 · p.14–16

To interpret a parameter, isolate it!

\[E[y_i\,|\,x_{i1},\dots,x_{ip}]=\beta_0+\beta_1x_{i1}\cdots+\beta_Px_{ip}\]

What does \(\beta_0\) represent?

\[E[y_i\,|\,x_{i1}=\cdots=x_{ip}=0]=\beta_0+\beta_1(0)+\cdots+\beta_P(0)=\beta_0\]

\(\implies\beta_0\) is the mean outcome when all covariates are set to 0
May not be interpretable: e.g. what does it mean for age to be 0 in a sample of adults?

What does \(\beta_1\) represent?

\[E[y_i\,|\,x_{i1}=x_1,x_{i2}=x_2,\dots,x_{ip}=x_P]=\beta_0+\beta_1(x_1)+\beta_2(x_2)+\cdots+\beta_P(x_P)\] \[E[y_i\,|\,x_{i1}=(x_1+1),x_{i2}=x_2,\dots,x_{ip}=x_P]=\beta_0+\beta_1(x_1+1)+\cdots+\beta_P(x_P)\]

\(E[y_i\,|\,x_{i1}=(x_1+1),x_{i2}=x_2,\dots,x_{ip}=x_P]-E[y_i\,|\,x_{i1}=x_1,x_{i2}=x_2,\dots,x_{ip}=x_P]=\beta_1\)
More generally \(\beta_j\) represents the difference in mean outcome for a one-unit change in the \(j^{th}\) covariate, holding other covariates fixed.

Conditional expectation \(E[y_i\,|\,x_{i1},\dots,x_{ip}]\): the mean of \(y_i\) at the given covariate values. Read \(|\) as "given".

If 0 is outside the range of the data, \(\beta_0\) has no practical meaning.

"Holding other covariates fixed" is a necessary part of each slope interpretation.

How Added

Interpret one coefficient \(\beta_j\)

Write \(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\).
Intercept: put 0 for all covariates. The result is \(\beta_0\).
Check in the summary table if 0 is in the range of each covariate.
Slope: write the expression with covariate \(j\) at \(x_j\), then at \(x_j+1\).
Give the other covariates the same values in both.
Subtract. Only \(\beta_j\) remains.
Write: a one-unit increase in covariate \(j\) changes the mean outcome by \(\beta_j\), other covariates held fixed.

Self-check: The sentence names the covariate and its unit, the direction and size, and "held fixed". With an estimate, it says "estimated mean outcome".

Example · Interpret \(\hat\beta_0\) and \(\hat\beta_1\) Added

Step 1.

\[\hat\beta_0+\hat\beta_1\,\text{Age}+\hat\beta_2\,\text{Severity}+\hat\beta_3\,\text{Stress}=158.491-1.142\,\text{Age}-0.442\,\text{Severity}-13.470\,\text{Stress}\]

Step 2 (intercept). Age = Severity = Stress = 0:

\[158.491-1.142(0)-0.442(0)-13.470(0)\]

\[=158.491=\hat\beta_0\]

Step 3. Minimum Age is 22, Severity 41, Stress 1.8 (Section 02.3). \(\hat\beta_0\) has no practical meaning. This is the second bullet of Slides p.15.

Steps 4–5 (Age). Hold Severity = 50, Stress = 2.3. Use Age 40, then 41.

\[\text{Age }40:\ 158.491-1.142(40)-0.442(50)-13.470(2.3)\]

\[=158.491-45.68-22.1-30.981\]

\[=59.73\]

\[\text{Age }41:\ 158.491-1.142(41)-0.442(50)-13.470(2.3)\]

\[=158.491-46.822-22.1-30.981\]

\[=58.588\]

Step 6.

\[58.588-59.73=-1.142=\hat\beta_1\]

\(-22.1\) and \(-30.981\) cancel.

Step 7. Each one-year increase in Age decreases the estimated mean satisfaction by 1.142, with Severity and Stress held fixed.

\(\hat\beta_2=-0.442\): each one-point increase in Severity decreases the estimated mean satisfaction by 0.442, with Age and Stress held fixed.

\(\hat\beta_3=-13.470\): each one-unit increase in Stress decreases the estimated mean satisfaction by 13.470, with Age and Severity held fixed.

Figure 2-2 holds Severity and Stress at their means, 50.43 and 2.287: \(158.491-0.442(50.43)-13.470(2.287)-1.142\,\text{Age}=105.395-1.142\,\text{Age}\) (3 decimals).

Figure 2-2. Data (blue) and the estimated mean line (orange), slope \(\hat\beta_1=-1.142\). Green: Age 30 and 50. The line drops \(20\times1.142=22.84\).

Other fixed values, for example Severity = 60, move the line up or down. The slope stays \(-1.142\). ▲

Why Slides p.14–16

\[E[y_i\,|\,x_{i1},\dots,x_{ip}]=E[\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\,|\,x_{i1},\dots,x_{ip}]\]

Model, Slides p.13

\[=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+E[\epsilon_i]\]

Given covariates, first part is fixed

\[=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\]

\(E[\epsilon_i]=0\)

\[E[y_i\,|\,x_{i1}=\cdots=x_{ip}=0]=\beta_0+\beta_1(0)+\cdots+\beta_p(0)\]

\[=\beta_0\]

\[E[y_i\,|\,x_{ij}=x_j+1,\ \text{others fixed}]=\beta_0+\cdots+\beta_j(x_j+1)+\cdots+\beta_px_p\]

\[E[y_i\,|\,x_{ij}=x_j,\ \text{others fixed}]=\beta_0+\cdots+\beta_jx_j+\cdots+\beta_px_p\]

\[\text{line 6}-\text{line 7}=\beta_j(x_j+1)-\beta_jx_j\]

Other terms are equal and cancel

\[=\beta_jx_j+\beta_j-\beta_jx_j\]

\[=\beta_j\]

Line 8 needs equal values of the other covariates. The result does not depend on \(x_j\): Age 40 to 41 and Age 30 to 31 both give \(\beta_1\). ∎

02.8Practice Added

Use \(\hat{\boldsymbol\beta}=(158.491,\,-1.142,\,-0.442,\,-13.470)^T\).

Q1. Interpret \(\hat\beta_3=-13.470\). Two patients have the same Age and Severity. Their Stress differs by 0.5. Find the difference in estimated mean satisfaction.

Answer

Each one-unit increase in Stress decreases the estimated mean satisfaction by 13.470, with Age and Severity held fixed.

\(0.5\times(-13.470)=-6.735\)

The patient with higher Stress has an estimated mean satisfaction 6.735 lower.

Q2. Two patients have Severity = 50 and Stress = 2.3. Their ages are 30 and 50. Find each estimated mean satisfaction and the difference. Write the difference with \(\hat\beta_1\).

Answer

\[\text{Age }30:\ 158.491-1.142(30)-0.442(50)-13.470(2.3)\]

\[=158.491-34.26-22.1-30.981\]

\[=71.15\]

\[\text{Age }50:\ 158.491-1.142(50)-0.442(50)-13.470(2.3)\]

\[=158.491-57.1-22.1-30.981\]

\[=48.31\]

Difference: \(48.31-71.15=-22.84\).

With \(\hat\beta_1\): \(20\times\hat\beta_1=20\times(-1.142)=-22.84\).

The Severity and Stress terms cancel. The older patient is 22.84 lower.

Q3. Patient 5 (Slides p.10) has Age 28, Severity 43, Stress 1.8. Write row 5 of \(\boldsymbol{X}\). Find the estimated mean satisfaction.

Answer

Row 5 of \(\boldsymbol{X}\): \((1,28,43,1.8)\).

\[1(158.491)+28(-1.142)+43(-0.442)+1.8(-13.470)\]

\[=158.491-31.976-19.006-24.246\]

\[=83.263\]

Estimate 83.263. Observed value 89.

Unit 03Estimating β: least squares and maximum likelihood

Plan · Slides p.17–22

Step 1: State the model (Slides p.17).

Step 2: Least squares: minimize \(S(\boldsymbol\beta)\) (Slides p.18–20).

Step 3: Solve a small data set by hand (Added).

Step 4: Maximum likelihood: link \(\ell\) to \(S(\boldsymbol\beta)\) (Slides p.21–22).

03.1Title page: Estimating \(\boldsymbol\beta\) Slides p.17

Model: \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\). The errors are independent normal, each with variance \(\sigma^2\).

Estimator \(\hat{\boldsymbol\beta}\): a formula that gives a value for \(\boldsymbol\beta\) from the data.

The slides use the satisfaction data (\(n=46\), \(p=3\), Slides p.9). The example uses a smaller data set.

03.2Least squares estimation Slides p.18–20

Slides p.18–19 build up p.20.

What Slides p.20

Least Squares Estimation · Lecture 6 · p.20 Want to minimize the sum of squares, i.e.: \[\begin{aligned} S(\boldsymbol\beta)&=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\\ &=\boldsymbol{y}^T\boldsymbol{y}-\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta-\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\\ &=\boldsymbol{y}^T\boldsymbol{y}-2\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta \end{aligned}\] \[\begin{aligned} \frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta&=0\\ (\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta&=\boldsymbol{X}^T\boldsymbol{y}\\ \hat{\boldsymbol\beta}&=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y} \end{aligned}\] assuming the columns of \(\boldsymbol{X}\) are linearly independent

Sum of squares: \(S(\boldsymbol\beta)=\sum_{i=1}^n\big(y_i-(\boldsymbol{X}\boldsymbol\beta)_i\big)^2\), one number.

Least squares (OLS, ordinary least squares): the estimate \(\hat{\boldsymbol\beta}\) is the \(\boldsymbol\beta\) with the smallest \(S(\boldsymbol\beta)\).

Linearly independent columns: no column of \(\boldsymbol{X}\) is a linear combination of the others.

Normal equations: \((\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\). With linearly independent columns, they have one solution.

How Added

Calculate \(\hat{\boldsymbol\beta}\) from data

Write \(\boldsymbol{y}\) (\(n\times1\)) and \(\boldsymbol{X}\) (\(n\times(p+1)\), first column all 1s).
Calculate \(\boldsymbol{X}^T\boldsymbol{X}\). Entry \((j,k)\) is \(\sum_i x_{ij}x_{ik}\).
Calculate \(\boldsymbol{X}^T\boldsymbol{y}\). Entry \(j\) is \(\sum_i x_{ij}y_i\).
Make sure that \(\det(\boldsymbol{X}^T\boldsymbol{X})\neq0\).
Solve \((\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\).

Self-check: (1) \(\boldsymbol{X}^T\boldsymbol{X}\) is symmetric, and its entry \((1,1)\) is \(n\). (2) Entry 1 of \(\boldsymbol{X}^T\boldsymbol{y}\) is \(\sum_iy_i\). (3) \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}\) equals \(\boldsymbol{X}^T\boldsymbol{y}\).

Example Added

\(n=4\), one covariate (\(p=1\)), \(\boldsymbol\beta=(\beta_0,\beta_1)^T\).

Data: \(x=1,2,3,4\) and \(y=2,3,5,6\).

Write \(\boldsymbol{y}\) and \(\boldsymbol{X}\) How step 1

\[\boldsymbol{y}=\left[\begin{array}{c}2\\3\\5\\6\end{array}\right],\qquad \boldsymbol{X}=\left[\begin{array}{cc}1&1\\1&2\\1&3\\1&4\end{array}\right]\]

Calculate \(\boldsymbol{X}^T\boldsymbol{X}\) How step 2

\[(1,1):\ 1+1+1+1=4\]

\[(1,2)=(2,1):\ 1+2+3+4=10\]

\[(2,2):\ 1^2+2^2+3^2+4^2=1+4+9+16=30\]

\[\boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}4&10\\10&30\end{array}\right]\]

Self-check: symmetric; entry \((1,1)=4=n\).

Calculate \(\boldsymbol{X}^T\boldsymbol{y}\) How step 3

\[\text{entry }1:\ 2+3+5+6=16\]

\[\text{entry }2:\ 1(2)+2(3)+3(5)+4(6)=2+6+15+24=47\]

\[\boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}16\\47\end{array}\right]\]

Self-check: entry 1 is \(\sum_iy_i\).

Make sure that the inverse exists How step 4

\[\det(\boldsymbol{X}^T\boldsymbol{X})=4(30)-10(10)=120-100=20\neq0\]

The columns of \(\boldsymbol{X}\) are linearly independent.

Find \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\) How step 5

Rows \((a,b),(c,d)\) → \(\frac{1}{ad-bc}\) times rows \((d,-b),(-c,a)\).

\[(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\frac{1}{20}\left[\begin{array}{cc}30&-10\\-10&4\end{array}\right]\]

Calculate \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\) How step 5 · Slides p.20

\[\hat{\boldsymbol\beta}=\frac{1}{20}\left[\begin{array}{c}30(16)-10(47)\\-10(16)+4(47)\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\frac{1}{20}\left[\begin{array}{c}480-470\\-160+188\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\frac{1}{20}\left[\begin{array}{c}10\\28\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}\hat\beta_0\\\hat\beta_1\end{array}\right]=\left[\begin{array}{c}0.5\\1.4\end{array}\right]\]

Basis: Slides p.20. Step 4 makes the solution unique.

Put \(\hat{\boldsymbol\beta}\) back into the equations Self-check 3

\[\text{row }1:\ 4(0.5)+10(1.4)=2+14=16\]

\[\text{row }2:\ 10(0.5)+30(1.4)=5+42=47\]

Self-check: 16 and 47 equal \(\boldsymbol{X}^T\boldsymbol{y}\).

Model values \(0.5+1.4x\): 1.9, 3.3, 4.7, 6.1. Differences \(y_i-(\boldsymbol{X}\hat{\boldsymbol\beta})_i\): 0.1, −0.3, 0.3, −0.1.

\[S(\hat{\boldsymbol\beta})=0.1^2+(-0.3)^2+0.3^2+(-0.1)^2=0.01+0.09+0.09+0.01=0.2\]

Result: \(\hat{\boldsymbol\beta}=(0.5,\,1.4)^T\), \(S(\hat{\boldsymbol\beta})=0.2\). ▲

In the figure, \(\beta_0\) stays at 0.5 and only \(\beta_1\) changes. Then \(S=0.2+30(\beta_1-1.4)^2\), with \(30=\sum_ix_i^2\).

Figure 3-1. \(x=1,2,3,4\), \(y=2,3,5,6\). Top: \(S\) is smallest at \(\hat\beta_1=1.4\). Bottom: the log-likelihood (Section 03.3) is largest at the same point.

At \(\beta_1=\hat\beta_1\pm0.1\), \(S=0.2+30(0.1)^2=0.5\).

Why Slides p.20

Rules from Unit 01, with \(\boldsymbol\beta\) in place of \(\boldsymbol{y}\):

Rule A: for a constant vector \(\boldsymbol{a}\), \(\dfrac{\partial(\boldsymbol{a}^T\boldsymbol{y})}{\partial \boldsymbol{y}}=\boldsymbol{a}\).

Rule B: for a constant symmetric matrix \(\boldsymbol{A}\), \(\dfrac{\partial(\boldsymbol{y}^T\boldsymbol{A}\boldsymbol{y})}{\partial \boldsymbol{y}}=2\boldsymbol{A}\boldsymbol{y}\).

\[S(\boldsymbol\beta)=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\[S(\boldsymbol\beta)=(\boldsymbol{y}^T-\boldsymbol\beta^T\boldsymbol{X}^T)(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\((\boldsymbol{X}\boldsymbol\beta)^T=\boldsymbol\beta^T\boldsymbol{X}^T\).

\[S(\boldsymbol\beta)=\boldsymbol{y}^T\boldsymbol{y}-\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta-\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

\[\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}=(\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y})^T=\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta\]

A \(1\times1\) value equals its transpose.

\[S(\boldsymbol\beta)=\boldsymbol{y}^T\boldsymbol{y}-2\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Line 3 of Slides p.20.

\[\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta=(\boldsymbol{X}^T\boldsymbol{y})^T\boldsymbol\beta\]

Form \(\boldsymbol{a}^T\boldsymbol\beta\), \(\boldsymbol{a}=\boldsymbol{X}^T\boldsymbol{y}\).

\[\frac{\partial(\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta)}{\partial\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\]

Rule A.

\[(\boldsymbol{X}^T\boldsymbol{X})^T=\boldsymbol{X}^T(\boldsymbol{X}^T)^T=\boldsymbol{X}^T\boldsymbol{X}\]

Symmetric: Rule B applies.

\[\frac{\partial(\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta)}{\partial\boldsymbol\beta}=2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Rule B with \(\boldsymbol{A}=\boldsymbol{X}^T\boldsymbol{X}\).

\[\frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=0-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Steps 7 and 9.

\[-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=0\]

The derivative is 0 at the minimum.

\[2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=2\boldsymbol{X}^T\boldsymbol{y}\]

\[(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\]

\[\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

Multiply on the left by \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\). ∎

Check of step 11 on the example: \(-2(16,47)^T+2(16,47)^T=(0,0)^T\).

03.3Maximum likelihood Slides p.21–22

Slide p.21 builds up p.22.

What Slides p.22

Maximum Likelihood · Lecture 6 · p.22 Under our assumptions, we have: \[\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\] \[\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})=\frac{1}{(2\pi)^{\frac n2}|\sigma^2\boldsymbol{I}|^{\frac12}}\exp\left\{-\frac12(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\sigma^2\boldsymbol{I})^{-1}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\right\}\] \[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\] Look familiar?
So \(\hat{\boldsymbol\beta}_{MLE}=\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\)
• I.e. MLE=OLS (when all our assumptions hold)

Likelihood \(\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})\): the MVN density of the observed data, as a function of \(\boldsymbol\beta\) and \(\sigma^2\).

Log-likelihood \(\ell\): \(\log\mathcal{L}\) (natural log). It has its maximum where \(\mathcal{L}\) has it.

Maximum likelihood estimator (MLE) \(\hat{\boldsymbol\beta}_{MLE}\): the \(\boldsymbol\beta\) with the largest \(\mathcal{L}\).

"Look familiar?": only the last term of \(\ell\) contains \(\boldsymbol\beta\), and it is \(-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\). The largest \(\ell\) occurs at the smallest \(S\). The assumptions are linearity, independence, normality, and equal variance.

How Added

Get \(\hat{\boldsymbol\beta}_{MLE}\) from the model

Write \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\).
Write \(\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})\). Take the log to get \(\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})\).
Find the term of \(\ell\) with \(\boldsymbol\beta\): \(-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\).
The factor \(-\frac{1}{2\sigma^2}\) is negative. Maximize \(\ell\) by minimizing \(S(\boldsymbol\beta)\).
Use Section 03.2: \(\hat{\boldsymbol\beta}_{MLE}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).

Self-check: (1) A change of \(\sigma^2\) does not move the maximizing \(\boldsymbol\beta\). (2) \(\ell(\hat{\boldsymbol\beta},\sigma^2|\mathbf{Y})\) is larger than \(\ell\) at any other \(\boldsymbol\beta\).

Example Added

Data of Section 03.2 (\(n=4\)). \(\sigma^2\) is unknown. Fix \(\sigma^2=1\), then \(\sigma^2=0.5\). Unit 05 estimates \(\sigma^2\).

Write the three terms of \(\ell\) How step 2

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac{4}{2}\log(2\pi)-\frac{4}{2}\log\sigma^2-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\]

\(\frac n2=2\).

Calculate at \(\boldsymbol\beta=\hat{\boldsymbol\beta}\) and \(\sigma^2=1\) Slides p.22

\[-2\log(2\pi)=-3.675754\]

\[-2\log 1=0\]

\[-\tfrac{1}{2(1)}S(\hat{\boldsymbol\beta})=-\tfrac{0.2}{2}=-0.1\]

\[\ell(\hat{\boldsymbol\beta},1|\mathbf{Y})=-3.675754+0-0.1=-3.775754\]

Move \(\beta_1\) by \(\pm0.1\) and keep \(\beta_0=0.5\) Self-check 2

\[S=0.2+30(0.1)^2=0.5\]

\[\ell=-3.675754+0-\tfrac{0.5}{2}=-3.925754\]

\[-3.925754<-3.775754\]

Both sides; bottom curve of Figure 3-1.

Calculate again with \(\sigma^2=0.5\) Self-check 1

\[-2\log 0.5=1.386294\]

\[\ell(\hat{\boldsymbol\beta},0.5|\mathbf{Y})=-3.675754+1.386294-\tfrac{0.2}{1}=-2.489460\]

\[\beta_1=\hat\beta_1\pm0.1:\ \ell=-3.675754+1.386294-\tfrac{0.5}{1}=-2.789460\]

Self-check: \(\ell\) changes, but its maximum stays at \(\hat{\boldsymbol\beta}\). \(\sigma^2\) changes only the positive factor \(\frac{1}{2\sigma^2}\).

Result: \(\hat{\boldsymbol\beta}_{MLE}=(0.5,\,1.4)^T\), the OLS estimate. ▲

Why Slides p.22

\[\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})=(2\pi)^{-\frac n2}|\sigma^2\boldsymbol{I}|^{-\frac12}\exp\left\{-\frac12(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\sigma^2\boldsymbol{I})^{-1}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\right\}\]

MVN density.

\[|\sigma^2\boldsymbol{I}|=(\sigma^2)^n\]

Diagonal matrix: product of diagonal entries.

\[(\sigma^2\boldsymbol{I})^{-1}=\frac{1}{\sigma^2}\boldsymbol{I}\]

\[\mathcal{L}(\boldsymbol\beta,\sigma^2|\mathbf{Y})=(2\pi)^{-\frac n2}(\sigma^2)^{-\frac n2}\exp\left\{-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\right\}\]

Put in steps 2 and 3.

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=\log\left[(2\pi)^{-\frac n2}\right]+\log\left[(\sigma^2)^{-\frac n2}\right]-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\(\log(abc)=\log a+\log b+\log c\) and \(\log e^{u}=u\).

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\(\log a^{k}=k\log a\). Slides p.22.

\[\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\]

Definition of \(S(\boldsymbol\beta)\): "Look familiar?"

\[\frac{\partial\ell}{\partial\boldsymbol\beta}=-\frac{1}{2\sigma^2}\frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=-\frac{1}{2\sigma^2}\left(-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\right)\]

Section 03.2, Why step 10.

\[-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=0\]

Set to 0; multiply by \(-2\sigma^2\).

\[\hat{\boldsymbol\beta}_{MLE}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}=\hat{\boldsymbol\beta}\]

Section 03.2, Why steps 11–14. ∎

03.4Practice Added

Q1. Use the data \(x=2,4,6\) and \(y=1,3,8\). Calculate \(\boldsymbol{X}^T\boldsymbol{X}\), \(\boldsymbol{X}^T\boldsymbol{y}\), and \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).

Answer

\(\boldsymbol{X}\) has rows \((1,2)\), \((1,4)\), \((1,6)\).

\(\boldsymbol{X}^T\boldsymbol{X}\): entry \((1,1)=3\); entry \((1,2)=2+4+6=12\); entry \((2,2)=4+16+36=56\).

\(\boldsymbol{X}^T\boldsymbol{y}\): entry \(1=1+3+8=12\); entry \(2=2(1)+4(3)+6(8)=2+12+48=62\).

\(\det(\boldsymbol{X}^T\boldsymbol{X})=3(56)-12(12)=168-144=24\).

\(\hat\beta_0=\frac{1}{24}\big(56(12)-12(62)\big)=\frac{1}{24}(672-744)=\frac{-72}{24}=-3\).

\(\hat\beta_1=\frac{1}{24}\big(-12(12)+3(62)\big)=\frac{1}{24}(-144+186)=\frac{42}{24}=1.75\).

Check row 1: \(3(-3)+12(1.75)=-9+21=12\). Check row 2: \(12(-3)+56(1.75)=-36+98=62\). \(\hat{\boldsymbol\beta}=(-3,\,1.75)^T\).

Q2. Intercept-only model \(y_i=\beta_0+\epsilon_i\): \(\boldsymbol{X}\) is an \(n\times1\) column of 1s. Use \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\) to find \(\hat\beta_0\). Calculate it for \(y=2,3,5,6\).

Answer

\(\boldsymbol{X}=(1,\dots,1)^T\).

\(\boldsymbol{X}^T\boldsymbol{X}=\sum_{i=1}^n1\cdot1=n\).

\(\boldsymbol{X}^T\boldsymbol{y}=\sum_{i=1}^ny_i\).

\(\hat\beta_0=n^{-1}\sum_{i=1}^ny_i=\bar y\), the sample mean.

Data: \(n=4\), \(\sum_iy_i=2+3+5+6=16\), \(\hat\beta_0=16/4=4\).

Q3. Use \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\). Why does the \(\boldsymbol\beta\) with the largest \(\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})\) not depend on \(\sigma^2\)? For \(n=4\), \(S(\hat{\boldsymbol\beta})=0.2\), calculate \(\ell(\hat{\boldsymbol\beta},2|\mathbf{Y})\).

Answer

\(\ell(\boldsymbol\beta,\sigma^2|\mathbf{Y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}S(\boldsymbol\beta)\).

Only the third term contains \(\boldsymbol\beta\): the positive \(\frac{1}{2\sigma^2}\) times \(-S(\boldsymbol\beta)\).

For each \(\sigma^2>0\), the largest \(\ell\) occurs at the smallest \(S\), at \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).

Numbers: \(-2\log(2\pi)=-3.675754\), \(-2\log2=-1.386294\), \(-\frac{0.2}{2(2)}=-0.05\).

\(\ell(\hat{\boldsymbol\beta},2|\mathbf{Y})=-3.675754-1.386294-0.05=-5.112048\).

Unit 04Properties of the LS estimator, fitted values, and residuals

Plan · Slides p.23–32

Step 1: Show \(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\) (Slides p.23–25).

Step 2: Find the covariance matrix of \(\hat{\boldsymbol\beta}\) (Slides p.26–27).

Step 3: Write the distribution of \(\hat{\boldsymbol\beta}\) and of each \(\hat\beta_j\) (Slides p.28–29).

Step 4: Define the fitted values and the hat matrix \(\boldsymbol H\) (Slides p.30).

Step 5: Define the residuals and show \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\) (Slides p.31–32).

Facts from earlier units:

Model (Slides p.7, p.13): \(\boldsymbol y=\boldsymbol X\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol 0,\sigma^2\boldsymbol I)\). Equivalently, \(\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\).

\(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\) and \(\mathrm{Var}[\boldsymbol y]=\sigma^2\boldsymbol I\).

\(\boldsymbol X\) is a known \(n\times(p+1)\) matrix of constants.

For a constant matrix \(\boldsymbol C\) (Slides p.4): \(E[\boldsymbol C\boldsymbol y]=\boldsymbol C\,E[\boldsymbol y]\) and \(\mathrm{Var}[\boldsymbol C\boldsymbol y]=\boldsymbol C\,\mathrm{Var}[\boldsymbol y]\,\boldsymbol C^T\).

\(\mathrm{Var}[\cdot]\) and \(\mathrm{Var}(\cdot)\) have the same meaning.

Least squares estimator (Unit 03, Slides p.20): \(\hat{\boldsymbol\beta}=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\).

All examples use this data set Added: \(n=4\), \(p=1\), row \(i\) of \(\boldsymbol X\) is \((1,\ x_i)\).

\[x=(0,\ 1,\ 2,\ 3),\qquad \boldsymbol y=(1,\ 3,\ 2,\ 5)^T,\qquad \boldsymbol X=\left[\begin{array}{cc}1&0\\1&1\\1&2\\1&3\end{array}\right]\]

\[\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}1+1+1+1&0+1+2+3\\0+1+2+3&0+1+4+9\end{array}\right]=\left[\begin{array}{cc}4&6\\6&14\end{array}\right]\]

Entry \((j,k)\): column \(j\) dot column \(k\).

\[\det(\boldsymbol X^T\boldsymbol X)=4\cdot14-6\cdot6=56-36=20\]

\[(\boldsymbol X^T\boldsymbol X)^{-1}=\frac{1}{20}\left[\begin{array}{cc}14&-6\\-6&4\end{array}\right]=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\]

\(2\times2\) inverse: swap diagonal, negate off-diagonal, divide.

\[\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}1+3+2+5\\0\cdot1+1\cdot3+2\cdot2+3\cdot5\end{array}\right]=\left[\begin{array}{c}11\\22\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}0.7\cdot11-0.3\cdot22\\-0.3\cdot11+0.2\cdot22\end{array}\right]=\left[\begin{array}{c}7.7-6.6\\-3.3+4.4\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}1.1\\1.1\end{array}\right]\]

04.1The mean of \(\hat{\boldsymbol\beta}\): unbiasedness Slides p.23–25

What Slides p.25

Properties of \(\hat{\boldsymbol\beta}\) · Lecture 6 · p.25 \[E[\hat{\boldsymbol\beta}]=E[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]=\boldsymbol\beta\]

\(\hat{\boldsymbol\beta}\) comes from the random \(\boldsymbol y\). Each new data set gives a different \(\hat{\boldsymbol\beta}\). Its mean over all data sets is exactly \(\boldsymbol\beta\). Unbiased: the mean of the estimator equals the parameter.

How Added

Test if an estimator \(\boldsymbol C\boldsymbol y\) is unbiased

Write the estimator as \(\boldsymbol C\boldsymbol y\), with \(\boldsymbol C\) built only from \(\boldsymbol X\) and numbers.
For \(\hat{\boldsymbol\beta}\), \(\boldsymbol C=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\).
Move \(E\) inside: \(E[\boldsymbol C\boldsymbol y]=\boldsymbol C\,E[\boldsymbol y]\).
Put in \(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\) to get \(\boldsymbol C\boldsymbol X\boldsymbol\beta\).
Calculate \(\boldsymbol C\boldsymbol X\). If \(\boldsymbol C\boldsymbol X=\boldsymbol I\), then \(E[\boldsymbol C\boldsymbol y]=\boldsymbol\beta\).

Self-check: \(\boldsymbol C\) is \((p+1)\times n\). \(\boldsymbol C\boldsymbol X\) is \((p+1)\times(p+1)\), with 1 on the diagonal and 0 elsewhere.

Example Added

Steps 2 and 5 for the small data set:

\[\boldsymbol C=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\left[\begin{array}{cccc}1&1&1&1\\0&1&2&3\end{array}\right]\]

\[\boldsymbol C=\left[\begin{array}{cccc}0.7&0.7-0.3&0.7-0.6&0.7-0.9\\-0.3&-0.3+0.2&-0.3+0.4&-0.3+0.6\end{array}\right]\]

\[\boldsymbol C=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\-0.3&-0.1&0.1&0.3\end{array}\right]\]

\[\boldsymbol C\boldsymbol X=\left[\begin{array}{cc}0.7+0.4+0.1-0.2&0+0.4+0.2-0.6\\-0.3-0.1+0.1+0.3&0-0.1+0.2+0.9\end{array}\right]\]

\[\boldsymbol C\boldsymbol X=\left[\begin{array}{cc}1&0\\0&1\end{array}\right]=\boldsymbol I\]

\(E[\hat{\boldsymbol\beta}]=\boldsymbol I\boldsymbol\beta=\boldsymbol\beta\). Check: row 1 of \(\boldsymbol C\boldsymbol y\) is \(0.7\cdot1+0.4\cdot3+0.1\cdot2-0.2\cdot5=0.7+1.2+0.2-1.0=1.1\). ▲

Why Slides p.25

\[E[\hat{\boldsymbol\beta}]=E[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]\]

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T E[\boldsymbol y]\]

\(E[\boldsymbol C\boldsymbol y]=\boldsymbol C\,E[\boldsymbol y]\).

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol X\boldsymbol\beta)\]

Model: \(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\).

\[=(\boldsymbol X^T\boldsymbol X)^{-1}(\boldsymbol X^T\boldsymbol X)\boldsymbol\beta\]

\[=\boldsymbol\beta\]

∎

04.2The covariance matrix of \(\hat{\boldsymbol\beta}\) Slides p.26–27

What Slides p.27

Properties of \(\hat{\boldsymbol\beta}\) · Lecture 6 · p.27 \[\mathrm{Var}[\hat{\boldsymbol\beta}]=\mathrm{Var}[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\]

\(\mathrm{Var}[\hat{\boldsymbol\beta}]\) is \((p+1)\times(p+1)\). Its diagonal entry for \(\beta_j\) is \(\mathrm{Var}[\hat\beta_j]\). \(\sigma^2\) is unknown. \((\boldsymbol X^T\boldsymbol X)^{-1}\) is known.

The slides write \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\) (p.29). Rows and columns follow the order \(\beta_0,\beta_1,\dots,\beta_p\). \(V_{jj}\) is the diagonal entry for \(\beta_j\).

How Added

Find \(\mathrm{Var}[\hat\beta_j]\)

Make \(\boldsymbol X\), with 1 in each entry of column 1.
Calculate \(\boldsymbol X^T\boldsymbol X\).
Find \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\).
Take \(V_{jj}\). The first diagonal entry is for \(\beta_0\).
Write \(\mathrm{Var}[\hat\beta_j]=\sigma^2V_{jj}\).

Self-check: \(\boldsymbol V\) is \((p+1)\times(p+1)\) and symmetric. Each \(V_{jj}\) is larger than 0.

Example Added

Step 1–3: \(\boldsymbol V=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\), \(2\times2\) because \(p+1=2\).

Step 4: \(V_{00}=0.7\) (for \(\beta_0\)) and \(V_{11}=0.2\) (for \(\beta_1\)).

Step 5: \(\mathrm{Var}[\hat\beta_0]=0.7\,\sigma^2\) (intercept).

Step 5: \(\mathrm{Var}[\hat\beta_1]=0.2\,\sigma^2\) (slope).

\(\sigma^2\) is still unknown. Unit 05 estimates it and calculates the standard error. ▲

Why Slides p.27

The slides do lines 3 and 4 in one step Added.

\[\mathrm{Var}[\hat{\boldsymbol\beta}]=\mathrm{Var}[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y]\]

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\,\mathrm{Var}[\boldsymbol y]\,((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T)^T\]

\(\mathrm{Var}[\boldsymbol C\boldsymbol y]=\boldsymbol C\,\mathrm{Var}[\boldsymbol y]\,\boldsymbol C^T\) (Slides p.4).

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\,\mathrm{Var}[\boldsymbol y]\,(\boldsymbol X^T)^T((\boldsymbol X^T\boldsymbol X)^{-1})^T\]

Added: \((\boldsymbol A\boldsymbol B)^T=\boldsymbol B^T\boldsymbol A^T\).

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\,\mathrm{Var}[\boldsymbol y]\,\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

Symmetric \(\boldsymbol X^T\boldsymbol X\) has a symmetric inverse.

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\sigma^2\boldsymbol I)\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

Model: \(\mathrm{Var}[\boldsymbol y]=\sigma^2\boldsymbol I\).

\[=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

\[=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\]

∎

04.3The distribution of \(\hat{\boldsymbol\beta}\) Slides p.28–29

What Slides p.29

Properties of \(\hat{\boldsymbol\beta}\) · Lecture 6 · p.29 And \(\hat{\boldsymbol\beta}\) is MVN-distributed. Why? \[\Longrightarrow\ \hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\ \sigma^2(\boldsymbol X^T\boldsymbol X)^{-1})\] From property 2 of MVN distributions, this gives: \[\hat\beta_j\sim N(\beta_j,\ \sigma^2V_{jj})\] where \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\).

Line 1 combines the mean (04.1) and the covariance matrix (04.2). "Property 2" is the marginal property (Slides p.6). It gives one component \(\hat\beta_j\) a normal distribution.

How Added

Write the distribution of one \(\hat\beta_j\)

Find \(j\): the intercept has \(j=0\), covariate \(k\) has \(j=k\).
Find \(\boldsymbol V\) (How box of 04.2).
Take \(V_{jj}\).
Write \(\hat\beta_j\sim N(\beta_j,\ \sigma^2V_{jj})\). The mean is the parameter \(\beta_j\), not a number.

Self-check: The variance is \(\sigma^2\) times a positive number.

Example Added

Slope estimator \(\hat\beta_1\), small data set:

\[j=1\]

\[\boldsymbol V=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\]

\[V_{11}=0.2\]

\[\hat\beta_1\sim N(\beta_1,\ 0.2\,\sigma^2)\]

The value \(\hat\beta_1=1.1\) is one draw from this distribution. ▲

Why Added

Added · An answer to the "Why?" on p.29

The slides give no reason. The reason is the linearity property (Slides p.6):

Lecture 6 · p.6 Linearity: If \(\boldsymbol u=\boldsymbol C\boldsymbol y+\boldsymbol d\), then: \[\boldsymbol u\sim MVN(\boldsymbol C\boldsymbol\mu+\boldsymbol d,\ \boldsymbol C\boldsymbol\Sigma\boldsymbol C^T)\]

with \(\boldsymbol y\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\).

\[\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\ \sigma^2\boldsymbol I)\]

Model (p.7).

\[\hat{\boldsymbol\beta}=\boldsymbol C\boldsymbol y+\boldsymbol 0,\qquad \boldsymbol C=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

\(\boldsymbol C\) is constant; \(\boldsymbol d=\boldsymbol 0\).

\[\hat{\boldsymbol\beta}\sim MVN(\boldsymbol C\boldsymbol X\boldsymbol\beta,\ \boldsymbol C(\sigma^2\boldsymbol I)\boldsymbol C^T)\]

Linearity.

\[\boldsymbol C\boldsymbol X\boldsymbol\beta=\boldsymbol\beta\]

Why of 04.1, lines 3–5.

\[\boldsymbol C(\sigma^2\boldsymbol I)\boldsymbol C^T=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\]

Why of 04.2, lines 2–7.

\[\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\ \sigma^2(\boldsymbol X^T\boldsymbol X)^{-1})\]

Line 1 of p.29. ∎

Line 2 of p.29: the marginal property (p.6) takes entry \(j\) of the mean and diagonal entry \(j\) of \(\sigma^2\boldsymbol V\).

04.4Fitted values and the hat matrix \(\boldsymbol H\) Slides p.30

What Slides p.30

Fitted values · Lecture 6 · p.30 \[\hat{\boldsymbol y}=\boldsymbol X\hat{\boldsymbol\beta}=\boldsymbol H\boldsymbol y\] where \(\boldsymbol H=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\) is called the Hat matrix.

Fitted value: the estimated regression equation at the covariates of observation \(i\), \(\hat y_i=\hat\beta_0+\hat\beta_1x_{i1}+\cdots+\hat\beta_px_{ip}\). Hat matrix \(\boldsymbol H\): the \(n\times n\) matrix that changes \(\boldsymbol y\) into \(\hat{\boldsymbol y}\). \(\boldsymbol H\) depends only on \(\boldsymbol X\).

How Added

Calculate the fitted values

Calculate \(\hat{\boldsymbol\beta}\).
Take row \(i\) of \(\boldsymbol X\), \((1,x_{i1},\dots,x_{ip})\).
Multiply it with \(\hat{\boldsymbol\beta}\) term by term and add, to get \(\hat y_i\).
Do Steps 2 and 3 for each \(i=1,\dots,n\).
Alternative: calculate \(\boldsymbol H=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\), then \(\boldsymbol H\boldsymbol y\).

Self-check: \(\boldsymbol H\) is \(n\times n\) and symmetric. Steps 4 and 5 give the same \(\hat{\boldsymbol y}\).

Example Added

Step 1: \(\hat\beta_0=1.1\), \(\hat\beta_1=1.1\).

Steps 2–3, \(i=1\): \(\hat y_1=1.1\cdot1+1.1\cdot0=1.1+0=1.1\).

Steps 2–3, \(i=2\): \(\hat y_2=1.1\cdot1+1.1\cdot1=1.1+1.1=2.2\).

Steps 2–3, \(i=3\): \(\hat y_3=1.1\cdot1+1.1\cdot2=1.1+2.2=3.3\).

Steps 2–3, \(i=4\): \(\hat y_4=1.1\cdot1+1.1\cdot3=1.1+3.3=4.4\).

Step 5: \(\boldsymbol H=\boldsymbol X\boldsymbol C\), with \(\boldsymbol C\) from 04.1. Row \(i\) of \(\boldsymbol H\) is \((\text{row 1 of }\boldsymbol C)+x_i\cdot(\text{row 2 of }\boldsymbol C)\).

\[\boldsymbol H=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\0.7-0.3&0.4-0.1&0.1+0.1&-0.2+0.3\\0.7-0.6&0.4-0.2&0.1+0.2&-0.2+0.6\\0.7-0.9&0.4-0.3&0.1+0.3&-0.2+0.9\end{array}\right]\]

\(x_i=0,1,2,3\).

\[\boldsymbol H=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\0.4&0.3&0.2&0.1\\0.1&0.2&0.3&0.4\\-0.2&0.1&0.4&0.7\end{array}\right]\]

\[\boldsymbol H\boldsymbol y=\left[\begin{array}{c}0.7\cdot1+0.4\cdot3+0.1\cdot2-0.2\cdot5\\0.4\cdot1+0.3\cdot3+0.2\cdot2+0.1\cdot5\\0.1\cdot1+0.2\cdot3+0.3\cdot2+0.4\cdot5\\-0.2\cdot1+0.1\cdot3+0.4\cdot2+0.7\cdot5\end{array}\right]\]

\[=\left[\begin{array}{c}0.7+1.2+0.2-1.0\\0.4+0.9+0.4+0.5\\0.1+0.6+0.6+2.0\\-0.2+0.3+0.8+3.5\end{array}\right]\]

\[=\left[\begin{array}{c}1.1\\2.2\\3.3\\4.4\end{array}\right]\]

Same as Steps 2–4. ▲

Why Slides p.30

\[\hat{\boldsymbol y}=\boldsymbol X\hat{\boldsymbol\beta}\]

\[=\boldsymbol X\left[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\right]\]

\[=\left[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\right]\boldsymbol y\]

Matrix multiplication is associative.

\[=\boldsymbol H\boldsymbol y\]

∎

04.5Residuals and \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\) Slides p.31–32

What Slides p.32

Residuals · Lecture 6 · p.32 \[\boldsymbol e=\boldsymbol y-\hat{\boldsymbol y}=(\boldsymbol I-\boldsymbol H)\boldsymbol y.\] Note: \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). Why?

Residual: observed value minus fitted value, \(e_i=y_i-\hat y_i\). \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\) gives one equation for each column of \(\boldsymbol X\):

Column of ones: \(\sum_{i=1}^n e_i=0\).

Column of covariate \(k\): \(\sum_{i=1}^n x_{ik}e_i=0\).

Orthogonal: two vectors with dot product 0 (sum of products of matching entries). \(\boldsymbol e\) is orthogonal to each column of \(\boldsymbol X\).

How Added

Calculate the residuals and check \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\)

Calculate \(\hat{\boldsymbol y}\) (How box of 04.4).
Calculate \(e_i=y_i-\hat y_i\) for each \(i\).
Calculate \(\sum e_i\).
For each covariate \(k\), calculate \(\sum x_{ik}e_i\).

Self-check: With exact arithmetic, each of the \(p+1\) entries is exactly 0.

Example Added

Step 1: \(\hat{\boldsymbol y}=(1.1,\ 2.2,\ 3.3,\ 4.4)^T\).

Step 2: \(e_1=1-1.1=-0.1\).

Step 2: \(e_2=3-2.2=0.8\).

Step 2: \(e_3=2-3.3=-1.3\).

Step 2: \(e_4=5-4.4=0.6\).

Step 3: \(\sum e_i=-0.1+0.8-1.3+0.6=0\).

Step 4: \(\sum x_ie_i=0\cdot(-0.1)+1\cdot0.8+2\cdot(-1.3)+3\cdot0.6=0+0.8-2.6+1.8=0\). ▲

Figure 4-1. Small data set (\(n=4\)): points \((x_i,y_i)\) in green, fitted line \(\hat y=1.1+1.1x\) in blue, residuals in orange. Segment length is \(|e_i|\); points above the line have \(e_i>0\). The residuals add to 0.

Why Slides p.31–32

Part 1 is Slides p.31. Part 2 answers the "Why?" on p.32. The slides do lines 2 and 3 of Part 2 in one step Added.

\[\boldsymbol e=\boldsymbol y-\hat{\boldsymbol y}\]

\[=\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta}\]

\[=\boldsymbol y-\boldsymbol H\boldsymbol y\]

04.4.

\[=(\boldsymbol I-\boldsymbol H)\boldsymbol y\]

∎

\[\boldsymbol X^T\boldsymbol e=\boldsymbol X^T(\boldsymbol y-\boldsymbol H\boldsymbol y)\]

Line 3 of Part 1.

\[=\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\boldsymbol H\boldsymbol y\]

Added.

\[=\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\]

\[=\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\boldsymbol y\]

\[=\boldsymbol 0\]

∎

Summary of this unit

\(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\). \(\mathrm{Var}[\hat{\boldsymbol\beta}]=\sigma^2\boldsymbol V\). \(\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\). \(\hat{\boldsymbol y}=\boldsymbol H\boldsymbol y\). \(\boldsymbol e=(\boldsymbol I-\boldsymbol H)\boldsymbol y\), \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). Unit 05 estimates \(\sigma^2\).

04.6Practice Added

Q1. Data: \(n=3\), \(x=(1,\ 2,\ 3)\), model \(y_i=\beta_0+\beta_1x_i+\epsilon_i\). Find \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\). Write the distribution of \(\hat\beta_1\) in terms of \(\sigma^2\).

Answer

\(\boldsymbol X\) has rows \((1,1)\), \((1,2)\), \((1,3)\).

\(\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}1+1+1&1+2+3\\1+2+3&1+4+9\end{array}\right]=\left[\begin{array}{cc}3&6\\6&14\end{array}\right]\).

\(\det(\boldsymbol X^T\boldsymbol X)=3\cdot14-6\cdot6=42-36=6\).

\(\boldsymbol V=\frac16\left[\begin{array}{cc}14&-6\\-6&3\end{array}\right]=\left[\begin{array}{cc}7/3&-1\\-1&0.5\end{array}\right]\).

\(V_{11}=0.5\). By p.29, \(\hat\beta_1\sim N(\beta_1,\ 0.5\,\sigma^2)\).

Q2. Use the data of Q1 with \(\boldsymbol y=(2,\ 2,\ 5)^T\). Calculate \(\hat{\boldsymbol\beta}\), the fitted values, and the residuals. Check \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\).

Answer

\(\boldsymbol X^T\boldsymbol y=(2+2+5,\ 1\cdot2+2\cdot2+3\cdot5)^T=(9,\ 21)^T\).

\(\hat\beta_0=\frac16(14\cdot9-6\cdot21)=\frac16(126-126)=0\).

\(\hat\beta_1=\frac16(-6\cdot9+3\cdot21)=\frac16(-54+63)=\frac96=1.5\).

\(\hat y_1=0+1.5\cdot1=1.5\). \(\hat y_2=0+1.5\cdot2=3\). \(\hat y_3=0+1.5\cdot3=4.5\).

\(e_1=2-1.5=0.5\). \(e_2=2-3=-1\). \(e_3=5-4.5=0.5\).

\(\sum e_i=0.5-1+0.5=0\).

\(\sum x_ie_i=1\cdot0.5+2\cdot(-1)+3\cdot0.5=0.5-2+1.5=0\).

Q3. Show \(\boldsymbol X^T\hat{\boldsymbol y}=\boldsymbol X^T\boldsymbol y\). Check row 1 with the small data set.

Answer

\[\boldsymbol X^T\boldsymbol e=\boldsymbol 0\]

p.32.

\[\boldsymbol X^T(\boldsymbol y-\hat{\boldsymbol y})=\boldsymbol 0\]

\[\boldsymbol X^T\boldsymbol y-\boldsymbol X^T\hat{\boldsymbol y}=\boldsymbol 0\]

\[\boldsymbol X^T\hat{\boldsymbol y}=\boldsymbol X^T\boldsymbol y\]

∎

Row 1 (column of ones): \(\sum\hat y_i=\sum y_i\). \(\sum\hat y_i=1.1+2.2+3.3+4.4=11\). \(\sum y_i=1+3+2+5=11\).

Unit 05Estimating σ², calculating the estimates, and practice

Plan · Slides p.33–41

Step 1: Find the MLE \(\hat\sigma^2_{MLE}=\frac1n\boldsymbol{e}^T\boldsymbol{e}\) (Slides p.33–35).

Step 2: Change the divisor to \(n-(p+1)\) to get the unbiased \(\hat\sigma^2\) (Slides p.36).

Step 3: Calculate \(\hat{\boldsymbol\beta}\) from a system of linear equations (Slides p.37).

Step 4: Calculate the fitted values, residuals, \(\hat\sigma^2\), and standard errors (Slides p.38).

Step 5: Do the two practice questions for simple linear regression (Slides p.39–41).

Model (Slides p.13): \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\). \(\boldsymbol{X}\) is \(n\times(p+1)\). \(p\) is the number of covariates.

The error variance \(\sigma^2\) is the variance of each \(\epsilon_i\). It is the last unknown parameter.

Least squares estimate (Unit 03): \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\). It is also the MLE.

Residuals (Unit 04): \(\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}\). The residual sum of squares is \(\boldsymbol{e}^T\boldsymbol{e}=\sum_{i=1}^n e_i^2\). Slides p.32: \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\).

The small data set of this unit Added

The slides do not print the fit for the satisfaction data (\(n=46\), \(p=3\)). This unit uses a small data set that you can do by hand.

\(n=5\), \(p=1\). Covariate: \(x=(1,\,2,\,3,\,4,\,5)\). Outcome: \(y=(2,\,4,\,5,\,4,\,5)\).

\[\boldsymbol{X}=\left[\begin{array}{cc}1&1\\1&2\\1&3\\1&4\\1&5\end{array}\right],\qquad \boldsymbol{y}=\left[\begin{array}{c}2\\4\\5\\4\\5\end{array}\right]\]

Section 5.3 calculates \(\hat{\boldsymbol\beta}=(2.2,\,0.6)^T\) from these data.

05.1MLE of \(\sigma^2\) Slides p.33–35

What Slides p.35

MLE of σ² · Lecture 6 · p.35 \[\ell(\boldsymbol\beta,\sigma^2\mid\boldsymbol{y})=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\] \[\frac{\partial\ell(\boldsymbol\beta,\sigma^2\mid\boldsymbol{y})}{\partial\sigma^2}=-\frac{n}{2\sigma^2}+\frac{1}{2(\sigma^2)^2}(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)=0\] \[(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)=n\sigma^2\] \[\Longrightarrow\ \hat\sigma^2_{MLE}=\frac1n(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})^T(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})\]

\(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{e}\), and \(\hat\sigma^2_{MLE}=\frac1n\boldsymbol{e}^T\boldsymbol{e}\): the mean of the squared residuals.

\(\partial\ell/\partial\sigma^2\) holds \(\boldsymbol\beta\) constant. Slides p.34 show the same page without the derivative.

How Added

Calculate \(\hat\sigma^2_{MLE}\)

Calculate the fitted values \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\).
Calculate the residuals: \(e_i=y_i-\hat y_i\).
Add the squared residuals to get \(\boldsymbol{e}^T\boldsymbol{e}\).
Divide by \(n\).

Self-check: \(\boldsymbol{e}^T\boldsymbol{e}\ge0\) and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) (Slides p.32). If not, \(\hat{\boldsymbol\beta}\) is wrong.

Example Added

Use \(\hat y=2.2+0.6x\) from Section 05.3.

Fitted values How step 1

\[\hat y_1=2.2+0.6(1)=2.8,\quad \hat y_2=2.2+0.6(2)=3.4,\quad \hat y_3=2.2+0.6(3)=4.0\]

\[\hat y_4=2.2+0.6(4)=4.6,\quad \hat y_5=2.2+0.6(5)=5.2\]

Residuals How step 2

\[e_1=2-2.8=-0.8,\quad e_2=4-3.4=0.6,\quad e_3=5-4.0=1.0\]

\[e_4=4-4.6=-0.6,\quad e_5=5-5.2=-0.2\]

Residual sum of squares How step 3

\[\boldsymbol{e}^T\boldsymbol{e}=(-0.8)^2+0.6^2+1.0^2+(-0.6)^2+(-0.2)^2\]

\[\boldsymbol{e}^T\boldsymbol{e}=0.64+0.36+1.00+0.36+0.04=2.4\]

Divide by \(n\) How step 4

\[\hat\sigma^2_{MLE}=\frac{2.4}{5}=0.48\]

Check: \(\sum e_i=-0.8+0.6+1.0-0.6-0.2=0\), \(\sum x_ie_i=-0.8+1.2+3.0-2.4-1.0=0\).

Figure 5-1. Data and fitted line \(\hat y=2.2+0.6x\) (green). Each vertical segment is a residual \(e_i\). The squared lengths add to \(\boldsymbol{e}^T\boldsymbol{e}=2.4\). Purple circle: \((\bar x,\bar y)\).

Why Slides p.35

Let \(Q=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\). \(Q\) does not contain \(\sigma^2\).

\[\ell=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{1}{2\sigma^2}Q\]

Line 1 of p.35.

\[\ell=-\frac n2\log(2\pi)-\frac n2\log\sigma^2-\frac{Q}{2}(\sigma^2)^{-1}\]

\[\frac{\partial\ell}{\partial\sigma^2}=0-\frac n2\cdot\frac{1}{\sigma^2}-\frac Q2\cdot\left(-(\sigma^2)^{-2}\right)\]

With \(u=\sigma^2\): \(\frac{d}{du}\log u=\frac1u\), \(\frac{d}{du}u^{-1}=-u^{-2}\).

\[\frac{\partial\ell}{\partial\sigma^2}=-\frac{n}{2\sigma^2}+\frac{Q}{2(\sigma^2)^2}\]

Line 2 of p.35.

\[-\frac{n}{2\sigma^2}+\frac{Q}{2(\sigma^2)^2}=0\]

\[-n\sigma^2+Q=0\]

Multiply by \(2(\sigma^2)^2\).

\[Q=n\sigma^2\]

Line 3 of p.35.

\[\sigma^2=\frac1n Q=\frac1n(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

\[\hat\sigma^2_{MLE}=\frac1n(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})^T(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})=\frac1n\boldsymbol{e}^T\boldsymbol{e}\]

\(\hat{\boldsymbol\beta}\) minimizes \(Q\) for every \(\sigma^2\). ∎

05.2Unbiased estimate of \(\sigma^2\) Slides p.36

What Slides p.36

The bias of an estimator \(\hat\theta\) is \(E[\hat\theta]-\theta\).

Unbiased estimate of σ² · Lecture 6 · p.36 The MLE of \(\sigma^2\) is \(\frac1n\boldsymbol{e}^T\boldsymbol{e}\), where \(\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}\).
Instead we usually use \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol{e}^T\boldsymbol{e}\). Why?

Only the divisor changes. \(n-(p+1)=n-p-1\) is the residual degrees of freedom: the number of residuals that can change freely. On this page, \(\hat\sigma^2\) is this unbiased estimate.

How Added

Calculate \(\hat\sigma^2\)

Calculate \(\boldsymbol{e}^T\boldsymbol{e}\) (Section 05.1, steps 1–3).
Count the coefficients: \(p+1\), with the intercept.
Calculate the degrees of freedom: \(n-p-1\).
Divide: \(\hat\sigma^2=\boldsymbol{e}^T\boldsymbol{e}/(n-p-1)\).

Self-check: \(\hat\sigma^2/\hat\sigma^2_{MLE}=n/(n-p-1)>1\).

Example Added

Residual sum of squares How step 1

\[\boldsymbol{e}^T\boldsymbol{e}=2.4\]

Section 05.1, step 3.

Count the coefficients How step 2

\[p=1,\qquad p+1=2\]

Degrees of freedom How step 3

\[n-p-1=5-1-1=3\]

Divide How step 4

\[\hat\sigma^2=\frac{2.4}{3}=0.8\]

Check: \(0.8/0.48=5/3=n/(n-p-1)\). \(\hat\sigma=\sqrt{0.8}=0.8944\).

Why Added

Slides p.36 give no proof.

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\]

p.32. \(\boldsymbol{X}^T\) has \(p+1\) rows.

\[\sum_{i=1}^n x_{ij}e_i=0,\qquad j=0,1,\dots,p\]

\(p+1\) linear constraints on \(e_1,\dots,e_n\).

\[\text{number of free residuals}=n-(p+1)\]

The constraints fix the other \(p+1\) residuals.

\[\hat\sigma^2=\frac{\boldsymbol{e}^T\boldsymbol{e}}{n-(p+1)}\]

Gives \(E[\hat\sigma^2]=\sigma^2\) (proof below). ∎

In the example, \(\sum e_i=0\) and \(\sum x_ie_i=0\) fix \(e_4\) and \(e_5\) from \(e_1,e_2,e_3\). Only \(5-2=3\) residuals are free.

Optional — not on the slides · Proof that \(E[\boldsymbol{e}^T\boldsymbol{e}]=(n-p-1)\sigma^2\)

The trace \(\mathrm{tr}(\boldsymbol{A})\) is the sum of the diagonal entries of a square matrix. \(\mathrm{tr}(\boldsymbol{AB})=\mathrm{tr}(\boldsymbol{BA})\). Use \(\boldsymbol{H}=\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\) (p.30) and \(\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}\) (p.31). \(\boldsymbol{I}-\boldsymbol{H}\) is symmetric and idempotent: it times itself gives itself.

\[\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}=(\boldsymbol{I}-\boldsymbol{H})(\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon)\]

\[\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\]

\(\boldsymbol{H}\boldsymbol{X}=\boldsymbol{X}\) gives \((\boldsymbol{I}-\boldsymbol{H})\boldsymbol{X}=\boldsymbol{0}\).

\[\boldsymbol{e}^T\boldsymbol{e}=\boldsymbol\epsilon^T(\boldsymbol{I}-\boldsymbol{H})^T(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon=\boldsymbol\epsilon^T(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\]

Symmetric and idempotent.

\[\boldsymbol{e}^T\boldsymbol{e}=\mathrm{tr}\!\left(\boldsymbol\epsilon^T(\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\right)=\mathrm{tr}\!\left((\boldsymbol{I}-\boldsymbol{H})\boldsymbol\epsilon\boldsymbol\epsilon^T\right)\]

A number is its own trace. Then \(\mathrm{tr}(\boldsymbol{AB})=\mathrm{tr}(\boldsymbol{BA})\).

\[E[\boldsymbol{e}^T\boldsymbol{e}]=\mathrm{tr}\!\left((\boldsymbol{I}-\boldsymbol{H})E[\boldsymbol\epsilon\boldsymbol\epsilon^T]\right)=\mathrm{tr}\!\left((\boldsymbol{I}-\boldsymbol{H})\sigma^2\boldsymbol{I}\right)=\sigma^2\,\mathrm{tr}(\boldsymbol{I}-\boldsymbol{H})\]

\(E[\boldsymbol\epsilon\boldsymbol\epsilon^T]=\mathrm{Var}[\boldsymbol\epsilon]=\sigma^2\boldsymbol{I}\).

\[\mathrm{tr}(\boldsymbol{H})=\mathrm{tr}\!\left(\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\right)=\mathrm{tr}\!\left((\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{X}\right)=\mathrm{tr}(\boldsymbol{I}_{p+1})=p+1\]

\[E[\boldsymbol{e}^T\boldsymbol{e}]=\sigma^2(n-(p+1))\quad\Longrightarrow\quad E\!\left[\tfrac{1}{n-(p+1)}\boldsymbol{e}^T\boldsymbol{e}\right]=\sigma^2\]

\(E[\hat\sigma^2_{MLE}]=\frac{n-p-1}{n}\sigma^2\): too small. ∎

05.3Calculating \(\hat{\boldsymbol\beta}\) as a system of equations Slides p.37

What Slides p.37

The code on Slides p.37 does these steps:

It makes \(\boldsymbol{X}\): a column of ones, then one column for each covariate.

It calculates \(\boldsymbol{X}^T\boldsymbol{X}\) and \(\boldsymbol{X}^T\boldsymbol{y}\).

It solves the system \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\) for \(\hat{\boldsymbol\beta}\).

The slide marks as worse: calculate \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\), then multiply by \(\boldsymbol{X}^T\boldsymbol{y}\). Both give the same \(\hat{\boldsymbol\beta}\).

In a system of linear equations, each unknown has only the first power.

How Added

Calculate \(\hat{\boldsymbol\beta}\) from the system \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\)

Make \(\boldsymbol{X}\) and \(\boldsymbol{y}\).
Calculate \(\boldsymbol{X}^T\boldsymbol{X}\). Entry \((j,k)\) is \(\sum_i\) (column \(j\) entry)(column \(k\) entry).
Calculate \(\boldsymbol{X}^T\boldsymbol{y}\). Entry \(j\) is \(\sum_i\) (column \(j\) entry)\(y_i\).
Write the \(p+1\) equations.
Solve by elimination.

Self-check: \(\boldsymbol{X}^T\boldsymbol{X}\) is symmetric, \((p+1)\times(p+1)\), with top-left entry \(n\). \(\hat{\boldsymbol\beta}\) satisfies each equation.

Example Added

Sums for \(\boldsymbol{X}^T\boldsymbol{X}\) How step 2

\[n=5,\quad \sum x_i=1+2+3+4+5=15,\quad \sum x_i^2=1+4+9+16+25=55\]

\[\boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}5&15\\15&55\end{array}\right]\]

Sums for \(\boldsymbol{X}^T\boldsymbol{y}\) How step 3

\[\sum y_i=2+4+5+4+5=20,\quad \sum x_iy_i=2+8+15+16+25=66\]

\[\boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}20\\66\end{array}\right]\]

The system How step 4

\[5\hat\beta_0+15\hat\beta_1=20\]

\[15\hat\beta_0+55\hat\beta_1=66\]

Elimination How step 5

\[15\hat\beta_0+45\hat\beta_1=60\]

First equation times 3.

\[(55-45)\hat\beta_1=66-60\]

Second equation minus step 4.

\[10\hat\beta_1=6\quad\Longrightarrow\quad \hat\beta_1=0.6\]

\[5\hat\beta_0+15(0.6)=20\quad\Longrightarrow\quad 5\hat\beta_0=20-9=11\quad\Longrightarrow\quad\hat\beta_0=2.2\]

Self-check

\[15(2.2)+55(0.6)=33+33=66\]

\(\hat{\boldsymbol\beta}=(2.2,\,0.6)^T\). ▲

Why Added

\[\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

Slides p.20.

\[(\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

\[(\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\]

A system \(\boldsymbol{A}\boldsymbol{b}=\boldsymbol{c}\). ∎

05.4Calculating \(\hat\sigma^2\) and the standard errors Slides p.38

What Slides p.38

The code on Slides p.38 does these steps:

\(n\) = number of rows of \(\boldsymbol{X}\). \(p\) = number of columns minus 1.

It calculates \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\) and \(\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}\).

It calculates \(\hat\sigma^2=\sum e_i^2/(n-p-1)\).

It calculates \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\) and the standard errors \(\hat\sigma\sqrt{V_{jj}}\).

The standard error (SE) of an estimator estimates its standard deviation, with \(\hat\sigma\) in place of \(\sigma\). Slides p.27–29: \(\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1})\) and \(\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\), \(j=0,1,\dots,p\).

How Added

Calculate \(\mathrm{SE}(\hat\beta_j)\)

Calculate \(\hat\sigma^2\) (Section 05.2).
Calculate \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\). Read \(V_{jj}\).
Multiply: \(\hat\sigma^2V_{jj}\) estimates \(\mathrm{Var}[\hat\beta_j]\).
Take the square root: \(\mathrm{SE}(\hat\beta_j)=\hat\sigma\sqrt{V_{jj}}\).

Self-check: Each \(V_{jj}>0\). \(\boldsymbol{V}\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{I}\).

Example Added

Use \(\boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}5&15\\15&55\end{array}\right]\) (Section 5.3) and the \(2\times2\) inverse:

\[\left[\begin{array}{cc}a&b\\c&d\end{array}\right]^{-1}=\frac{1}{ad-bc}\left[\begin{array}{cc}d&-b\\-c&a\end{array}\right]\]

\(\hat\sigma^2\) How step 1

\[\hat\sigma^2=\frac{2.4}{5-1-1}=\frac{2.4}{3}=0.8\]

Determinant How step 2

\[ad-bc=5(55)-15(15)=275-225=50\]

\(\boldsymbol{V}\) How step 2

\[\boldsymbol{V}=\frac{1}{50}\left[\begin{array}{cc}55&-15\\-15&5\end{array}\right]=\left[\begin{array}{cc}1.1&-0.3\\-0.3&0.1\end{array}\right]\]

Check: \(1.1(5)-0.3(15)=5.5-4.5=1\).

Multiply How step 3

\[\hat\sigma^2V_{00}=0.8(1.1)=0.88,\qquad \hat\sigma^2V_{11}=0.8(0.1)=0.08\]

Square root How step 4

\[\mathrm{SE}(\hat\beta_0)=\sqrt{0.88}=0.9381,\qquad \mathrm{SE}(\hat\beta_1)=\sqrt{0.08}=0.2828\]

3 degrees of freedom. Lecture 7 uses these. ▲

Why Added

\[\mathrm{Var}[\hat{\boldsymbol\beta}]=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\sigma^2\boldsymbol{V}\]

p.27.

\[\mathrm{Var}[\hat\beta_j]=\sigma^2V_{jj}\]

Diagonal entry \(j\) (p.29).

\[\mathrm{sd}(\hat\beta_j)=\sqrt{\sigma^2V_{jj}}\]

\[\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\]

Put in \(\hat\sigma^2\) for unknown \(\sigma^2\). ∎

05.5Practice Slides p.39–41

All questions use \(p=1\): simple linear regression (SLR). Lecture 2 notation: \(\bar x=\frac1n\sum x_i\), \(\bar y=\frac1n\sum y_i\), \(S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2\), \(S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)\).

Least squares estimates · Lecture 2 \[\hat\beta_1=\frac{S_{xy}}{S_{xx}},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.\]

\[\boldsymbol{X}=\left[\begin{array}{cc}1&x_1\\1&x_2\\\vdots&\vdots\\1&x_n\end{array}\right],\qquad \boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}n&\sum x_i\\\sum x_i&\sum x_i^2\end{array}\right],\qquad \boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}\sum y_i\\\sum x_iy_i\end{array}\right].\]

Five-point data: \(\bar x=15/5=3\), \(\bar y=20/5=4\).

\(S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2=4+1+0+1+4=10\).

\(S_{xy}=(-2)(-2)+(-1)(0)+(0)(1)+(1)(0)+(2)(1)=4+0+0+0+2=6\).

Q1 Slides p.40

Practice Q1 · Lecture 6 · p.40 In the case of \(p=1\) covariate, show that the matrix formula for \(\hat{\boldsymbol\beta}\) derived for multiple linear regression yields the familiar \(\hat\beta_0\) and \(\hat\beta_1\) from simple linear regression.

Answer Q1

Plan: invert \(\boldsymbol{X}^T\boldsymbol{X}\), multiply by \(\boldsymbol{X}^T\boldsymbol{y}\), write the sums with \(S_{xx}\) and \(S_{xy}\). First, two identities:

\[S_{xx}=\sum(x_i-\bar x)^2=\sum x_i^2-2\bar x\sum x_i+n\bar x^2=\sum x_i^2-2n\bar x^2+n\bar x^2=\sum x_i^2-n\bar x^2\]

\(\sum x_i=n\bar x\).

\[S_{xy}=\sum(x_i-\bar x)(y_i-\bar y)=\sum x_iy_i-\bar y\sum x_i-\bar x\sum y_i+n\bar x\bar y=\sum x_iy_i-n\bar x\bar y\]

Use the \(2\times2\) inverse rule of Section 05.4.

\[\det(\boldsymbol{X}^T\boldsymbol{X})=n\sum x_i^2-\left(\sum x_i\right)^2=n\sum x_i^2-n^2\bar x^2=n\left(\sum x_i^2-n\bar x^2\right)=nS_{xx}\]

Identity a.

\[(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\frac{1}{nS_{xx}}\left[\begin{array}{cc}\sum x_i^2&-\sum x_i\\-\sum x_i&n\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\frac{1}{nS_{xx}}\left[\begin{array}{cc}\sum x_i^2&-\sum x_i\\-\sum x_i&n\end{array}\right]\left[\begin{array}{c}\sum y_i\\\sum x_iy_i\end{array}\right]\]

\[\hat{\boldsymbol\beta}=\frac{1}{nS_{xx}}\left[\begin{array}{c}\sum x_i^2\sum y_i-\sum x_i\sum x_iy_i\\-\sum x_i\sum y_i+n\sum x_iy_i\end{array}\right]\]

Row 2: \(\hat\beta_1\)

\[\hat\beta_1=\frac{n\sum x_iy_i-n\bar x\cdot n\bar y}{nS_{xx}}=\frac{\sum x_iy_i-n\bar x\bar y}{S_{xx}}=\frac{S_{xy}}{S_{xx}}\]

\(\sum x_i=n\bar x\), \(\sum y_i=n\bar y\), then identity b.

Row 1: \(\hat\beta_0\)

\[\hat\beta_0=\frac{n\bar y\sum x_i^2-n\bar x\sum x_iy_i}{nS_{xx}}=\frac{\bar y\sum x_i^2-\bar x\sum x_iy_i}{S_{xx}}\]

\[\hat\beta_0=\frac{\bar y\left(S_{xx}+n\bar x^2\right)-\bar x\left(S_{xy}+n\bar x\bar y\right)}{S_{xx}}\]

Identities a and b in reverse.

\[\hat\beta_0=\frac{\bar yS_{xx}+n\bar x^2\bar y-\bar xS_{xy}-n\bar x^2\bar y}{S_{xx}}=\frac{\bar yS_{xx}-\bar xS_{xy}}{S_{xx}}\]

\[\hat\beta_0=\bar y-\frac{S_{xy}}{S_{xx}}\bar x=\bar y-\hat\beta_1\bar x\]

The Lecture 2 formula. ∎

Numerical check (five-point data). \(n=5\), \(\sum x_i=15\), \(\sum x_i^2=55\), \(\sum y_i=20\), \(\sum x_iy_i=66\).

Determinant: \(5(55)-15^2=275-225=50=nS_{xx}=5(10)\).

Row 2: \(\hat\beta_1=\frac{5(66)-15(20)}{50}=\frac{330-300}{50}=\frac{30}{50}=0.6\).

Row 1: \(\hat\beta_0=\frac{55(20)-15(66)}{50}=\frac{1100-990}{50}=\frac{110}{50}=2.2\).

SLR: \(\hat\beta_1=6/10=0.6\), \(\hat\beta_0=4-0.6(3)=4-1.8=2.2\).

Elimination in Section 5.3 gave the same \(\hat{\boldsymbol\beta}=(2.2,\,0.6)^T\). ▲

Q2 Slides p.41

Practice Q2 · Lecture 6 · p.41 In the case of \(p=1\) covariate, find the covariance between \(\hat\beta_0\) and \(\hat\beta_1\) using the above variance covariance matrix for \(\hat{\boldsymbol\beta}\).

Answer Q2

\(\mathrm{cov}(U,W)=E[(U-E[U])(W-E[W])]\). The "above" matrix is \(\mathrm{Var}[\hat{\boldsymbol\beta}]=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}\) (p.27). Its off-diagonal entry is \(\mathrm{cov}(\hat\beta_0,\hat\beta_1)\).

\[\mathrm{Var}[\hat{\boldsymbol\beta}]=\left[\begin{array}{cc}\mathrm{Var}(\hat\beta_0)&\mathrm{cov}(\hat\beta_0,\hat\beta_1)\\\mathrm{cov}(\hat\beta_1,\hat\beta_0)&\mathrm{Var}(\hat\beta_1)\end{array}\right]=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}\]

\[\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\frac{\sigma^2}{nS_{xx}}\left[\begin{array}{cc}\sum x_i^2&-\sum x_i\\-\sum x_i&n\end{array}\right]\]

Q1, step 2.

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=\frac{\sigma^2}{nS_{xx}}\left(-\sum x_i\right)\]

Row 1, column 2.

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=\frac{\sigma^2}{nS_{xx}}\left(-n\bar x\right)\]

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=-\frac{\sigma^2\bar x}{S_{xx}}\]

Negative when \(\bar x>0\). ∎

The fitted line goes through \((\bar x,\bar y)\) (Figure 5-1). A larger \(\hat\beta_1\) gives a smaller \(\hat\beta_0=\bar y-\hat\beta_1\bar x\).

Numerical check (five-point data). Use \(\hat\sigma^2=0.8\) for \(\sigma^2\).

Formula: \(-\frac{0.8(3)}{10}=-\frac{2.4}{10}=-0.24\).

Matrix: \(V_{01}=-0.3\) (Section 05.4), \(\hat\sigma^2V_{01}=0.8(-0.3)=-0.24\). ▲

Q3 Added

Data: \(n=4\), \(x=(0,1,2,3)\), \(y=(1,3,2,6)\). Fitted line: \(\hat y=0.9+1.4x\). (a) Calculate \(\boldsymbol{e}^T\boldsymbol{e}\), \(\hat\sigma^2_{MLE}\), and \(\hat\sigma^2\). (b) Calculate \(S_{xx}\) and \(\mathrm{SE}(\hat\beta_1)=\sqrt{\hat\sigma^2V_{11}}\). (c) Calculate \(\mathrm{cov}(\hat\beta_0,\hat\beta_1)\) with \(\hat\sigma^2\) for \(\sigma^2\).

Answer Q3

(a) Fitted values 05.1 How step 1

\[\hat y_1=0.9,\quad \hat y_2=0.9+1.4=2.3,\quad \hat y_3=0.9+2.8=3.7,\quad \hat y_4=0.9+4.2=5.1\]

(a) Residuals 05.1 How step 2

\[e_1=1-0.9=0.1,\quad e_2=3-2.3=0.7,\quad e_3=2-3.7=-1.7,\quad e_4=6-5.1=0.9\]

Check: \(\sum e_i=0.1+0.7-1.7+0.9=0\), \(\sum x_ie_i=0+0.7-3.4+2.7=0\).

(a) Residual sum of squares 05.1 How step 3

\[\boldsymbol{e}^T\boldsymbol{e}=0.01+0.49+2.89+0.81=4.2\]

(a) MLE 05.1 How step 4

\[\hat\sigma^2_{MLE}=\frac{4.2}{4}=1.05\]

(a) Unbiased estimate 05.2 How

\[n-p-1=4-1-1=2,\qquad \hat\sigma^2=\frac{4.2}{2}=2.1\]

Check: \(2.1/1.05=2=4/2=n/(n-p-1)\).

(b) \(S_{xx}\) and \(V_{11}\) Q1 step 2

\[\bar x=\frac{6}{4}=1.5,\qquad S_{xx}=2.25+0.25+0.25+2.25=5\]

\[V_{11}=\frac{n}{nS_{xx}}=\frac{1}{S_{xx}}=\frac15=0.2\]

Bottom-right entry of the Q1 inverse.

(b) SE 05.4 How steps 3–4

\[\hat\sigma^2V_{11}=2.1(0.2)=0.42,\qquad \mathrm{SE}(\hat\beta_1)=\sqrt{0.42}=0.6481\]

SLR form: \(\hat\sigma/\sqrt{S_{xx}}=\sqrt{2.1}/\sqrt{5}=1.4491/2.2361=0.6481\).

\[\mathrm{cov}(\hat\beta_0,\hat\beta_1)=-\frac{\hat\sigma^2\bar x}{S_{xx}}=-\frac{2.1(1.5)}{5}=-\frac{3.15}{5}=-0.63\]

Negative because \(\bar x=1.5>0\). ▲

Row	Satisfaction	Age	Severity	Stress
1	48	50	51	2.3
2	57	36	46	2.3
3	66	40	48	2.2
4	70	41	44	1.8
5	89	28	43	1.8
6	36	49	54	2.9

Row	Satisfaction	Age	Severity	Stress
1	48	50	51	2.3
2	57	36	46	2.3
3	66	40	48	2.2
4	70	41	44	1.8
5	89	28	43	1.8
6	36	49	54	2.9

Row	Satisfaction	Age	Severity	Stress
1	48	50	51	2.3
2	57	36	46	2.3
3	66	40	48	2.2
4	70	41	44	1.8
5	89	28	43	1.8
6	36	49	54	2.9