STAT 331 · Lecture 7 · Multiple Linear Regression: Estimation and Inference I

# Lecture 7: Multiple Linear Regression: Estimation and Inference I

This lecture finds the distributions of \(\hat{\boldsymbol\beta}\), \(\boldsymbol e\), and \(\hat\sigma^2\) and uses them for t tests and confidence intervals for each \(\beta_j\) (Lecture 7 · p.1–55).

Contents
01 · Recap, fitted values, residuals, and the hat matrix 02 · Estimating \(\sigma^2\) and the joint distribution of \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) 03 · The t statistic in MLR and property (3): independence 04 · Property (2): \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\) 05 · t tests and confidence intervals for each \(\beta_j\) 06 · Practice questions Q1–Q3

## 01 · Recap, fitted values, residuals, and the hat matrix

All examples use data set 1 (\(n=5\)), small enough to do by hand.

Plan · Slides p.1–9

Step 1: Model, scalar and matrix form (Slides p.1–3).

Step 2: Least squares estimate \(\hat{\boldsymbol\beta}\) (Slides p.4).

Step 3: Distribution of \(\hat{\boldsymbol\beta}\) (Slides p.5).

Step 4: Fitted values and the hat matrix \(\boldsymbol{H}\) (Slides p.6–7).

Step 5: Residuals and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) (Slides p.8–9).

### 01.1 Recap: the model Slides p.1–3

p.1: title page, "Multiple Linear Regression—Estimation and Inference I". p.2: section page "Recap". p.3–5 repeat Lecture 6.

**What** Slides p.3

Recap: Multiple Linear Regression · Lecture 7 · p.3 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] We can write this: \[\left[\begin{array}{c}y_1\\y_2\\\vdots\\y_n\end{array}\right]=\left[\begin{array}{ccccc}1&x_{11}&x_{12}&\cdots&x_{1p}\\1&x_{21}&x_{22}&\cdots&x_{2p}\\\vdots&\vdots&\vdots&&\vdots\\1&x_{n1}&x_{n2}&\cdots&x_{np}\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\vdots\\\epsilon_n\end{array}\right]\] Or more simply: \[\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon,\quad \boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\iff \boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\]

Multiple linear regression: a linear model that uses \(p\) explanatory variables to describe one response.

Response \(y_i\): the value we explain in observation \(i\), \(i=1,\dots,n\). Sample size \(n\): the number of observations.

Covariate \(x_{ij}\): the value of explanatory variable \(j\) in observation \(i\), \(j=1,\dots,p\).

Regression coefficients \(\beta_0,\dots,\beta_p\): unknown constants. \(\beta_0\) is the intercept.

\(\beta_j\): the change in the mean of \(y_i\) when \(x_{ij}\) increases by 1 and other covariates stay the same.

Error \(\epsilon_i\): the random part of \(y_i\) that the covariates do not explain.

\(\overset{iid}{\sim}\): independent and identically distributed (the errors do not affect each other and have one distribution).

Normal distribution \(N(\mu,\sigma^2)\): the bell-shaped distribution with mean \(\mu\) and variance \(\sigma^2\).

Expectation (mean) \(E[Z]\): the theoretical average of \(Z\). Variance \(\mathrm{Var}(Z)=E[(Z-E[Z])^2]\): the spread of \(Z\).

Bold letters are vectors or matrices.

\(\boldsymbol{y}\): the \(n\times1\) response vector.

\(\boldsymbol{X}\): the \(n\times(p+1)\) design matrix. Column 1 is all 1s (intercept). Column \(j+1\) is covariate \(j\). Row \(i\) is observation \(i\).

\(\boldsymbol\beta=(\beta_0,\beta_1,\dots,\beta_p)^T\): the coefficient vector. Transpose \(T\): rows become columns.

\(\boldsymbol\epsilon\): the \(n\times1\) error vector.

\(MVN(\boldsymbol\mu,\boldsymbol\Sigma)\): multivariate normal with mean vector \(\boldsymbol\mu\) and variance matrix \(\boldsymbol\Sigma\).

Variance matrix: variances on the diagonal, covariances elsewhere. Covariance: how two random variables change together.

\(\boldsymbol{I}\): the \(n\times n\) identity matrix (1s on the diagonal, 0s elsewhere).

\(\sigma^2\boldsymbol{I}\): each \(\epsilon_i\) has variance \(\sigma^2\); each pair has covariance 0.

The matrix form puts the \(n\) equations into one. All derivations use it.

**How** Added

Write \(\boldsymbol{y}\) and \(\boldsymbol{X}\) from a data table

- Put the response column in \(\boldsymbol{y}\).
- Count the covariates: \(p\).
- Write a column of 1s.
- Put the \(p\) covariate columns to its right. The result is \(\boldsymbol{X}\).

**Self-check:** \(\boldsymbol{X}\) is \(n\times(p+1)\). \(\boldsymbol{y}\) has \(n\) rows.

**Self-check:** Column 1 of \(\boldsymbol{X}\) is all 1s.

**Example · Data set 1** Added

A made-up data set: \(n=5\), \(p=2\).

\[\begin{array}{c|ccccc} i&1&2&3&4&5\\\hline x_{i1}&-2&-1&0&1&2\\ x_{i2}&1&-1&0&-1&1\\ y_i&3&5&6&8&13\end{array}\]

1

Write the model How step 2

\[y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\qquad i=1,\dots,5\]

2 covariates: \(p=2\).

2

Write \(\boldsymbol{y}\) and \(\boldsymbol{X}\) How steps 1, 3, 4

\[\underbrace{\left[\begin{array}{c}3\\5\\6\\8\\13\end{array}\right]}_{\boldsymbol{y}:\ 5\times1}=\underbrace{\left[\begin{array}{ccc}1&-2&1\\1&-1&-1\\1&0&0\\1&1&-1\\1&2&1\end{array}\right]}_{\boldsymbol{X}:\ 5\times3}\left[\begin{array}{c}\beta_0\\\beta_1\\\beta_2\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\\\epsilon_4\\\epsilon_5\end{array}\right]\]

Self-check: \(5\times3=5\times(2+1)\); column 1 all 1s. ▲

**Why · Row \(i\) of the matrix form is the scalar model** Added

Entry \(i\) of \(\boldsymbol{X}\boldsymbol\beta\) is row \(i\) of \(\boldsymbol{X}\) times \(\boldsymbol\beta\): multiply entry by entry, then add.

1

\[(\boldsymbol{y})_i=(\boldsymbol{X}\boldsymbol\beta)_i+(\boldsymbol\epsilon)_i\]

2

\[y_i=\left[\begin{array}{ccccc}1&x_{i1}&x_{i2}&\cdots&x_{ip}\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]+\epsilon_i\]

Row \(i\) of \(\boldsymbol{X}\).

3

\[y_i=1\cdot\beta_0+x_{i1}\beta_1+\cdots+x_{ip}\beta_p+\epsilon_i\]

The scalar model on p.3. ∎

Adding the constant \(\boldsymbol{X}\boldsymbol\beta\) to \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\) moves the mean and keeps the variance. This gives \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\).

### 01.2 Recap: least squares estimation Slides p.4

Estimate: a value calculated from the data to approximate an unknown parameter. A hat marks an estimate.

**What** Slides p.4

Least Squares Estimation · Lecture 7 · p.4 Want to minimize the sum of squares, i.e.: \[S(\boldsymbol\beta)=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)=\boldsymbol{y}^T\boldsymbol{y}-\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta-\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=\boldsymbol{y}^T\boldsymbol{y}-2\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\] \[\frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=0\ \Rightarrow\ (\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\ \Rightarrow\ \hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\] assuming the columns of \(\boldsymbol{X}\) are linearly independent

Least squares: select \(\boldsymbol\beta\) to make the sum of squared differences between \(\boldsymbol{y}\) and \(\boldsymbol{X}\boldsymbol\beta\) smallest.

Sum of squares, scalar form: \(S(\boldsymbol\beta)=\sum_{i=1}^n(y_i-\beta_0-\beta_1x_{i1}-\cdots-\beta_px_{ip})^2\).

Gradient \(\partial S/\partial\boldsymbol\beta\): the column of partial derivatives of \(S\) for \(\beta_0,\dots,\beta_p\).

At the minimum the gradient is \(\boldsymbol{0}\). This gives the normal equations \((\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\): \(p+1\) equations, \(p+1\) unknowns.

Inverse \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\): the matrix with \((\boldsymbol{X}^T\boldsymbol{X})^{-1}(\boldsymbol{X}^T\boldsymbol{X})=\boldsymbol{I}\).

Linearly independent columns: no column is a linear combination of the others. Then \(\boldsymbol{X}^T\boldsymbol{X}\) is invertible.

**How** Added

Calculate \(\hat{\boldsymbol\beta}\)

- Calculate \(\boldsymbol{X}^T\boldsymbol{X}\). Entry \((j,k)\): multiply columns \(j\) and \(k\) of \(\boldsymbol{X}\) entry by entry, then add.
- Calculate \(\boldsymbol{X}^T\boldsymbol{y}\). Entry \(j\): multiply column \(j\) and \(\boldsymbol{y}\) entry by entry, then add.
- Calculate \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\).
- Multiply: \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).

**Self-check:** \(\boldsymbol{X}^T\boldsymbol{X}\) is \((p+1)\times(p+1)\), symmetric (equal to its transpose), top-left entry \(n\).

**Self-check:** \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\).

**Example · Data set 1** Added

1

Diagonal of \(\boldsymbol{X}^T\boldsymbol{X}\) How step 1

\[\sum 1\cdot1=5,\qquad \sum x_{i1}^2=4+1+0+1+4=10,\qquad \sum x_{i2}^2=1+1+0+1+1=4\]

2

Other entries How step 1

\[\sum x_{i1}=-2-1+0+1+2=0,\qquad \sum x_{i2}=1-1+0-1+1=0\]

\[\sum x_{i1}x_{i2}=(-2)(1)+(-1)(-1)+(0)(0)+(1)(-1)+(2)(1)=-2+1+0-1+2=0\]

3

\[\boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{ccc}5&0&0\\0&10&0\\0&0&4\end{array}\right]\]

Self-check: \(3\times3\), symmetric, top-left \(5=n\).

4

\(\boldsymbol{X}^T\boldsymbol{y}\) How step 2

\[\sum y_i=3+5+6+8+13=35\]

\[\sum x_{i1}y_i=(-2)(3)+(-1)(5)+(0)(6)+(1)(8)+(2)(13)=-6-5+0+8+26=23\]

\[\sum x_{i2}y_i=(1)(3)+(-1)(5)+(0)(6)+(-1)(8)+(1)(13)=3-5+0-8+13=3\]

5

\[\boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}35\\23\\3\end{array}\right]\]

6

Inverse How step 3

\[(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\left[\begin{array}{ccc}1/5&0&0\\0&1/10&0\\0&0&1/4\end{array}\right]=\left[\begin{array}{ccc}0.2&0&0\\0&0.1&0\\0&0&0.25\end{array}\right]\]

Diagonal matrix: invert each diagonal entry.

7

Multiply How step 4

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}0.2\times35\\0.1\times23\\0.25\times3\end{array}\right]\]

8

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}\hat\beta_0\\\hat\beta_1\\\hat\beta_2\end{array}\right]=\left[\begin{array}{c}7\\2.3\\0.75\end{array}\right]\]

Self-check: \(5\times7=35\), \(10\times2.3=23\), \(4\times0.75=3\). ▲

Fitted model: \(\hat y=7+2.3x_1+0.75x_2\). \(\boldsymbol{X}^T\boldsymbol{X}\) is diagonal because the columns of \(\boldsymbol{X}\) are orthogonal.

Inner product of two vectors: the sum of the products of their entries. Orthogonal: inner product 0.

**Why · The formula for \(\hat{\boldsymbol\beta}\)** Slides p.4

Rule A: \((\boldsymbol{A}\boldsymbol{B})^T=\boldsymbol{B}^T\boldsymbol{A}^T\).

Rule B: A \(1\times1\) matrix is a number and equals its transpose.

Rule C: \(\partial(\boldsymbol{a}^T\boldsymbol\beta)/\partial\boldsymbol\beta=\boldsymbol{a}\). For symmetric \(\boldsymbol{A}\), \(\partial(\boldsymbol\beta^T\boldsymbol{A}\boldsymbol\beta)/\partial\boldsymbol\beta=2\boldsymbol{A}\boldsymbol\beta\).

1

\[S(\boldsymbol\beta)=(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

2

\[S(\boldsymbol\beta)=(\boldsymbol{y}^T-\boldsymbol\beta^T\boldsymbol{X}^T)(\boldsymbol{y}-\boldsymbol{X}\boldsymbol\beta)\]

Rule A.

3

\[S(\boldsymbol\beta)=\boldsymbol{y}^T\boldsymbol{y}-\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta-\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

4

\[S(\boldsymbol\beta)=\boldsymbol{y}^T\boldsymbol{y}-2\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta+\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Rule B: \(\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta\).

5

\[\frac{\partial S(\boldsymbol\beta)}{\partial\boldsymbol\beta}=-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Rule C, \(\boldsymbol{a}=\boldsymbol{X}^T\boldsymbol{y}\), \(\boldsymbol{A}=\boldsymbol{X}^T\boldsymbol{X}\).

6

\[-2\boldsymbol{X}^T\boldsymbol{y}+2\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta=\boldsymbol{0}\]

7

\[(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\]

8

\[\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

Independent columns: the inverse exists. ∎

### 01.3 Recap: properties of \(\hat{\boldsymbol\beta}\) Slides p.5

Sampling distribution: the distribution of \(\hat{\boldsymbol\beta}\) over many new data sets with the same \(\boldsymbol{X}\).

**What** Slides p.5

Properties of \(\hat{\boldsymbol\beta}\) · Lecture 7 · p.5 \[\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\ \sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1})\ \Longrightarrow\ \hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\] where \(V_{jj}\) is the corresponding diagonal of \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\).
Note: the notation here is a little funky due to 0-indexing \(\boldsymbol\beta\): \[\mathrm{Var}(\hat{\boldsymbol\beta})=\mathrm{Var}\left(\left[\begin{array}{c}\hat\beta_0\\\hat\beta_1\\\vdots\\\hat\beta_p\end{array}\right]\right)=\sigma^2\left[\begin{array}{cccc}V_{00}&V_{01}&\cdots&V_{0p}\\V_{10}&V_{11}&\cdots&V_{1p}\\\vdots&&&\vdots\\V_{p0}&V_{p1}&\cdots&V_{pp}\end{array}\right]\]

\(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\): \(\hat{\boldsymbol\beta}\) is unbiased. Unbiased: the mean of the estimator equals the parameter.

\(\mathrm{Var}(\hat\beta_j)=\sigma^2V_{jj}\), with \(V_{jj}\) a diagonal entry of \(\boldsymbol{V}\).

0-indexing: row and column numbers of \(\boldsymbol{V}\) start at 0. \(V_{jj}\) is in row \(j+1\), column \(j+1\).

**How** Added

Find \(\mathrm{Var}(\hat\beta_j)\)

- Calculate \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\).
- Find \(V_{jj}\) in row \(j+1\), column \(j+1\).
- Write \(\mathrm{Var}(\hat\beta_j)=\sigma^2V_{jj}\) and \(\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\).

**Self-check:** \(V_{jj}>0\), because a variance cannot be negative.

**Self-check:** Row number \(=j+1\).

**Example · Data set 1** Added

1

\(\boldsymbol{V}\) How step 1

\[\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\left[\begin{array}{ccc}0.2&0&0\\0&0.1&0\\0&0&0.25\end{array}\right]\]

Section 01.2, step 6.

2

Diagonal entries How step 2

\[V_{00}=0.2\ (\text{row }1),\qquad V_{11}=0.1\ (\text{row }2),\qquad V_{22}=0.25\ (\text{row }3)\]

3

Distributions How step 3

\[\hat\beta_0\sim N(\beta_0,\,0.2\,\sigma^2),\qquad \hat\beta_1\sim N(\beta_1,\,0.1\,\sigma^2),\qquad \hat\beta_2\sim N(\beta_2,\,0.25\,\sigma^2)\]

Self-check: all positive. ▲

\(\sigma^2\) is unknown. Unit 02 estimates it from the residuals.

**Why · The distribution of \(\hat{\boldsymbol\beta}\)** Added

If \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) and \(\boldsymbol{A}\) is constant, then \(\boldsymbol{A}\boldsymbol{y}\sim MVN(\boldsymbol{A}\boldsymbol\mu,\boldsymbol{A}\boldsymbol\Sigma\boldsymbol{A}^T)\). Let \(\boldsymbol{A}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\), so \(\hat{\boldsymbol\beta}=\boldsymbol{A}\boldsymbol{y}\).

1

Mean

\[E[\hat{\boldsymbol\beta}]=\boldsymbol{A}\boldsymbol{X}\boldsymbol\beta=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{X}\boldsymbol\beta\]

Substitute \(\boldsymbol\mu=\boldsymbol{X}\boldsymbol\beta\).

2

\[E[\hat{\boldsymbol\beta}]=\boldsymbol{I}\boldsymbol\beta=\boldsymbol\beta\]

3

Variance

\[\mathrm{Var}(\hat{\boldsymbol\beta})=\boldsymbol{A}(\sigma^2\boldsymbol{I})\boldsymbol{A}^T=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{X}\left[(\boldsymbol{X}^T\boldsymbol{X})^{-1}\right]^T\]

\(\boldsymbol\Sigma=\sigma^2\boldsymbol{I}\); Rule A for \(\boldsymbol{A}^T\).

4

\[\mathrm{Var}(\hat{\boldsymbol\beta})=\sigma^2\boldsymbol{I}\left[(\boldsymbol{X}^T\boldsymbol{X})^{-1}\right]^T\]

5

\[\mathrm{Var}(\hat{\boldsymbol\beta})=\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}\]

Inverse of a symmetric matrix is symmetric. ∎

### 01.4 Fitted values and the hat matrix Slides p.6–7

p.6 repeats the title page. New material starts here.

**What** Slides p.7

Fitted value: \(\hat y_i=\hat\beta_0+\hat\beta_1x_{i1}+\cdots+\hat\beta_px_{ip}\), the value the estimated model gives for observation \(i\).

Fitted values · Lecture 7 · p.7 \[\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{X}\left[(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\right]=\left[\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}=\boldsymbol{H}\boldsymbol{y}\] where \(\boldsymbol{H}=\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\) is called the Hat matrix.

Hat matrix \(\boldsymbol{H}\): the \(n\times n\) matrix that "puts the hat on" \(\boldsymbol{y}\).

\(\boldsymbol{H}\) depends only on \(\boldsymbol{X}\), not on \(\boldsymbol{y}\).

Each \(\hat y_i\) is a linear combination of \(y_1,\dots,y_n\), with weights from row \(i\) of \(\boldsymbol{H}\).

**How** Added

Calculate \(\hat{\boldsymbol{y}}\) and \(\boldsymbol{H}\)

- Calculate \(\hat{\boldsymbol\beta}\) (Section 01.2).
- For each \(i\), multiply row \(i\) of \(\boldsymbol{X}\) by \(\hat{\boldsymbol\beta}\) entry by entry, then add. The result is \(\hat y_i\).
- For \(\boldsymbol{H}\), entry \((i,k)\) is row \(i\) of \(\boldsymbol{X}\), times \(\boldsymbol{V}\), times row \(k\) transposed.

**Self-check:** \(\boldsymbol{H}\) is \(n\times n\) and symmetric.

**Self-check:** Row \(i\) of \(\boldsymbol{H}\) times \(\boldsymbol{y}\) equals \(\hat y_i\) from step 2.

**Example · Data set 1** Added

1

Fitted value of observation 2 How step 2

\[\hat y_2=7+2.3\times(-1)+0.75\times(-1)\]

Row 2 of \(\boldsymbol{X}\): \((1,\,-1,\,-1)\).

2

\[\hat y_2=7-2.3-0.75=3.95\]

3

All fitted values How step 2

\[\hat y_1=7+2.3(-2)+0.75(1)=7-4.6+0.75=3.15\]

\[\hat y_3=7+2.3(0)+0.75(0)=7\]

\[\hat y_4=7+2.3(1)+0.75(-1)=7+2.3-0.75=8.55\]

\[\hat y_5=7+2.3(2)+0.75(1)=7+4.6+0.75=12.35\]

4

\[\hat{\boldsymbol{y}}=(3.15,\ 3.95,\ 7,\ 8.55,\ 12.35)^T\]

5

Entries of \(\boldsymbol{H}\) How step 3

\(\boldsymbol{V}\) is diagonal: three terms.

\[H_{ik}=\frac{1}{5}+\frac{x_{i1}x_{k1}}{10}+\frac{x_{i2}x_{k2}}{4}\]

6

\[H_{11}=0.2+\frac{(-2)(-2)}{10}+\frac{(1)(1)}{4}=0.2+0.4+0.25=0.85\]

\[H_{12}=0.2+\frac{(-2)(-1)}{10}+\frac{(1)(-1)}{4}=0.2+0.2-0.25=0.15\]

7

\[\boldsymbol{H}=\left[\begin{array}{ccccc}0.85&0.15&0.2&-0.25&0.05\\0.15&0.55&0.2&0.35&-0.25\\0.2&0.2&0.2&0.2&0.2\\-0.25&0.35&0.2&0.55&0.15\\0.05&-0.25&0.2&0.15&0.85\end{array}\right]\]

Self-check: \(5\times5\), symmetric.

8

Check row 1 of \(\boldsymbol{H}\boldsymbol{y}\) How self-check

\[0.85(3)+0.15(5)+0.2(6)-0.25(8)+0.05(13)=2.55+0.75+1.2-2+0.65=3.15\]

Equals \(\hat y_1\) from step 3. ▲

**Why · \(\hat{\boldsymbol{y}}=\boldsymbol{H}\boldsymbol{y}\)** Slides p.7

1

\[\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\]

2

\[\hat{\boldsymbol{y}}=\boldsymbol{X}\left[(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\right]\]

Substitute \(\hat{\boldsymbol\beta}\).

3

\[\hat{\boldsymbol{y}}=\left[\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}\]

Matrix multiplication is associative.

4

\[\hat{\boldsymbol{y}}=\boldsymbol{H}\boldsymbol{y}\]

Size: \((n\times(p+1))((p+1)\times(p+1))((p+1)\times n)=n\times n\). ∎

### 01.5 Residuals and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) Slides p.8–9

**What** Slides p.8–9

Residual \(e_i=y_i-\hat y_i\): the part of \(y_i\) that the estimated model does not fit.

Residuals · Lecture 7 · p.8–9 \[\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{y}-\boldsymbol{H}\boldsymbol{y}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}.\] Note: \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\). Why? \[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{H}\boldsymbol{y})=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{0}\]

The error \(\epsilon_i=y_i-\boldsymbol{x}_i^T\boldsymbol\beta\) uses the true \(\boldsymbol\beta\). We cannot see it.

The residual \(e_i=y_i-\boldsymbol{x}_i^T\hat{\boldsymbol\beta}\) uses \(\hat{\boldsymbol\beta}\). We can calculate it. \(\boldsymbol{x}_i^T\) is row \(i\) of \(\boldsymbol{X}\).

\(\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}\): a matrix that depends only on \(\boldsymbol{X}\), times \(\boldsymbol{y}\).

\(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\): each column of \(\boldsymbol{X}\) is orthogonal to \(\boldsymbol{e}\) (\(p+1\) equations).

Column 1 is all 1s, which gives \(\sum_{i=1}^ne_i=0\).

**How** Added

Calculate \(\boldsymbol{e}\) and check \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\)

- Calculate \(\hat{\boldsymbol{y}}\) (Section 01.4).
- For each \(i\), calculate \(e_i=y_i-\hat y_i\).
- For each column of \(\boldsymbol{X}\), multiply it by \(\boldsymbol{e}\) entry by entry, then add.

**Self-check:** All \(p+1\) sums are 0. If not, \(\hat{\boldsymbol\beta}\) has an error.

**Example · Data set 1** Added

1

Residual of observation 2 How step 2

\[e_2=y_2-\hat y_2=5-3.95=1.05\]

\(\hat y_2\) from Section 01.4, step 2.

2

All residuals How step 2

\[e_1=3-3.15=-0.15,\quad e_3=6-7=-1,\quad e_4=8-8.55=-0.55,\quad e_5=13-12.35=0.65\]

\[\boldsymbol{e}=(-0.15,\ 1.05,\ -1,\ -0.55,\ 0.65)^T\]

3

Column 1 times \(\boldsymbol{e}\) How step 3

\[\sum e_i=-0.15+1.05-1-0.55+0.65=0\]

4

Column 2 times \(\boldsymbol{e}\) How step 3

\[\sum x_{i1}e_i=(-2)(-0.15)+(-1)(1.05)+(0)(-1)+(1)(-0.55)+(2)(0.65)\]

\[\sum x_{i1}e_i=0.3-1.05+0-0.55+1.3=0\]

5

Column 3 times \(\boldsymbol{e}\) How step 3

\[\sum x_{i2}e_i=(1)(-0.15)+(-1)(1.05)+(0)(-1)+(-1)(-0.55)+(1)(0.65)\]

\[\sum x_{i2}e_i=-0.15-1.05+0+0.55+0.65=0\]

All sums 0: \(\hat{\boldsymbol\beta}\) is correct. ▲

[figure]
Figure 01.1 · Data set 1: observed \(y_i\) against fitted \(\hat y_i\). Grey line: \(y=\hat y\). Each vertical segment is a residual. Blue: \(e_i>0\). Orange: \(e_i<0\). Purple circle: observation 2. Equal axis scales.

Blue and orange segments have equal total length, \(1.05+0.65=1.7=0.15+1+0.55\). This is \(\sum e_i=0\).

**Why · \(\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}\) and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\)** Slides p.8–9

1

\[\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}\]

2

\[\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}\]

Substitute \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\).

3

\[\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{H}\boldsymbol{y}\]

Substitute \(\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{H}\boldsymbol{y}\).

4

\[\boldsymbol{e}=\boldsymbol{I}\boldsymbol{y}-\boldsymbol{H}\boldsymbol{y}\]

5

\[\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}\]

∎

1

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{H}\boldsymbol{y})\]

Line 3 above.

2

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{H}\boldsymbol{y}\]

3

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

Substitute \(\boldsymbol{H}\).

4

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{I}\,\boldsymbol{X}^T\boldsymbol{y}\]

\(\boldsymbol{X}^T\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\boldsymbol{I}\).

5

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{y}\]

6

\[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\]

∎

### 01.6 Practice Added

**Q1.** In simple linear regression, \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), write \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) as two scalar equations.

Answer

1

\[\boldsymbol{X}^T\boldsymbol{e}=\left[\begin{array}{cccc}1&1&\cdots&1\\x_1&x_2&\cdots&x_n\end{array}\right]\left[\begin{array}{c}e_1\\e_2\\\vdots\\e_n\end{array}\right]\]

Rows of \(\boldsymbol{X}^T\) are columns of \(\boldsymbol{X}\).

2

\[\boldsymbol{X}^T\boldsymbol{e}=\left[\begin{array}{c}\sum_{i=1}^ne_i\\\sum_{i=1}^nx_ie_i\end{array}\right]\]

3

\[\sum_{i=1}^ne_i=0,\qquad \sum_{i=1}^nx_ie_i=0\]

▲

**Q2.** Data: \(n=3\), \(x=(0,1,2)\), \(y=(1,3,2)\). Model: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\). Calculate \(\hat{\boldsymbol\beta}\), \(\hat{\boldsymbol{y}}\), \(\boldsymbol{e}\). Check \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\).

Answer

1

\(\boldsymbol{X}\) and \(\boldsymbol{X}^T\boldsymbol{X}\) Section 01.2, How step 1

\[\boldsymbol{X}=\left[\begin{array}{cc}1&0\\1&1\\1&2\end{array}\right],\qquad \boldsymbol{X}^T\boldsymbol{X}=\left[\begin{array}{cc}3&0+1+2\\0+1+2&0+1+4\end{array}\right]=\left[\begin{array}{cc}3&3\\3&5\end{array}\right]\]

2

\(\boldsymbol{X}^T\boldsymbol{y}\) Section 01.2, How step 2

\[\boldsymbol{X}^T\boldsymbol{y}=\left[\begin{array}{c}1+3+2\\0(1)+1(3)+2(2)\end{array}\right]=\left[\begin{array}{c}6\\7\end{array}\right]\]

3

Inverse Section 01.2, How step 3

Rows \((a,b),(c,d)\): inverse \(\frac{1}{ad-bc}\) times rows \((d,-b),(-c,a)\).

\[ad-bc=3\times5-3\times3=15-9=6\]

\[(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\frac{1}{6}\left[\begin{array}{cc}5&-3\\-3&3\end{array}\right]\]

4

\(\hat{\boldsymbol\beta}\) Section 01.2, How step 4

\[\hat{\boldsymbol\beta}=\frac{1}{6}\left[\begin{array}{c}5(6)-3(7)\\-3(6)+3(7)\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}30-21\\-18+21\end{array}\right]=\frac{1}{6}\left[\begin{array}{c}9\\3\end{array}\right]=\left[\begin{array}{c}1.5\\0.5\end{array}\right]\]

5

\(\hat{\boldsymbol{y}}\) Section 01.4, How step 2

\[\hat y_1=1.5+0.5(0)=1.5,\quad \hat y_2=1.5+0.5(1)=2,\quad \hat y_3=1.5+0.5(2)=2.5\]

6

\(\boldsymbol{e}\) Section 01.5, How step 2

\[e_1=1-1.5=-0.5,\quad e_2=3-2=1,\quad e_3=2-2.5=-0.5\]

7

Check Section 01.5, How step 3

\[\sum e_i=-0.5+1-0.5=0,\qquad \sum x_ie_i=0(-0.5)+1(1)+2(-0.5)=0+1-1=0\]

▲

**Q3.** Use \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) to show \(\boldsymbol{X}^T\hat{\boldsymbol{y}}=\boldsymbol{X}^T\boldsymbol{y}\). Check it for data set 1.

Answer

1

\[\boldsymbol{X}^T\hat{\boldsymbol{y}}=\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{e})\]

\(\hat{\boldsymbol{y}}=\boldsymbol{y}-\boldsymbol{e}\).

2

\[\boldsymbol{X}^T\hat{\boldsymbol{y}}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{e}\]

3

\[\boldsymbol{X}^T\hat{\boldsymbol{y}}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{0}\]

\(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) (Slides p.9).

4

\[\boldsymbol{X}^T\hat{\boldsymbol{y}}=\boldsymbol{X}^T\boldsymbol{y}\]

5

Check with \(\hat{\boldsymbol{y}}=(3.15,\,3.95,\,7,\,8.55,\,12.35)^T\)

\[\sum\hat y_i=3.15+3.95+7+8.55+12.35=35\]

\[\sum x_{i1}\hat y_i=-6.3-3.95+0+8.55+24.7=23\]

\[\sum x_{i2}\hat y_i=3.15-3.95+0-8.55+12.35=3\]

6

\[\boldsymbol{X}^T\hat{\boldsymbol{y}}=(35,\,23,\,3)^T=\boldsymbol{X}^T\boldsymbol{y}\]

Section 01.2, step 5. ∎

## 02 · Estimating \(\sigma^2\) and the joint distribution of \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\)

Plan · Slides p.10–25

Step 1: Maximum likelihood estimator \(\hat\sigma^2_{MLE}\) (Slides p.10–12).

Step 2: The course estimator \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\) (Slides p.13).

Step 3: Stack \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\); show the stack is MVN (Slides p.14).

Step 4: Mean of the stack (Slides p.15–16).

Step 5: Variance matrix of the stack, block by block (Slides p.17–22).

Step 6: Distributions of \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\), and their independence (Slides p.23–25).

Error variance \(\sigma^2\): the unknown variance of each \(\epsilon_i\).

Units 02, 04, and 06 use data set 2. Added

Data set 2 · Added

\(n=4\), \(p=1\), \(x=(0,1,2,3)\), \(\boldsymbol y=(1,3,2,5)^T\).
\[\boldsymbol X=\left[\begin{array}{cc}1&0\\1&1\\1&2\\1&3\end{array}\right],\qquad \boldsymbol y=\left[\begin{array}{c}1\\3\\2\\5\end{array}\right]\]

1

\[\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}1+1+1+1&0+1+2+3\\0+1+2+3&0+1+4+9\end{array}\right]=\left[\begin{array}{cc}4&6\\6&14\end{array}\right]\]

2

\[\det(\boldsymbol X^T\boldsymbol X)=4\times14-6\times6=56-36=20\]

3

\[\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}=\frac{1}{20}\left[\begin{array}{cc}14&-6\\-6&4\end{array}\right]=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\]

2×2 inverse: swap diagonal, negate others, divide by determinant.

4

\[\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}1+3+2+5\\0+3+4+15\end{array}\right]=\left[\begin{array}{c}11\\22\end{array}\right]\]

5

\[\hat{\boldsymbol\beta}=\boldsymbol V\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}0.7(11)-0.3(22)\\-0.3(11)+0.2(22)\end{array}\right]=\left[\begin{array}{c}7.7-6.6\\-3.3+4.4\end{array}\right]=\left[\begin{array}{c}1.1\\1.1\end{array}\right]\]

6

\[\boldsymbol X\hat{\boldsymbol\beta}=\left[\begin{array}{c}1.1+0(1.1)\\1.1+1(1.1)\\1.1+2(1.1)\\1.1+3(1.1)\end{array}\right]=\left[\begin{array}{c}1.1\\2.2\\3.3\\4.4\end{array}\right]\]

7

\[\boldsymbol e=\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta}=\left[\begin{array}{c}1-1.1\\3-2.2\\2-3.3\\5-4.4\end{array}\right]=\left[\begin{array}{c}-0.1\\0.8\\-1.3\\0.6\end{array}\right]\]

Entries of \(\boldsymbol H\): \(h_{ij}=\left[\begin{array}{cc}1&x_i\end{array}\right]\boldsymbol V\left[\begin{array}{c}1\\x_j\end{array}\right]=0.7-0.3(x_i+x_j)+0.2x_ix_j\). For example, \(h_{12}=0.7-0.3(0+1)+0.2(0)(1)=0.4\).
\[\boldsymbol H=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\0.4&0.3&0.2&0.1\\0.1&0.2&0.3&0.4\\-0.2&0.1&0.4&0.7\end{array}\right],\qquad \boldsymbol I-\boldsymbol H=\left[\begin{array}{cccc}0.3&-0.4&-0.1&0.2\\-0.4&0.7&-0.2&-0.1\\-0.1&-0.2&0.7&-0.4\\0.2&-0.1&-0.4&0.3\end{array}\right]\]
Self-check: row 1 of \((\boldsymbol I-\boldsymbol H)\boldsymbol y\) is \(0.3(1)-0.4(3)-0.1(2)+0.2(5)=0.3-1.2-0.2+1.0=-0.1=e_1\).

### 02.1 The MLE of \(\sigma^2\) Slides p.10–12

p.10: section page "Estimating \(\sigma^2\)". \(\mathrm{Var}(\hat{\boldsymbol\beta})\) contains \(\sigma^2\). Estimate \(\sigma^2\) before you calculate standard errors and confidence intervals.

**What** Slides p.11–12

Likelihood: the joint density of the observed \(\boldsymbol y\), as a function of \((\boldsymbol\beta,\sigma^2)\).

Log-likelihood \(\ell(\boldsymbol\beta,\sigma^2|\boldsymbol y)\): the natural log of the likelihood. Both have their maximum at the same point.

Maximum likelihood estimate (MLE): the parameter value that makes \(\ell\) largest.

Lecture 7 · p.11–12 \[\ell(\boldsymbol\beta,\sigma^2|\boldsymbol y)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log\sigma^2-\frac{1}{2\sigma^2}(\boldsymbol y-\boldsymbol X\boldsymbol\beta)^T(\boldsymbol y-\boldsymbol X\boldsymbol\beta)\] \[\frac{\partial\ell(\boldsymbol\beta,\sigma^2|\boldsymbol y)}{\partial\sigma^2}=-\frac{n}{2\sigma^2}+\frac{1}{2(\sigma^2)^2}(\boldsymbol y-\boldsymbol X\boldsymbol\beta)^T(\boldsymbol y-\boldsymbol X\boldsymbol\beta)=0\] \[(\boldsymbol y-\boldsymbol X\boldsymbol\beta)^T(\boldsymbol y-\boldsymbol X\boldsymbol\beta)=n\sigma^2\] \[\Longrightarrow\ \hat\sigma^2_{MLE}=\frac{1}{n}(\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})^T(\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})\]

Line 1 is the log density of \(\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\). Line 2 sets the partial derivative for \(\sigma^2\) to 0. The MLE of \(\boldsymbol\beta\) is \(\hat{\boldsymbol\beta}\).

\((\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})^T(\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})=\boldsymbol e^T\boldsymbol e=\sum_{i=1}^n e_i^2\), the sum of squared residuals. \(\hat\sigma^2_{MLE}\) is their mean.

**How** Added

Calculate \(\hat\sigma^2_{MLE}\)

- Calculate \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e=\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta}\).
- Square each residual.
- Add the squares: \(\boldsymbol e^T\boldsymbol e\).
- Divide by \(n\).

**Self-check:** After step 1, the residuals add to 0 (model with intercept).

**Self-check:** In \(\ell\), the last term equals \(-n/2\).

**Example · Data set 2** Added

1

Residuals How step 1

\[\boldsymbol e=(-0.1,\ 0.8,\ -1.3,\ 0.6)^T\]

Self-check: \(-0.1+0.8-1.3+0.6=0\).

2

Squares How step 2

\[e_i^2=0.01,\ 0.64,\ 1.69,\ 0.36\]

3

Sum How step 3

\[\boldsymbol e^T\boldsymbol e=0.01+0.64+1.69+0.36=2.70\]

4

Divide by \(n\) How step 4

\[\hat\sigma^2_{MLE}=\frac{2.70}{4}=0.675\]

▲

Self-check (\(n=4\)): the last term is \(-\frac{2.70}{2(0.675)}=-2=-n/2\). Then \(\ell=-2(1.837877)-2(-0.393043)-2=-3.675754+0.786085-2=-4.8897\).

[figure]
Figure 02.1 · \(\ell(\hat{\boldsymbol\beta},\sigma^2|\boldsymbol y)\) for data set 2. Orange: maximum at \(\hat\sigma^2_{MLE}=0.675\). Green: \(\hat\sigma^2=1.35\) (Section 02.2).

**Why** Slides p.11–12

Let \(Q=(\boldsymbol y-\boldsymbol X\boldsymbol\beta)^T(\boldsymbol y-\boldsymbol X\boldsymbol\beta)\). \(Q\) does not contain \(\sigma^2\).

1

\[f(\boldsymbol y)=\frac{1}{(2\pi)^{n/2}|\sigma^2\boldsymbol I|^{1/2}}\exp\left(-\frac{1}{2}(\boldsymbol y-\boldsymbol X\boldsymbol\beta)^T(\sigma^2\boldsymbol I)^{-1}(\boldsymbol y-\boldsymbol X\boldsymbol\beta)\right)\]

MVN density (Lecture 5).

2

\[=\frac{1}{(2\pi)^{n/2}(\sigma^2)^{n/2}}\exp\left(-\frac{1}{2\sigma^2}Q\right)\]

\(|\sigma^2\boldsymbol I|=(\sigma^2)^n\), \((\sigma^2\boldsymbol I)^{-1}=\frac{1}{\sigma^2}\boldsymbol I\).

3

\[\ell(\boldsymbol\beta,\sigma^2|\boldsymbol y)=\log f(\boldsymbol y)=-\frac{n}{2}\log(2\pi)-\frac{n}{2}\log\sigma^2-\frac{1}{2\sigma^2}Q\]

Take the log. Line 1 of p.11.

4

\[\frac{\partial\ell}{\partial\sigma^2}=0-\frac{n}{2}\cdot\frac{1}{\sigma^2}-\frac{Q}{2}\cdot\left(-\frac{1}{(\sigma^2)^2}\right)\]

With \(t=\sigma^2\): \(\frac{d}{dt}\log t=\frac1t\), \(\frac{d}{dt}\frac1t=-\frac{1}{t^2}\).

5

\[=-\frac{n}{2\sigma^2}+\frac{1}{2(\sigma^2)^2}Q\]

6

\[-\frac{n}{2\sigma^2}+\frac{1}{2(\sigma^2)^2}Q=0\]

Derivative is 0 at the maximum.

7

\[\frac{1}{2(\sigma^2)^2}Q=\frac{n}{2\sigma^2}\]

8

\[Q=n\sigma^2\]

Line 3 of p.12.

9

\[\sigma^2=\frac{1}{n}(\boldsymbol y-\boldsymbol X\boldsymbol\beta)^T(\boldsymbol y-\boldsymbol X\boldsymbol\beta)\]

10

\[\hat\sigma^2_{MLE}=\frac{1}{n}(\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})^T(\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})=\frac1n\boldsymbol e^T\boldsymbol e\]

MLE of \(\boldsymbol\beta\) is \(\hat{\boldsymbol\beta}\) (Lecture 6). ∎

### 02.2 The unbiased estimate of \(\sigma^2\) Slides p.13

**What** Slides p.13

Estimator: a rule that calculates an estimate from the data. It changes with the data, so it is a random variable.

Lecture 7 · p.13
The MLE of \(\sigma^2\) is \(\frac{1}{n}\boldsymbol e^T\boldsymbol e\), where \(\boldsymbol e=\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta}\).

Instead we usually use \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\). Why?

Both estimators use \(\boldsymbol e^T\boldsymbol e\). Only the denominator differs.

\(p+1\) is the number of entries in \(\boldsymbol\beta\). Residual degrees of freedom \(n-(p+1)\): observations minus estimated coefficients. The slide title says \(\hat\sigma^2\) is unbiased.

p.13 only asks "Why?". Unit 04 answers it with \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\). Slides p.14–25 prepare that proof.

**How** Added

Calculate \(\hat\sigma^2\) and \(\hat\sigma\)

- Calculate \(\boldsymbol e^T\boldsymbol e\) (Section 02.1, How steps 1–3).
- Count the coefficients \(p+1\), the columns of \(\boldsymbol X\).
- Calculate \(n-(p+1)\).
- Divide \(\boldsymbol e^T\boldsymbol e\) by \(n-(p+1)\) to get \(\hat\sigma^2\).
- Take the square root to get \(\hat\sigma\).

**Self-check:** \(\hat\sigma^2>\hat\sigma^2_{MLE}\), and \(\hat\sigma^2/\hat\sigma^2_{MLE}=n/(n-(p+1))\).

**Example · Data set 2** Added

1

Sum of squared residuals How step 1

\[\boldsymbol e^T\boldsymbol e=2.70\]

Section 02.1, Example, step 3.

2

Count the coefficients How step 2

\[p+1=1+1=2\]

Columns: intercept and \(x\).

3

Degrees of freedom How step 3

\[n-(p+1)=4-2=2\]

4

Divide How step 4

\[\hat\sigma^2=\frac{2.70}{2}=1.35\]

5

Square root How step 5

\[\hat\sigma=\sqrt{1.35}=1.1619\]

▲

Self-check: \(1.35/0.675=2=4/2\).

### 02.3 Stack \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) into one vector Slides p.14

Joint distribution: the distribution of several random quantities together.

**What** Slides p.14

Lecture 7 · p.14
Recall \(\hat{\boldsymbol\beta}=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\) and \(\boldsymbol e=(\boldsymbol I-\boldsymbol H)\boldsymbol y\).

Let's consider the vector \(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\), which is a linear combination of \(\boldsymbol y\):
\[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]=\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ \boldsymbol I-\boldsymbol H\end{array}\right]\boldsymbol y\]
And since \(\boldsymbol y\sim N(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\), then \(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\) is MVN-distributed.

The stack is \((p+1+n)\times1\). The matrix is \((p+1+n)\times n\). Linear combination of \(\boldsymbol y\): a sum of \(y_1,\dots,y_n\), each times a constant.

**How** Added

Show that a vector is MVN

- Write the vector as \(\boldsymbol M\boldsymbol y+\boldsymbol d\), with \(\boldsymbol M\) and \(\boldsymbol d\) free of \(\boldsymbol y\).
- Confirm \(\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\).
- Use MVN linearity (Lecture 5): \(\boldsymbol M\boldsymbol y+\boldsymbol d\) is MVN.

**Self-check:** \(\boldsymbol M\) has \(n\) columns.

**Self-check:** The first \(p+1\) entries of \(\boldsymbol M\boldsymbol y\) are \(\hat{\boldsymbol\beta}\); the rest are \(\boldsymbol e\).

Here \(\boldsymbol M=\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ \boldsymbol I-\boldsymbol H\end{array}\right]\), \(\boldsymbol d=\boldsymbol 0\). The slides use \(\boldsymbol C\) for a block in Section 02.5, so this page uses \(\boldsymbol M\).

**Example · Data set 2** Added

1

Top block How step 1

\[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\left[\begin{array}{cccc}1&1&1&1\\0&1&2&3\end{array}\right]=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\-0.3&-0.1&0.1&0.3\end{array}\right]\]

Row 1: \(0.7-0.3x_j\). Row 2: \(-0.3+0.2x_j\).

2

Stack the two blocks How step 1

\[\boldsymbol M=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\-0.3&-0.1&0.1&0.3\\ \hline 0.3&-0.4&-0.1&0.2\\-0.4&0.7&-0.2&-0.1\\-0.1&-0.2&0.7&-0.4\\0.2&-0.1&-0.4&0.3\end{array}\right]\]

\(6\times4\): \(p+1+n=6\), \(n=4\). Self-check 1 passes.

3

Row 1 times \(\boldsymbol y\)

\[0.7(1)+0.4(3)+0.1(2)-0.2(5)=0.7+1.2+0.2-1.0=1.1\]

4

Row 2 times \(\boldsymbol y\)

\[-0.3(1)-0.1(3)+0.1(2)+0.3(5)=-0.3-0.3+0.2+1.5=1.1\]

5

Rows 3 to 6 times \(\boldsymbol y\)

\[(\boldsymbol I-\boldsymbol H)\boldsymbol y=(-0.1,\ 0.8,\ -1.3,\ 0.6)^T\]

Row 3: \(0.3-1.2-0.2+1.0=-0.1\). Row 4: \(-0.4+2.1-0.4-0.5=0.8\). Row 5: \(-0.1-0.6+1.4-2.0=-1.3\). Row 6: \(0.2-0.3-0.8+1.5=0.6\).

6

\[\boldsymbol M\boldsymbol y=(1.1,\ 1.1,\ -0.1,\ 0.8,\ -1.3,\ 0.6)^T\]

\(\hat\beta_0=1.1\), \(\hat\beta_1=1.1\), then \(\boldsymbol e\). Self-check 2 passes. ▲

A mean vector and a variance matrix fully specify an MVN.

### 02.4 The mean of the stacked vector Slides p.15–16

\(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\) is known. Only \(E[\boldsymbol e]\) is new.

**What** Slides p.15–16

Lecture 7 · p.16
Recall \(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\). Now,
\[\begin{array}{rl}E[\boldsymbol e]&=E[(\boldsymbol I-\boldsymbol H)\boldsymbol y]\\&=(\boldsymbol I-\boldsymbol H)E[\boldsymbol y]\\&=(\boldsymbol I-\boldsymbol H)\boldsymbol X\boldsymbol\beta\\&=\boldsymbol X\boldsymbol\beta-\boldsymbol H\boldsymbol X\boldsymbol\beta\\&=\boldsymbol X\boldsymbol\beta-\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X\boldsymbol\beta\\&=\boldsymbol X\boldsymbol\beta-\boldsymbol X\boldsymbol\beta=\boldsymbol 0\end{array}\]
So \(E\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]=\left[\begin{array}{c}\boldsymbol\beta\\ \boldsymbol 0\end{array}\right]\).

On average, a residual is 0. The key step is \(\boldsymbol H\boldsymbol X=\boldsymbol X\), which gives \((\boldsymbol I-\boldsymbol H)\boldsymbol X=\boldsymbol 0\).

**How** Added

Find the expected value of a constant matrix times \(\boldsymbol y\)

- Move the constant matrix out: \(E[\boldsymbol M\boldsymbol y]=\boldsymbol M E[\boldsymbol y]\).
- Substitute \(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\) to get \(\boldsymbol M\boldsymbol X\boldsymbol\beta\).
- Calculate \(\boldsymbol M\boldsymbol X\). For \(\boldsymbol M=\boldsymbol I-\boldsymbol H\), the result is \(\boldsymbol 0\).

**Self-check:** \(\boldsymbol M\boldsymbol X\) uses only \(\boldsymbol X\). For \(\boldsymbol M=\boldsymbol I-\boldsymbol H\), each entry is 0.

**Example · Data set 2** Added

One sample cannot give \(E[\boldsymbol e]\). How step 3 uses only \(\boldsymbol X\), so you can check it.

1

Column 1 of \(\boldsymbol X\) is \((1,1,1,1)^T\) How step 3

\[(\boldsymbol I-\boldsymbol H)\left[\begin{array}{c}1\\1\\1\\1\end{array}\right]=\left[\begin{array}{c}0.3-0.4-0.1+0.2\\-0.4+0.7-0.2-0.1\\-0.1-0.2+0.7-0.4\\0.2-0.1-0.4+0.3\end{array}\right]=\left[\begin{array}{c}0\\0\\0\\0\end{array}\right]\]

2

Column 2 of \(\boldsymbol X\) is \((0,1,2,3)^T\) How step 3

\[(\boldsymbol I-\boldsymbol H)\left[\begin{array}{c}0\\1\\2\\3\end{array}\right]=\left[\begin{array}{c}0-0.4-0.2+0.6\\0+0.7-0.4-0.3\\0-0.2+1.4-1.2\\0-0.1-0.8+0.9\end{array}\right]=\left[\begin{array}{c}0\\0\\0\\0\end{array}\right]\]

3

\[E[\boldsymbol e]=(\boldsymbol I-\boldsymbol H)\boldsymbol X\boldsymbol\beta=\boldsymbol 0\,\boldsymbol\beta=\boldsymbol 0\]

True for each \(\boldsymbol\beta\). ▲

**Why** Slides p.15–16

1

\[E[\boldsymbol e]=E[(\boldsymbol I-\boldsymbol H)\boldsymbol y]\]

Substitute \(\boldsymbol e\).

2

\[=(\boldsymbol I-\boldsymbol H)E[\boldsymbol y]\]

\(\boldsymbol I-\boldsymbol H\) is constant (Lecture 5).

3

\[=(\boldsymbol I-\boldsymbol H)\boldsymbol X\boldsymbol\beta\]

\(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\).

4

\[=\boldsymbol X\boldsymbol\beta-\boldsymbol H\boldsymbol X\boldsymbol\beta\]

5

\[=\boldsymbol X\boldsymbol\beta-\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X\boldsymbol\beta\]

Substitute \(\boldsymbol H\).

6

\[=\boldsymbol X\boldsymbol\beta-\boldsymbol X\boldsymbol I\boldsymbol\beta\]

\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).

7

\[=\boldsymbol X\boldsymbol\beta-\boldsymbol X\boldsymbol\beta\]

8

\[=\boldsymbol 0\]

With \(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\): \(E\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]=\left[\begin{array}{c}\boldsymbol\beta\\ \boldsymbol 0\end{array}\right]\). ∎

### 02.5 The variance matrix of the stacked vector Slides p.17–18

**What** Slides p.17–18

Lecture 7 · p.18 \[\begin{array}{rl}\mathrm{Var}\left(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\right)&=\mathrm{Var}\left(\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]\boldsymbol y\right)\\ &=\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]\mathrm{Var}(\boldsymbol y)\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]^T\\ &=\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ (\boldsymbol I-\boldsymbol H)\end{array}\right](\sigma^2\boldsymbol I)\left[((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T)^T,\ (\boldsymbol I-\boldsymbol H)^T\right]\\ &=\sigma^2\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]\left[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1},\ (\boldsymbol I-\boldsymbol H)\right]\\ &=\sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1} & (\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol I-\boldsymbol H)\\ (\boldsymbol I-\boldsymbol H)\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1} & (\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)\end{array}\right]\\ &=\sigma^2\left[\begin{array}{cc}\boldsymbol A & \boldsymbol B\\ \boldsymbol C & \boldsymbol D\end{array}\right]\end{array}\]

\(\sigma^2\boldsymbol A\): \(\mathrm{Var}(\hat{\boldsymbol\beta})\). Size \((p+1)\times(p+1)\); example \(2\times2\).

\(\sigma^2\boldsymbol B\): covariances of \(\hat{\boldsymbol\beta}\) with \(\boldsymbol e\). Size \((p+1)\times n\); example \(2\times4\).

\(\sigma^2\boldsymbol C\): covariances of \(\boldsymbol e\) with \(\hat{\boldsymbol\beta}\). Size \(n\times(p+1)\); example \(4\times2\).

\(\sigma^2\boldsymbol D\): \(\mathrm{Var}(\boldsymbol e)\). Size \(n\times n\); example \(4\times4\).

**How** Added

Find \(\mathrm{Var}(\boldsymbol M\boldsymbol y)\) and split it into blocks

- Use linearity (Lecture 5): \(\mathrm{Var}(\boldsymbol M\boldsymbol y)=\boldsymbol M\,\mathrm{Var}(\boldsymbol y)\,\boldsymbol M^T\).
- Substitute \(\mathrm{Var}(\boldsymbol y)=\sigma^2\boldsymbol I\) to get \(\sigma^2\boldsymbol M\boldsymbol M^T\).
- Split \(\boldsymbol M\) into top and bottom blocks. \(\boldsymbol M^T\) then has left and right blocks.
- Multiply each row block by each column block to get \(\boldsymbol A\), \(\boldsymbol B\), \(\boldsymbol C\), \(\boldsymbol D\).

**Self-check:** \(\boldsymbol A\) is \((p+1)\times(p+1)\). \(\boldsymbol D\) is \(n\times n\).

**Example · Data set 2** Added

One entry of each block of \(\boldsymbol M\boldsymbol M^T\), with \(\boldsymbol M\) from Section 02.3.

1

Block \(\boldsymbol A\), entry (2,2): row 2 of \(\boldsymbol M\) times row 2 How step 4

\[(-0.3)^2+(-0.1)^2+0.1^2+0.3^2=0.09+0.01+0.01+0.09=0.2\]

Equals \(V_{11}=0.2\).

2

Block \(\boldsymbol B\), entry (1,1): row 1 of \(\boldsymbol M\) times row 3 How step 4

\[0.7(0.3)+0.4(-0.4)+0.1(-0.1)+(-0.2)(0.2)=0.21-0.16-0.01-0.04=0\]

3

Block \(\boldsymbol C\), entry (1,2): row 3 of \(\boldsymbol M\) times row 2 How step 4

\[0.3(-0.3)+(-0.4)(-0.1)+(-0.1)(0.1)+0.2(0.3)=-0.09+0.04-0.01+0.06=0\]

4

Block \(\boldsymbol D\), entry (1,1): row 3 of \(\boldsymbol M\) times row 3 How step 4

\[0.3^2+(-0.4)^2+(-0.1)^2+0.2^2=0.09+0.16+0.01+0.04=0.3\]

Equals entry (1,1) of \(\boldsymbol I-\boldsymbol H\). ▲

**Why** Slides p.17–18

Let \(\boldsymbol L=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\).

1

\[\mathrm{Var}\left(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\right)=\mathrm{Var}\left(\left[\begin{array}{c}\boldsymbol L\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]\boldsymbol y\right)\]

Stacked form of p.14.

2

\[=\left[\begin{array}{c}\boldsymbol L\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]\mathrm{Var}(\boldsymbol y)\left[\begin{array}{c}\boldsymbol L\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]^T\]

Linearity (Lecture 5).

3

\[=\left[\begin{array}{c}\boldsymbol L\\ (\boldsymbol I-\boldsymbol H)\end{array}\right](\sigma^2\boldsymbol I)\left[\boldsymbol L^T,\ (\boldsymbol I-\boldsymbol H)^T\right]\]

\(\mathrm{Var}(\boldsymbol y)=\sigma^2\boldsymbol I\); transpose turns a stack into a pair.

4

\[=\sigma^2\left[\begin{array}{c}\boldsymbol L\\ (\boldsymbol I-\boldsymbol H)\end{array}\right]\left[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1},\ (\boldsymbol I-\boldsymbol H)^T\right]\]

\(\boldsymbol L^T=\boldsymbol X((\boldsymbol X^T\boldsymbol X)^{-1})^T=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\).

5

\[=\sigma^2\left[\begin{array}{cc}\boldsymbol L\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1} & \boldsymbol L(\boldsymbol I-\boldsymbol H)^T\\ (\boldsymbol I-\boldsymbol H)\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1} & (\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)^T\end{array}\right]\]

Block multiplication.

6

\[=\sigma^2\left[\begin{array}{cc}\boldsymbol A & \boldsymbol B\\ \boldsymbol C & \boldsymbol D\end{array}\right]\]

p.18 writes \(\boldsymbol B=\boldsymbol L(\boldsymbol I-\boldsymbol H)\); \((\boldsymbol I-\boldsymbol H)^T=\boldsymbol I-\boldsymbol H\) (Section 02.6). ∎

### 02.6 Simplify the blocks \(\boldsymbol A\), \(\boldsymbol B\), \(\boldsymbol D\), and the Aside Slides p.19–22

**What** Slides p.19–22

Lecture 7 · p.22 \[\boldsymbol A=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}=(\boldsymbol X^T\boldsymbol X)^{-1}\] \[\begin{array}{rl}\boldsymbol B&=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol I-\boldsymbol H)\\&=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T-(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T)\\&=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T-(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\&=\boldsymbol 0\end{array}\]
Aside:
\[\begin{array}{rl}\boldsymbol H\boldsymbol H^T&=[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T][\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T]^T\\&=[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T][\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T]\\&=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T=\boldsymbol H\end{array}\] \[\begin{array}{rl}\boldsymbol D&=(\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)^T\\&=(\boldsymbol I\boldsymbol I^T-\boldsymbol I\boldsymbol H^T-\boldsymbol H\boldsymbol I^T+\boldsymbol H\boldsymbol H^T)\\&=(\boldsymbol I-2\boldsymbol H+\boldsymbol H)\\&=(\boldsymbol I-\boldsymbol H)\end{array}\]

\(\boldsymbol A=(\boldsymbol X^T\boldsymbol X)^{-1}\): \(\mathrm{Var}(\hat{\boldsymbol\beta})=\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\), as in Lecture 6.

\(\boldsymbol B=\boldsymbol 0\): each entry of \(\hat{\boldsymbol\beta}\) has covariance 0 with each entry of \(\boldsymbol e\).

Aside: \(\boldsymbol H\boldsymbol H^T=\boldsymbol H\). The derivation also shows \(\boldsymbol H^T=\boldsymbol H\), so \(\boldsymbol H\boldsymbol H=\boldsymbol H\).

\(\boldsymbol D=\boldsymbol I-\boldsymbol H\): \(\mathrm{Var}(\boldsymbol e)=\sigma^2(\boldsymbol I-\boldsymbol H)\).

\(\boldsymbol C\): not on the slides. A variance matrix is symmetric, so \(\boldsymbol C=\boldsymbol B^T=\boldsymbol 0\). Added

**How** Added

Simplify a product that contains \(\boldsymbol H\)

- Write each \(\boldsymbol H\) as \(\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\).
- Find each adjacent pair \((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X\) or \(\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\).
- Replace each pair with \(\boldsymbol I\).
- Use \(\boldsymbol H^T=\boldsymbol H\) and \(\boldsymbol I^T=\boldsymbol I\).
- Replace \(\boldsymbol H\boldsymbol H\) or \(\boldsymbol H\boldsymbol H^T\) with \(\boldsymbol H\) (Aside).

**Self-check:** One entry of the product is the same before and after.

**Example · Data set 2** Added

1

Block \(\boldsymbol A\): first find \(\boldsymbol L\boldsymbol X\)

\[\boldsymbol L\boldsymbol X=\left[\begin{array}{cc}0.7+0.4+0.1-0.2&0+0.4+0.2-0.6\\-0.3-0.1+0.1+0.3&0-0.1+0.2+0.9\end{array}\right]=\left[\begin{array}{cc}1&0\\0&1\end{array}\right]\]

\(\boldsymbol A=\boldsymbol I\boldsymbol V=\boldsymbol V=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\).

2

Block \(\boldsymbol B=\boldsymbol L(\boldsymbol I-\boldsymbol H)\): row 2 of \(\boldsymbol L\) times column 2 of \(\boldsymbol I-\boldsymbol H\)

\[-0.3(-0.4)+(-0.1)(0.7)+0.1(-0.2)+0.3(-0.1)=0.12-0.07-0.02-0.03=0\]

The other 7 entries are also 0.

3

Aside: entry (1,2) of \(\boldsymbol H\boldsymbol H^T\), row 1 of \(\boldsymbol H\) times row 2

\[0.7(0.4)+0.4(0.3)+0.1(0.2)+(-0.2)(0.1)=0.28+0.12+0.02-0.02=0.4\]

Equals \(h_{12}=0.4\).

4

Block \(\boldsymbol D\): entry (1,2), row 1 of \(\boldsymbol I-\boldsymbol H\) times row 2

\[0.3(-0.4)+(-0.4)(0.7)+(-0.1)(-0.2)+0.2(-0.1)=-0.12-0.28+0.02-0.02=-0.4\]

Equals entry (1,2) of \(\boldsymbol I-\boldsymbol H\). ▲

**Why** Slides p.19–22

Block \(\boldsymbol A\):

1

\[\boldsymbol A=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

2

\[=\boldsymbol I(\boldsymbol X^T\boldsymbol X)^{-1}\]

\((\boldsymbol X^T\boldsymbol X)^{-1}(\boldsymbol X^T\boldsymbol X)=\boldsymbol I\).

3

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\]

∎

Block \(\boldsymbol B\):

1

\[\boldsymbol B=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol I-\boldsymbol H)\]

2

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T-(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol H\]

3

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T-(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T)\]

Substitute \(\boldsymbol H\).

4

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T-\boldsymbol I(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).

5

\[=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T-(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

6

\[=\boldsymbol 0\]

∎

Aside \(\boldsymbol H\boldsymbol H^T=\boldsymbol H\). Rule: \((\boldsymbol P\boldsymbol Q\boldsymbol S)^T=\boldsymbol S^T\boldsymbol Q^T\boldsymbol P^T\).

1

\[\boldsymbol H\boldsymbol H^T=[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T][\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T]^T\]

2

\[=[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T][(\boldsymbol X^T)^T((\boldsymbol X^T\boldsymbol X)^{-1})^T\boldsymbol X^T]\]

Transpose of a product: reverse the order.

3

\[=[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T][\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T]\]

Inverse of symmetric \(\boldsymbol X^T\boldsymbol X\) is symmetric. Shows \(\boldsymbol H^T=\boldsymbol H\).

4

\[=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}(\boldsymbol X^T\boldsymbol X)(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

Associative.

5

\[=\boldsymbol X\boldsymbol I(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

6

\[=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T=\boldsymbol H\]

∎

Block \(\boldsymbol D\):

1

\[\boldsymbol D=(\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)^T\]

2

\[=(\boldsymbol I-\boldsymbol H)(\boldsymbol I^T-\boldsymbol H^T)\]

Transpose each term.

3

\[=\boldsymbol I\boldsymbol I^T-\boldsymbol I\boldsymbol H^T-\boldsymbol H\boldsymbol I^T+\boldsymbol H\boldsymbol H^T\]

4

\[=\boldsymbol I-\boldsymbol H-\boldsymbol H+\boldsymbol H\boldsymbol H^T\]

\(\boldsymbol I^T=\boldsymbol I\), \(\boldsymbol H^T=\boldsymbol H\).

5

\[=\boldsymbol I-2\boldsymbol H+\boldsymbol H\]

Aside: \(\boldsymbol H\boldsymbol H^T=\boldsymbol H\).

6

\[=\boldsymbol I-\boldsymbol H\]

∎

### 02.7 Implications Slides p.23–25

**What** Slides p.23–25

Lecture 7 · p.25 \[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\sim N\left(\left[\begin{array}{c}\boldsymbol\beta\\ \boldsymbol 0\end{array}\right],\ \sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1} & \boldsymbol 0\\ \boldsymbol 0 & (\boldsymbol I-\boldsymbol H)\end{array}\right]\right)\]

- \(\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1})\)
- \(\boldsymbol e\sim N(\boldsymbol 0,\sigma^2(\boldsymbol I-\boldsymbol H))\)
- \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) are independent

[figure]
Figure 02.2 · Variance matrix of \(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\) for data set 2, without \(\sigma^2\). Top-left: \((\boldsymbol X^T\boldsymbol X)^{-1}\). Bottom-right: \(\boldsymbol I-\boldsymbol H\). Grey blocks: zeros.

Each result uses one MVN property (Lecture 5).

Marginal distribution property: a part of an MVN vector is MVN. The top part gives \(\hat{\boldsymbol\beta}\), with variance \(\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\).

The bottom part gives \(\boldsymbol e\), with variance \(\sigma^2(\boldsymbol I-\boldsymbol H)\). \(\mathrm{Var}(e_i)=\sigma^2(1-h_{ii})\); \(h_{ii}\) is diagonal entry \(i\) of \(\boldsymbol H\).

Independent: one value does not change the distribution of the other. MVN parts with zero covariance are independent.

**How** Added

Estimate \(\mathrm{Var}(\hat{\boldsymbol\beta})\) and \(\mathrm{Var}(e_i)\)

- Calculate \(\hat\sigma^2=\boldsymbol e^T\boldsymbol e/(n-(p+1))\) (Section 02.2).
- Calculate \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\).
- Estimate \(\mathrm{Var}(\hat\beta_j)\) as \(\hat\sigma^2V_{jj}\).
- Take its square root: the standard error of \(\hat\beta_j\).
- Estimate \(\mathrm{Var}(e_i)\) as \(\hat\sigma^2(1-h_{ii})\).

**Self-check:** Each variance is positive. Each \(1-h_{ii}\) is between 0 and 1.

**Example · Data set 2** Added

1

Estimate \(\sigma^2\) How step 1

\[\hat\sigma^2=1.35\]

Section 02.2.

2

Read \(V_{11}\) How step 2

\[V_{11}=0.2\]

Second diagonal entry (0-indexing).

3

Variance of \(\hat\beta_1\) How step 3

\[\widehat{\mathrm{Var}}(\hat\beta_1)=\hat\sigma^2V_{11}=1.35\times0.2=0.27\]

4

Standard error How step 4

\[\mathrm{SE}(\hat\beta_1)=\sqrt{0.27}=0.5196\]

5

Variances of \(e_1\) and \(e_2\) How step 5

\[\widehat{\mathrm{Var}}(e_1)=1.35(1-0.7)=1.35\times0.3=0.405,\qquad \widehat{\mathrm{Var}}(e_2)=1.35(1-0.3)=1.35\times0.7=0.945\]

▲

The end points \(x=0\) and \(x=3\) have larger \(h_{ii}\), so their residuals have smaller variance.

**Why** Slides p.23–25

1

\[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\ \text{is MVN}\]

Section 02.3 (p.14).

2

\[E\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]=\left[\begin{array}{c}\boldsymbol\beta\\ \boldsymbol 0\end{array}\right]\]

Section 02.4 (p.16).

3

\[\mathrm{Var}\left(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\right)=\sigma^2\left[\begin{array}{cc}\boldsymbol A & \boldsymbol B\\ \boldsymbol C & \boldsymbol D\end{array}\right]\]

Section 02.5 (p.18).

4

\[=\sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1} & \boldsymbol 0\\ \boldsymbol 0 & (\boldsymbol I-\boldsymbol H)\end{array}\right]\]

Section 02.6 (p.19–22).

5

\[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\sim N\left(\left[\begin{array}{c}\boldsymbol\beta\\ \boldsymbol 0\end{array}\right],\ \sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1} & \boldsymbol 0\\ \boldsymbol 0 & (\boldsymbol I-\boldsymbol H)\end{array}\right]\right)\]

Mean and variance specify an MVN.

6

\[\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}),\qquad \boldsymbol e\sim N(\boldsymbol 0,\sigma^2(\boldsymbol I-\boldsymbol H))\]

Marginal distribution property.

7

\[\hat{\boldsymbol\beta}\ \text{and}\ \boldsymbol e\ \text{are independent}\]

Zero covariance gives independence. ∎

\(\hat\sigma^2\) uses only \(\boldsymbol e\), so \(\hat\sigma^2\) is also independent of \(\hat{\boldsymbol\beta}\). Unit 03 uses this for the t statistic.

### 02.8 Practice Added

**Q1.** A regression has \(n=12\), \(p=2\), and \(\boldsymbol e^T\boldsymbol e=45\). Calculate \(\hat\sigma^2_{MLE}\), \(\hat\sigma^2\), and \(\hat\sigma\).

Answer

\(\hat\sigma^2_{MLE}=\frac{1}{n}\boldsymbol e^T\boldsymbol e=\frac{45}{12}=3.75\).

\(n-(p+1)=12-(2+1)=9\), so \(\hat\sigma^2=\frac{45}{9}=5\).

\(\hat\sigma=\sqrt{5}=2.2361\).

Self-check: \(5/3.75=1.3333=12/9\).

**Q2.** Show that the lower-left block \(\boldsymbol C=(\boldsymbol I-\boldsymbol H)\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\) of \(\mathrm{Var}\left(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\right)/\sigma^2\) is \(\boldsymbol 0\).

Answer

1

\[\boldsymbol C=(\boldsymbol I-\boldsymbol H)\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

2

\[=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}-\boldsymbol H\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

3

\[=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}-\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\]

Substitute \(\boldsymbol H\).

4

\[=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}-\boldsymbol X\boldsymbol I(\boldsymbol X^T\boldsymbol X)^{-1}\]

\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).

5

\[=\boldsymbol 0\]

∎ Or: \(\boldsymbol C=\boldsymbol B^T=\boldsymbol 0^T=\boldsymbol 0\).

**Q3.** Data set 2: \(h_{33}=0.3\), \(\hat\sigma^2=1.35\). Give the distribution of \(e_3\). Estimate \(\mathrm{Var}(e_3)\). Is \(e_3\) independent of \(\hat\beta_1\)?

Answer

Entry 3 of \(\boldsymbol e\sim N(\boldsymbol 0,\sigma^2(\boldsymbol I-\boldsymbol H))\): \(e_3\sim N(0,\ \sigma^2(1-h_{33}))=N(0,\ 0.7\,\sigma^2)\).

\(\widehat{\mathrm{Var}}(e_3)=1.35\times0.7=0.945\).

Yes. Their covariance is an entry of \(\sigma^2\boldsymbol B=\boldsymbol 0\), and the joint distribution is MVN.

## 03 · The t statistic in MLR and property (3): independence

Plan · Slides p.26–28

Step 1: Symbols and distributions (Added).

Step 2: The SLR t statistic: three facts and one combination rule (Slides p.26).

Step 3: The same plan in MLR, and the properties still to prove (Slides p.27).

Step 4: Prove property (3) (Slides p.28). Unit 04 proves property (2).

### 03.1 Symbols and distributions Added

Random variable: a number whose value comes from a random experiment, for example \(\hat\beta_1\), \(\hat\sigma^2\), \(e_i\).

Standard normal \(N(0,1)\): mean 0, variance 1. If \(W\sim N(\mu,\sigma^2)\), the standardized value \((W-\mu)/\sigma\sim N(0,1)\).

Chi-squared \(\chi^2_\nu\): the distribution of a sum of \(\nu\) squared independent \(N(0,1)\) variables. \(\nu\) is the degrees of freedom (Slides p.29).

t distribution \(t_\nu\): bell-shaped and symmetric about 0, with thicker tails than \(N(0,1)\). Slides p.26 defines it as \(Z/\sqrt{V/\nu}\).

SLR (simple linear regression) model: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\). \(S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2\).

Plain \(V\): the chi-squared quantity \(\frac{1}{\sigma^2}\sum e_i^2\) of Slides p.26. Bold \(\boldsymbol V\): the matrix \((\boldsymbol X^T\boldsymbol X)^{-1}\).

Standard error (SE): the standard deviation of an estimator, with \(\hat\sigma\) for \(\sigma\). SLR: \(\mathrm{SE}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}\). MLR: \(\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\).

### 03.2 Recall: the t statistic in SLR Slides p.26

**What** Slides p.26

Recall: OLS in Simple Linear Regression · Lecture 7 · p.26 \[\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}\sim N(0,1)\] But \(\sigma\) is unknown in practice, so we actually need the distribution when we replace \(\sigma\) by \(\hat\sigma\).
And we used the following: (we showed 1 but only claimed 2 & 3)
1. \(Z=\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}\sim N(0,1)\)
2. \(V=\dfrac{1}{\sigma^2}\sum_{i=1}^ne_i^2\sim\chi^2_{(n-2)}\)
3. \(Z\) and \(V\) are independent
Then if (1) \(Z\sim N(0,1)\), (2) \(V\sim\chi^2_\nu\) and (3) \(V\) independent of \(Z\): \(\Longrightarrow\dfrac{Z}{\sqrt{V/\nu}}\sim t_\nu\) \[\Longrightarrow\frac{Z}{\sqrt{V/\nu}}=\frac{\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\sqrt{\dfrac{1}{\sigma^2}\sum_{i=1}^ne_i^2/(n-2)}}\sim t_{n-2}\] \[\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{n-2}\]

With \(\hat\sigma\) for the unknown \(\sigma\), the denominator is random and the distribution is not \(N(0,1)\).

\(Z\) uses only \(\hat\beta_1\). It contains the true \(\sigma\), so it is used only in the proof.

\(V\) uses only the residuals. It has \(n-2\) degrees of freedom because SLR estimates 2 coefficients.

Combination rule: an \(N(0,1)\) divided by an independent \(\sqrt{\chi^2_\nu/\nu}\) has the \(t_\nu\) distribution.

Lecture 3 proved fact 1 only. This lecture proves facts 2 and 3 for MLR. SLR is the case \(p=1\).

[figure]
Figure 03.1 · Added. Densities of \(N(0,1)\) (blue) and \(t_3\) (purple, dashed). At 0: 0.399 and 0.368. At 3: 0.0044 and 0.0230.

The random \(\hat\sigma\) adds variation, so large values of the statistic occur more often.

**How** Added

Get a t statistic from the three facts and the combination rule

- Write \(Z\): (estimator − parameter) / standard deviation with the true \(\sigma\). Confirm \(Z\sim N(0,1)\).
- Write \(V\): residual sum of squares / \(\sigma^2\). Confirm \(V\sim\chi^2_\nu\). Record \(\nu\).
- Confirm that \(Z\) and \(V\) are independent.
- Put them into \(Z/\sqrt{V/\nu}\). Cancel \(\sigma\). Write \(\sum e_i^2/\nu\) as \(\hat\sigma^2\). The result is \(t_\nu\).

**Self-check:** (1) The result has \(\hat\sigma\), not \(\sigma\). (2) The t degrees of freedom equal those of \(V\). (3) The denominator is \(\mathrm{SE}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}\).

**Example · Brain and head data** Numbers from Lecture 3 · p.45

\(y\): brain weight. \(x\): head size. Lecture 3 p.45 prints:

\(\hat\beta_1=0.26082\) and \(\mathrm{SE}(\hat\beta_1)=0.01307\).

\(\hat\sigma=72.35\), with 234 degrees of freedom.

1

Find \(n\) from the degrees of freedom How step 2

\[n-2=234\]

\[n=236\]

2

Find \(\sqrt{S_{xx}}\) from the SE How step 4

\[\mathrm{SE}(\hat\beta_1)=\frac{\hat\sigma}{\sqrt{S_{xx}}}\]

\[\sqrt{S_{xx}}=\frac{\hat\sigma}{\mathrm{SE}(\hat\beta_1)}=\frac{72.35}{0.01307}=5535.58\]

Approximate: the printed values are rounded.

3

Write the statistic and its distribution How step 4

\[\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}=\frac{0.26082-\beta_1}{0.01307}\sim t_{234}\]

4

Substitute \(\beta_1=0\)

\[\frac{0.26082-0}{0.01307}=19.956\]

**Self-check:** Lecture 3 p.44–45 prints 19.957. The difference comes from rounded inputs.

**Why** Slides p.26

1

\[\frac{Z}{\sqrt{V/(n-2)}}=\frac{\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\sqrt{\dfrac{1}{\sigma^2}\sum_{i=1}^ne_i^2/(n-2)}}\]

substitute \(Z\) and \(V\), \(\nu=n-2\)

2

\[=\frac{\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\dfrac{1}{\sigma}\sqrt{\sum_{i=1}^ne_i^2/(n-2)}}\]

3

\[=\frac{\dfrac{\hat\beta_1-\beta_1}{1/\sqrt{S_{xx}}}}{\sqrt{\sum_{i=1}^ne_i^2/(n-2)}}\]

4

\[=\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\]

\(\hat\sigma=\sqrt{\sum_{i=1}^ne_i^2/(n-2)}\)

5

\[\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{n-2}\]

combination rule; needs facts 1, 2, 3 ∎

### 03.3 The matching result in MLR Slides p.27

**What** Slides p.27

In Multiple Linear Regression · Lecture 7 · p.27 Similarly, \(\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1})\Longrightarrow\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\Longrightarrow\dfrac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}\sim N(0,1)\)
but \(\sigma\) is unknown in practice, so we need the distribution where we've replaced \(\sigma\) by \(\hat\sigma\).
We have shown:
1. \(\dfrac{\hat\beta_1-\beta_1}{\sqrt{\sigma^2V_{jj}}}\sim N(0,1)\)
If we can further show:
2. \(\dfrac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\)
3. \(\hat{\boldsymbol\beta}\) and \(\hat{\boldsymbol e}\) are independent
Then by the same argument we have: \[\frac{\dfrac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}}{\sqrt{\left(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\right)/(n-(p+1))}}\sim t_{n-(p+1)}\] \[\frac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\sim t_{n-(p+1)}\] Let's (finally) prove properties (2) & (3)!

Note · slide notation

Item 1 prints \(\hat\beta_1-\beta_1\); it means \(\hat\beta_j-\beta_j\). In item 3, \(\hat{\boldsymbol e}\) is \(\boldsymbol e\).

Property (1) matches SLR fact 1

\(Z=\dfrac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}\sim N(0,1)\) (Slides p.5). Element \(j\) of an MVN vector is normal.

Property (2) matches SLR fact 2

\(V=\dfrac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\): MLR estimates \(p+1\) coefficients. Proof in Unit 04 (Slides p.29–43).

Property (3) matches SLR fact 3

\(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) are independent. \(Z\) uses only \(\hat{\boldsymbol\beta}\); \(V\) uses only \(\boldsymbol e\). Proof in Section 03.4 (Slides p.28).

For \(p=1\): \(n-(p+1)=n-2\) and \(\sqrt{\sigma^2V_{11}}=\sigma/\sqrt{S_{xx}}\), the SLR result.

**How** Added

Calculate the t statistic for \(\hat\beta_j\) in MLR

- Write \(\boldsymbol X\). Calculate \(\boldsymbol X^T\boldsymbol X\) and \(\boldsymbol X^T\boldsymbol y\).
- Calculate \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\). Find \(V_{jj}\).
- Calculate \(\hat{\boldsymbol\beta}=\boldsymbol V\boldsymbol X^T\boldsymbol y\).
- Calculate \(\hat{\boldsymbol y}=\boldsymbol X\hat{\boldsymbol\beta}\) and \(\boldsymbol e=\boldsymbol y-\hat{\boldsymbol y}\).
- Calculate \(\boldsymbol e^T\boldsymbol e\) and \(\hat\sigma^2=\boldsymbol e^T\boldsymbol e/(n-(p+1))\).
- Calculate \(\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\) and \(\dfrac{\hat\beta_j-\beta_j}{\mathrm{SE}(\hat\beta_j)}\).
- Use the \(t_{n-(p+1)}\) distribution.

**Self-check:** (1) \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). (2) \(V_{jj}>0\). (3) Degrees of freedom: \(n\) minus the columns of \(\boldsymbol X\).

**Example · Data set 3** Added

\(n=6\), \(p=2\). Part 1 does the How steps. Part 2 assumes true values and shows \(Z/\sqrt{V/\nu}\) equals the t statistic.

\(x_1=(-1,-1,0,0,1,1)\), \(x_2=(-1,1,-1,1,-1,1)\), \(\boldsymbol y=(1,3,6,8,5,7)^T\).

**Part 1: the t statistic for \(\hat\beta_1\).**

1

Write \(\boldsymbol X\), \(\boldsymbol X^T\boldsymbol X\), and \(\boldsymbol X^T\boldsymbol y\) How step 1

\[\boldsymbol X=\left[\begin{array}{ccc}1&-1&-1\\1&-1&1\\1&0&-1\\1&0&1\\1&1&-1\\1&1&1\end{array}\right]\]

\[\boldsymbol X^T\boldsymbol X=\left[\begin{array}{ccc}6&0&0\\0&4&0\\0&0&6\end{array}\right]\]

Diagonal: \(1\cdot6=6\); \(1+1+0+0+1+1=4\); \(6\times1=6\). Off-diagonal: \(\sum x_{i1}=0\), \(\sum x_{i2}=0\), \(\sum x_{i1}x_{i2}=1-1+0+0-1+1=0\).

\[\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}1+3+6+8+5+7\\-1-3+0+0+5+7\\-1+3-6+8-5+7\end{array}\right]=\left[\begin{array}{c}30\\8\\6\end{array}\right]\]

2

Calculate \(\boldsymbol V\) and \(V_{11}\) How step 2

\[\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}=\left[\begin{array}{ccc}1/6&0&0\\0&1/4&0\\0&0&1/6\end{array}\right]\]

\[V_{11}=\frac14=0.25\]

Diagonal matrix: invert each diagonal entry.

3

Calculate \(\hat{\boldsymbol\beta}\) How step 3

\[\hat{\boldsymbol\beta}=\boldsymbol V\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}30/6\\8/4\\6/6\end{array}\right]=\left[\begin{array}{c}5\\2\\1\end{array}\right]\]

4

Fitted values and residuals How step 4

\[\hat y_i=5+2x_{i1}+1x_{i2}\]

\[\hat{\boldsymbol y}=(5-2-1,\ 5-2+1,\ 5+0-1,\ 5+0+1,\ 5+2-1,\ 5+2+1)^T=(2,4,4,6,6,8)^T\]

\[\boldsymbol e=\boldsymbol y-\hat{\boldsymbol y}=(1-2,\ 3-4,\ 6-4,\ 8-6,\ 5-6,\ 7-8)^T=(-1,-1,2,2,-1,-1)^T\]

5

Calculate \(\boldsymbol e^T\boldsymbol e\) and \(\hat\sigma^2\) How step 5

\[\boldsymbol e^T\boldsymbol e=1+1+4+4+1+1=12\]

\[\hat\sigma^2=\frac{\boldsymbol e^T\boldsymbol e}{n-(p+1)}=\frac{12}{6-3}=\frac{12}{3}=4\]

6

SE and the statistic for a stated \(\beta_1=1\) How step 6

\[\mathrm{SE}(\hat\beta_1)=\sqrt{\hat\sigma^2V_{11}}=\sqrt{4\times0.25}=\sqrt1=1\]

\[\frac{\hat\beta_1-\beta_1}{\sqrt{\hat\sigma^2V_{11}}}=\frac{2-1}{1}=1\]

7

State the distribution How step 7

\[\frac{\hat\beta_1-\beta_1}{\sqrt{\hat\sigma^2V_{11}}}\sim t_{n-(p+1)}=t_{6-3}=t_3\]

**Self-check:** \(\boldsymbol X^T\boldsymbol e=(-1-1+2+2-1-1,\ 1+1+0+0-1-1,\ 1-1-2+2+1-1)^T=(0,0,0)^T\).

**Part 2: the same statistic as \(Z/\sqrt{V/\nu}\).** Assume the true values \(\beta_1=1\) and \(\sigma=1\).

1

\(Z\) with the true \(\sigma=1\) Slides p.27, property (1)

\[Z=\frac{\hat\beta_1-\beta_1}{\sqrt{\sigma^2V_{11}}}=\frac{2-1}{\sqrt{1\times0.25}}=\frac{1}{0.5}=2\]

2

\(V\) with the true \(\sigma=1\) Slides p.27, property (2)

\[V=\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\frac{12}{1}=12\]

3

Combine with \(\nu=3\) Slides p.27, combination rule

\[\frac{Z}{\sqrt{V/3}}=\frac{2}{\sqrt{12/3}}=\frac{2}{\sqrt4}=\frac22=1\]

**Self-check:** 1, as in Part 1, step 6, but Part 1 needs no true \(\sigma\).

\(Z=2\) uses the true \(\sigma=1\). The t statistic uses \(\hat\sigma=2\).

**Why** Slides p.27

1

\[\hat{\boldsymbol\beta}\sim N\big(\boldsymbol\beta,\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\big)\]

Slides p.5

2

\[\hat\beta_j\sim N\big(\beta_j,\sigma^2V_{jj}\big)\]

one element of an MVN vector; variance is the diagonal element

3

\[Z=\frac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}\sim N(0,1)\]

standardize

4

\[\frac{Z}{\sqrt{V/(n-(p+1))}}=\frac{\dfrac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}}{\sqrt{\left(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\right)/(n-(p+1))}}\]

substitute \(V\), \(\nu=n-(p+1)\)

5

\[=\frac{\dfrac{\hat\beta_j-\beta_j}{\sigma\sqrt{V_{jj}}}}{\dfrac{1}{\sigma}\sqrt{\boldsymbol e^T\boldsymbol e/(n-(p+1))}}\]

6

\[=\frac{\hat\beta_j-\beta_j}{\sqrt{V_{jj}}\sqrt{\boldsymbol e^T\boldsymbol e/(n-(p+1))}}\]

7

\[=\frac{\hat\beta_j-\beta_j}{\sqrt{V_{jj}}\sqrt{\hat\sigma^2}}\]

\(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\) (Slides p.13)

8

\[=\frac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\]

9

\[\frac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\sim t_{n-(p+1)}\]

combination rule; needs properties (1), (2), (3) ∎

### 03.4 Property (3): \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) are independent Slides p.28

**What** Slides p.28

\(\hat\sigma^2\): (3) Independence · Lecture 7 · p.28 Recall: \[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\\boldsymbol e\end{array}\right]\sim N\left(\left[\begin{array}{c}\boldsymbol\beta\\\boldsymbol 0\end{array}\right],\ \sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1}&\boldsymbol 0\\\boldsymbol 0&(\boldsymbol I-\boldsymbol H)\end{array}\right]\right)\] \[\Longrightarrow \boldsymbol e\text{ is independent of }\hat{\boldsymbol\beta}\] \[\Longrightarrow\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\text{ is independent of }\hat{\boldsymbol\beta}\]

Top-left block \(\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1}\): \(\mathrm{Var}(\hat{\boldsymbol\beta})\).

Bottom-right block \(\sigma^2(\boldsymbol I-\boldsymbol H)\): \(\mathrm{Var}(\boldsymbol e)\).

Off-diagonal blocks \(\boldsymbol 0\): \(\mathrm{Cov}(\hat{\boldsymbol\beta},\boldsymbol e)\), all 0.

[figure]
Figure 03.2 · Blocks of the variance matrix of \((\hat{\boldsymbol\beta},\boldsymbol e)\) for \(p+1=3\), \(n=6\) (data set 3). Green blocks: covariances between \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\), all 0.

**How** Added

Use joint normality to show that two random vectors are independent

- Confirm the two vectors together are MVN. Here both are linear functions of \(\boldsymbol y\) (Slides p.14).
- Write the variance matrix. Find the block \(\mathrm{Cov}(\hat{\boldsymbol\beta},\boldsymbol e)\).
- Confirm this block is 0. Here it is \(\sigma^2\boldsymbol B\), \(\boldsymbol B=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol I-\boldsymbol H)=\boldsymbol 0\) (Slides p.21).
- Write that the two vectors are independent (Lecture 5 · p.40).
- Apply this to each function of \(\boldsymbol e\) only, for example \(\boldsymbol e^T\boldsymbol e\) and \(\hat\sigma^2\).

**Self-check:** (1) Zero covariance gives independence only for jointly normal vectors. (2) \(\boldsymbol B\boldsymbol y=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol e\), so \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\).

**Example · Data set 3** Added

How step 3 with \(\boldsymbol e=(-1,-1,2,2,-1,-1)^T\) from Section 03.3.

1

Calculate \(\boldsymbol X^T\boldsymbol e\) How step 3

\[\boldsymbol X^T\boldsymbol e=\left[\begin{array}{c}-1-1+2+2-1-1\\(-1)(-1)+(-1)(-1)+0+0+(1)(-1)+(1)(-1)\\(-1)(-1)+(1)(-1)+(-1)(2)+(1)(2)+(-1)(-1)+(1)(-1)\end{array}\right]\]

\[=\left[\begin{array}{c}0\\1+1+0+0-1-1\\1-1-2+2+1-1\end{array}\right]=\left[\begin{array}{c}0\\0\\0\end{array}\right]\]

2

Calculate \(\boldsymbol B\boldsymbol y\) Slides p.21

\[\boldsymbol B\boldsymbol y=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol I-\boldsymbol H)\boldsymbol y=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol e\]

\[=\left[\begin{array}{ccc}1/6&0&0\\0&1/4&0\\0&0&1/6\end{array}\right]\left[\begin{array}{c}0\\0\\0\end{array}\right]=\left[\begin{array}{c}0\\0\\0\end{array}\right]\]

3

Functions of \(\boldsymbol e\) only How step 5

\[\boldsymbol e^T\boldsymbol e=12,\qquad\hat\sigma^2=\frac{12}{3}=4\]

Independent of \(\hat{\boldsymbol\beta}=(5,2,1)^T\) by property (3).

One data set cannot show independence; it is a property of the distribution. The proof shows it in general.

**Why** Slides p.28

1

\[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\\boldsymbol e\end{array}\right]=\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\\boldsymbol I-\boldsymbol H\end{array}\right]\boldsymbol y\]

Slides p.14

2

\[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\\boldsymbol e\end{array}\right]\text{ is MVN}\]

linear function of MVN \(\boldsymbol y\) (Lecture 5 · p.40, Linearity)

3

\[\mathrm{Var}\left(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\\boldsymbol e\end{array}\right]\right)=\sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1}&\boldsymbol 0\\\boldsymbol 0&(\boldsymbol I-\boldsymbol H)\end{array}\right]\]

\(\boldsymbol A=(\boldsymbol X^T\boldsymbol X)^{-1}\), \(\boldsymbol B=\boldsymbol 0\), \(\boldsymbol D=\boldsymbol I-\boldsymbol H\) (Slides p.21–22)

4

\[\mathrm{Cov}(\hat{\boldsymbol\beta},\boldsymbol e)=\sigma^2\cdot\boldsymbol 0=\boldsymbol 0\]

top-right block

5

\[\boldsymbol e\text{ is independent of }\hat{\boldsymbol\beta}\]

jointly MVN, zero covariance (Lecture 5 · p.40, Independence)

6

\[\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\text{ is independent of }\hat{\boldsymbol\beta}\]

function of \(\boldsymbol e\) only

7

\[V=\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\ \text{ is independent of }\ Z=\frac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}\]

\(Z\) is a function of \(\hat{\boldsymbol\beta}\) only ∎

### 03.5 Practice Added

**Q1.** MLR with \(n=6\), \(p=2\). One data set gives \(\hat\beta_1=2\), \(V_{11}=0.25\), \(\boldsymbol e^T\boldsymbol e=6\). The true value is \(\beta_1=1.5\).

(a) Calculate \(\hat\sigma^2\). (b) Calculate \(\dfrac{\hat\beta_1-\beta_1}{\sqrt{\hat\sigma^2V_{11}}}\). (c) State its distribution.

Answer

a

\[\hat\sigma^2=\frac{\boldsymbol e^T\boldsymbol e}{n-(p+1)}=\frac{6}{6-3}=\frac{6}{3}=2\]

b

\[\sqrt{\hat\sigma^2V_{11}}=\sqrt{2\times0.25}=\sqrt{0.5}=0.7071\]

\[\frac{\hat\beta_1-\beta_1}{\sqrt{\hat\sigma^2V_{11}}}=\frac{2-1.5}{0.7071}=\frac{0.5}{0.7071}=0.7071\]

c

\[\frac{\hat\beta_1-\beta_1}{\sqrt{\hat\sigma^2V_{11}}}\sim t_{n-(p+1)}=t_{3}\]

Source: the Section 03.3 design with \(\boldsymbol y=(1,4,3,9,5,8)^T\): \(\hat{\boldsymbol\beta}=(5,2,2)^T\), \(\boldsymbol e=(0,-1,0,2,0,-1)^T\).

**Q2.** List the three properties that give \(\dfrac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\sim t_{n-(p+1)}\). Why does \(\sigma\) not appear? Why does the \(t\) result need all three?

Answer

(1) \(Z=\dfrac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}\sim N(0,1)\).

(2) \(V=\dfrac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\).

(3) \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) are independent, so \(Z\) and \(V\) are independent.

\(Z\) has \(\sigma\) in its denominator; \(\sqrt{V/\nu}\) has \(1/\sigma\). They cancel, and \(\sqrt{\boldsymbol e^T\boldsymbol e/(n-(p+1))}=\hat\sigma\) stays. The t definition needs an \(N(0,1)\) numerator, a \(\chi^2_\nu\) denominator, and their independence.

**Q3.** Use the joint distribution \[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\\boldsymbol e\end{array}\right]\sim N\left(\left[\begin{array}{c}\boldsymbol\beta\\\boldsymbol 0\end{array}\right],\ \sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1}&\boldsymbol 0\\\boldsymbol 0&(\boldsymbol I-\boldsymbol H)\end{array}\right]\right),\] to show that \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\) is independent of \(\hat{\boldsymbol\beta}\).

Answer

1

\[\mathrm{Cov}(\hat{\boldsymbol\beta},\boldsymbol e)=\sigma^2\cdot\boldsymbol 0=\boldsymbol 0\]

top-right block

2

\[\boldsymbol e\text{ is independent of }\hat{\boldsymbol\beta}\]

jointly MVN, zero covariance

3

\[\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\text{ is independent of }\hat{\boldsymbol\beta}\]

function of \(\boldsymbol e\) only ∎

## 04 · Property (2): \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\)

Plan · Slides p.29–43

Step 1: Goal and problem: the \(e_i\) are not independent (Slides p.29–30).

Step 2: Eigen decomposition turns \(\boldsymbol e\) into \(\tilde{\boldsymbol e}\) with independent entries (Slides p.31–36).

Step 3: \(\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=\boldsymbol e^T\boldsymbol e\) (Slides p.37–38).

Step 4: Each \(\lambda_i\) is 0 or 1 (Slides p.39–40).

Step 5: \(\sum\lambda_i=n-(p+1)\) (Slides p.41–42).

Step 6: Combine (Slides p.43).

Results used: \(\boldsymbol e\sim N(\boldsymbol 0,\sigma^2(\boldsymbol I-\boldsymbol H))\) (Section 02.7). \(\boldsymbol I-\boldsymbol H\) is symmetric and \((\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)=\boldsymbol I-\boldsymbol H\) (Section 02.6).

Data set 2 · Added

From Unit 02: \(\boldsymbol x=(0,1,2,3)^T\), \(\boldsymbol y=(1,3,2,5)^T\), \(n=4\), \(p=1\).

\(\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}4&6\\6&14\end{array}\right]\), \((\boldsymbol X^T\boldsymbol X)^{-1}=\frac{1}{20}\left[\begin{array}{cc}14&-6\\-6&4\end{array}\right]\).

\(\hat{\boldsymbol\beta}=(1.1,\ 1.1)^T\), \(\boldsymbol e=(-0.1,\ 0.8,\ -1.3,\ 0.6)^T\), \(\boldsymbol e^T\boldsymbol e=2.70\).

\(h_{11}=0.7\), \(h_{22}=0.3\), \(h_{33}=0.3\), \(h_{44}=0.7\).
\[\boldsymbol I-\boldsymbol H=\left[\begin{array}{cccc}0.3&-0.4&-0.1&0.2\\-0.4&0.7&-0.2&-0.1\\-0.1&-0.2&0.7&-0.4\\0.2&-0.1&-0.4&0.3\end{array}\right]\]

### 04.1 The plan of the proof Slides p.29–30

**What** Slides p.29–30

Lecture 7 · p.29–30 We want to show that \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\). Recall for \(Z_i\overset{iid}{\sim}N(0,1)\): \[\sum_{i=1}^{\nu}Z_i^2\sim\chi^2_\nu\] So we want to write \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\) as a sum of squared independent standard normal r.v.s. We could write: \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\sum_{i=1}^n\left(\frac{e_i}{\sigma}\right)^2\) ... Problem: the \(e_i\) are not independent! (why?) Solution: let's rewrite \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\) as a sum of squared independent normal r.v.s

r.v.: random variable.

The chi-squared definition needs terms that are \(N(0,1)\) and independent. Each \(e_i/\sigma\) is normal, but the terms are not independent. The proof finds independent variables with the same sum of squares.

**How** Added

Find if a sum of squares has a chi-squared distribution

- Write the sum as \(\sum_i Z_i^2\). Identify each \(Z_i\).
- Confirm each \(Z_i\) is normal with mean 0 and variance 1.
- Confirm the \(Z_i\) are independent. For an MVN vector, all off-diagonal variance entries are 0.
- Count the terms: \(\nu\).

**Self-check:** If an off-diagonal entry is not 0, change the variables first.

**Example** Added

Data set 2: test \(e_1/\sigma\) and \(e_2/\sigma\) against steps 2 and 3.

1

\[\mathrm{Var}[e_1]=\sigma^2(\boldsymbol I-\boldsymbol H)_{11}=0.3\,\sigma^2\]

Entry \((1,1)\) of \(\mathrm{Var}[\boldsymbol e]\).

2

\[\mathrm{Var}\left[\frac{e_1}{\sigma}\right]=\frac{0.3\,\sigma^2}{\sigma^2}=0.3\neq1\]

3

\[\mathrm{Cov}(e_1,e_2)=\sigma^2(\boldsymbol I-\boldsymbol H)_{12}=-0.4\,\sigma^2\neq0\]

Entry \((1,2)\) of \(\mathrm{Var}[\boldsymbol e]\).

Steps 2 and 3 fail. \(\sum_i(e_i/\sigma)^2\) is not directly a chi-squared sum. ▲

**Why** Slides p.29–30

All \(e_i\) come from one \(\boldsymbol y\) through \(\boldsymbol e=(\boldsymbol I-\boldsymbol H)\boldsymbol y\). The off-diagonal covariances \(\sigma^2(\boldsymbol I-\boldsymbol H)_{ij}\) are usually not 0.

Also \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). In data set 2, \(-0.1+0.8-1.3+0.6=0\): three residuals fix the fourth. ∎

### 04.2 Eigen decomposition and the distribution of \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\) Slides p.31–36

**What** Slides p.31–36

Lecture 7 · p.31–36 We have \(\boldsymbol e=(\boldsymbol I-\boldsymbol H)\boldsymbol y\), and we consider the eigen decomposition \(\boldsymbol I-\boldsymbol H=\boldsymbol\Gamma^T\boldsymbol D\boldsymbol\Gamma\) where \[\boldsymbol\Gamma^{-1}=\boldsymbol\Gamma^T\quad\text{and}\quad\boldsymbol D=\left[\begin{array}{ccc}\lambda_1&&\\&\ddots&\\&&\lambda_n\end{array}\right]\] Define \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\). Now, what is the distribution of \(\tilde{\boldsymbol e}\)? \[E[\tilde{\boldsymbol e}]=E[\boldsymbol\Gamma\boldsymbol e]=\boldsymbol\Gamma E[\boldsymbol e]=\boldsymbol 0\] \[\mathrm{Var}[\tilde{\boldsymbol e}]=\mathrm{Var}[\boldsymbol\Gamma\boldsymbol e]=\boldsymbol\Gamma \mathrm{Var}[\boldsymbol e]\boldsymbol\Gamma^T=\sigma^2\boldsymbol\Gamma(\boldsymbol I-\boldsymbol H)\boldsymbol\Gamma^T=\sigma^2\boldsymbol\Gamma(\boldsymbol\Gamma^T\boldsymbol D\boldsymbol\Gamma)\boldsymbol\Gamma^T=\sigma^2\boldsymbol D\] and \(\tilde{\boldsymbol e}\sim N(\boldsymbol 0,\sigma^2\boldsymbol D)\) (why?) \[\Longrightarrow\ \tilde e_i\overset{indep}{\sim}N(0,\sigma^2\lambda_i)\]

Eigenvector of \(\boldsymbol A\): a nonzero \(\boldsymbol v_i\) with \(\boldsymbol A\boldsymbol v_i=\lambda_i\boldsymbol v_i\). The number \(\lambda_i\) is its eigenvalue.

Diagonal matrix: a square matrix with all off-diagonal entries 0.

Orthogonal matrix: \(\boldsymbol\Gamma^{-1}=\boldsymbol\Gamma^T\), so \(\boldsymbol\Gamma\boldsymbol\Gamma^T=\boldsymbol\Gamma^T\boldsymbol\Gamma=\boldsymbol I\). It does not change vector lengths.

Eigen decomposition: a symmetric matrix written as \(\boldsymbol\Gamma^T\boldsymbol D\boldsymbol\Gamma\). Row \(i\) of \(\boldsymbol\Gamma\) is a length-1 eigenvector for \(\lambda_i\).

\(\overset{indep}{\sim}\): independent, entry \(i\) with the stated distribution.

\(\mathrm{Var}[\boldsymbol e]\) is not diagonal. \(\mathrm{Var}[\tilde{\boldsymbol e}]=\sigma^2\boldsymbol D\) is diagonal, so the \(\tilde e_i\) are independent.

**How** Added

Find \(\boldsymbol\Gamma\), \(\boldsymbol D\) and \(\tilde{\boldsymbol e}\) by hand

- Calculate \(\boldsymbol I-\boldsymbol H\).
- Find \(n\) mutually orthogonal eigenvectors. Read each \(\lambda\) from \((\boldsymbol I-\boldsymbol H)\boldsymbol v=\lambda\boldsymbol v\).
- Divide each eigenvector by its length \(\sqrt{\boldsymbol v^T\boldsymbol v}\).
- Put them in the rows of \(\boldsymbol\Gamma\). Put their eigenvalues on the diagonal of \(\boldsymbol D\), same order.
- Calculate \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\).

**Self-check:** (1) \(\boldsymbol\Gamma\boldsymbol\Gamma^T=\boldsymbol I\). (2) Each row satisfies \((\boldsymbol I-\boldsymbol H)\boldsymbol v=\lambda\boldsymbol v\).

**Example** Added

Step 2 How step 2: try \(\boldsymbol v_1=(1,-1,-1,1)^T\), \(\boldsymbol v_2=(1,-3,3,-1)^T\), \(\boldsymbol v_3=(1,1,1,1)^T\), \(\boldsymbol v_4=(-3,-1,1,3)^T\).

Basis: \((\boldsymbol I-\boldsymbol H)\boldsymbol v=\lambda\boldsymbol v\). Each line multiplies the rows of \(\boldsymbol I-\boldsymbol H\) by \(\boldsymbol v\).

1

\[(\boldsymbol I-\boldsymbol H)\boldsymbol v_1=\left[\begin{array}{c}0.3+0.4+0.1+0.2\\-0.4-0.7+0.2-0.1\\-0.1+0.2-0.7-0.4\\0.2+0.1+0.4+0.3\end{array}\right]=\left[\begin{array}{c}1\\-1\\-1\\1\end{array}\right]=1\cdot\boldsymbol v_1\]

2

\[(\boldsymbol I-\boldsymbol H)\boldsymbol v_2=\left[\begin{array}{c}0.3+1.2-0.3-0.2\\-0.4-2.1-0.6+0.1\\-0.1+0.6+2.1+0.4\\0.2+0.3-1.2-0.3\end{array}\right]=\left[\begin{array}{c}1\\-3\\3\\-1\end{array}\right]=1\cdot\boldsymbol v_2\]

3

\[(\boldsymbol I-\boldsymbol H)\boldsymbol v_3=\left[\begin{array}{c}0.3-0.4-0.1+0.2\\-0.4+0.7-0.2-0.1\\-0.1-0.2+0.7-0.4\\0.2-0.1-0.4+0.3\end{array}\right]=\left[\begin{array}{c}0\\0\\0\\0\end{array}\right]=0\cdot\boldsymbol v_3\]

4

\[(\boldsymbol I-\boldsymbol H)\boldsymbol v_4=\left[\begin{array}{c}-0.9+0.4-0.1+0.6\\1.2-0.7-0.2-0.3\\0.3+0.2+0.7-1.2\\-0.6+0.1-0.4+0.9\end{array}\right]=\left[\begin{array}{c}0\\0\\0\\0\end{array}\right]=0\cdot\boldsymbol v_4\]

\(\lambda_1=\lambda_2=1\), \(\lambda_3=\lambda_4=0\).

Step 3 How step 3: \(\boldsymbol v_1^T\boldsymbol v_1=1+1+1+1=4\), \(\boldsymbol v_2^T\boldsymbol v_2=1+9+9+1=20\), \(\boldsymbol v_3^T\boldsymbol v_3=4\), \(\boldsymbol v_4^T\boldsymbol v_4=9+1+1+9=20\). Divide by \(\sqrt4=2\) or \(\sqrt{20}\).

Step 4 How step 4:

\[\boldsymbol\Gamma=\left[\begin{array}{cccc}\frac12&-\frac12&-\frac12&\frac12\\\frac{1}{\sqrt{20}}&\frac{-3}{\sqrt{20}}&\frac{3}{\sqrt{20}}&\frac{-1}{\sqrt{20}}\\\frac12&\frac12&\frac12&\frac12\\\frac{-3}{\sqrt{20}}&\frac{-1}{\sqrt{20}}&\frac{1}{\sqrt{20}}&\frac{3}{\sqrt{20}}\end{array}\right],\qquad\boldsymbol D=\left[\begin{array}{cccc}1&0&0&0\\0&1&0&0\\0&0&0&0\\0&0&0&0\end{array}\right]\]

Step 5 How step 5: rows of \(\boldsymbol\Gamma\) times \(\boldsymbol e=(-0.1,0.8,-1.3,0.6)^T\).

1

\[\tilde e_1=\tfrac12(-0.1-0.8+1.3+0.6)=\tfrac12(1.0)=0.5\]

2

\[\tilde e_2=\tfrac{1}{\sqrt{20}}(-0.1-2.4-3.9-0.6)=\tfrac{-7}{\sqrt{20}}\approx-1.5652\]

3

\[\tilde e_3=\tfrac12(-0.1+0.8-1.3+0.6)=\tfrac12(0)=0\]

4

\[\tilde e_4=\tfrac{1}{\sqrt{20}}(0.3-0.8-1.3+1.8)=\tfrac{1}{\sqrt{20}}(0)=0\]

Self-check (1): each row has length 1; distinct rows have inner product 0, for example rows 1 and 3: \(\frac14(1-1-1+1)=0\). \(\tilde e_3=\tilde e_4=0\) because \(\lambda_i=0\) gives variance 0. ▲

**Why** Slides p.31–36

1

\[E[\tilde{\boldsymbol e}]=E[\boldsymbol\Gamma\boldsymbol e]\]

Definition of \(\tilde{\boldsymbol e}\).

2

\[=\boldsymbol\Gamma E[\boldsymbol e]\]

\(\boldsymbol\Gamma\) is constant.

3

\[=\boldsymbol 0\]

Slides p.25.

1

\[\mathrm{Var}[\tilde{\boldsymbol e}]=\mathrm{Var}[\boldsymbol\Gamma\boldsymbol e]\]

2

\[=\boldsymbol\Gamma\,\mathrm{Var}[\boldsymbol e]\,\boldsymbol\Gamma^T\]

Lecture 5 · p.40, Linearity.

3

\[=\sigma^2\boldsymbol\Gamma(\boldsymbol I-\boldsymbol H)\boldsymbol\Gamma^T\]

Slides p.25.

4

\[=\sigma^2\boldsymbol\Gamma(\boldsymbol\Gamma^T\boldsymbol D\boldsymbol\Gamma)\boldsymbol\Gamma^T\]

Eigen decomposition.

5

\[=\sigma^2(\boldsymbol\Gamma\boldsymbol\Gamma^T)\boldsymbol D(\boldsymbol\Gamma\boldsymbol\Gamma^T)\]

6

\[=\sigma^2\boldsymbol D\]

\(\boldsymbol\Gamma\boldsymbol\Gamma^T=\boldsymbol I\).

The slide asks why \(\tilde{\boldsymbol e}\) is normal. It is a linear transformation of the MVN \(\boldsymbol e\), so it is MVN (Lecture 5 · p.40, Linearity). \(\sigma^2\boldsymbol D\) has zero off-diagonal entries, so the \(\tilde e_i\) are independent (Lecture 5 · p.40, Independence), with variances \(\sigma^2\lambda_i\). ∎

### 04.3 Why \(\tilde{\boldsymbol e}\) is useful: \(\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=\boldsymbol e^T\boldsymbol e\) Slides p.37–38

**What** Slides p.37–38

Lecture 7 · p.37–38 And we defined \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\) such that \(\tilde e_i\overset{indep}{\sim}N(0,\sigma^2\lambda_i)\). Why is this useful? \[\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=(\boldsymbol\Gamma\boldsymbol e)^T(\boldsymbol\Gamma\boldsymbol e)=\boldsymbol e^T\boldsymbol\Gamma^T\boldsymbol\Gamma\boldsymbol e=\boldsymbol e^T\boldsymbol e,\] so we can write: \[\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\frac{1}{\sigma^2}\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=\sum_{i=1}^n\left(\frac{\tilde e_i}{\sigma}\right)^2=\sum_{i=1}^n Z_i^2\] where \(Z_i\overset{indep}{\sim}N(0,\lambda_i)\).

\(\boldsymbol e^T\boldsymbol e\) is the squared length of \(\boldsymbol e\), and \(\boldsymbol\Gamma\) keeps lengths. \(Z_i=\tilde e_i/\sigma\) has variance \(\lambda_i\).

**How** Added

Check that \(\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=\boldsymbol e^T\boldsymbol e\)

- Get \(\boldsymbol e\) and \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\) (Section 04.2).
- Add the squares of the entries of \(\boldsymbol e\).
- Add the squares of the entries of \(\tilde{\boldsymbol e}\).
- Compare the two sums.

**Self-check:** Both sums equal \(\boldsymbol e^T\boldsymbol e\).

**Example** Added

1

\[\boldsymbol e^T\boldsymbol e=(-0.1)^2+0.8^2+(-1.3)^2+0.6^2=0.01+0.64+1.69+0.36=2.70\]

How step 2.

2

\[\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=0.5^2+\left(\tfrac{-7}{\sqrt{20}}\right)^2+0^2+0^2=0.25+\tfrac{49}{20}+0+0\]

How step 3.

3

\[=0.25+2.45=2.70\]

\(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\frac{2.70}{\sigma^2}=Z_1^2+Z_2^2+Z_3^2+Z_4^2\), with observed \(Z_3=Z_4=0\). ▲

**Why** Slides p.37–38

1

\[\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=(\boldsymbol\Gamma\boldsymbol e)^T(\boldsymbol\Gamma\boldsymbol e)\]

Definition of \(\tilde{\boldsymbol e}\).

2

\[=\boldsymbol e^T\boldsymbol\Gamma^T\boldsymbol\Gamma\boldsymbol e\]

\((\boldsymbol A\boldsymbol B)^T=\boldsymbol B^T\boldsymbol A^T\).

3

\[=\boldsymbol e^T\boldsymbol e\]

\(\boldsymbol\Gamma^T\boldsymbol\Gamma=\boldsymbol I\).

4

\[\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\frac{1}{\sigma^2}\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}\]

5

\[=\frac{1}{\sigma^2}\sum_{i=1}^n\tilde e_i^2\]

6

\[=\sum_{i=1}^n\left(\frac{\tilde e_i}{\sigma}\right)^2\]

7

\[=\sum_{i=1}^n Z_i^2\]

\(Z_i=\tilde e_i/\sigma\overset{indep}{\sim}N(0,\lambda_i)\). ∎

Two conditions remain: each variance is 1, and there are \(n-(p+1)\) terms.

### 04.4 Each \(\lambda_i\) is 0 or 1 Slides p.39–40

**What** Slides p.39–40

Lecture 7 · p.39–40 So \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\) is a sum of squared independent normally distributed r.v.'s.
• Recall: need a sum of independent standard normally distributed r.v.'s (var=1)
• And: we need to sum up exactly \(n-(p+1)\) of these r.v.s
So: to show \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{(n-(p+1))}\), we will show that \(n-(p+1)\) of the \(\lambda_i\)'s are equal to 1, and the remainder are all 0.
Let's first show that all of the \(\lambda_i\)'s are equal to either 0 or 1:
• We know that \((\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)=(\boldsymbol I-\boldsymbol H)\); i.e. \((\boldsymbol I-\boldsymbol H)\) is idempotent.
• The eigenvalues (elements \(\lambda_i\) of \(\boldsymbol D\)) of an idempotent matrix are equal to 0 or 1.
• Why? By definition of eigenvalue \(\lambda_i\) and corresponding eigenvector \(\boldsymbol v_i\), we know: \(\lambda_i\boldsymbol v_i=(\boldsymbol I-\boldsymbol H)\boldsymbol v_i\)
Thus all \(\lambda_i\) are either 0 or 1.

Idempotent matrix: a square \(\boldsymbol A\) with \(\boldsymbol A\boldsymbol A=\boldsymbol A\).

Degenerate random variable: variance 0, so it always has one value. \(N(0,0)\) is always 0.

\(\lambda_i=1\) gives \(Z_i\sim N(0,1)\). \(\lambda_i=0\) gives \(Z_i=0\), which adds 0 to the sum.

**How** Added

Show that a matrix is idempotent and find its possible eigenvalues

- Calculate \(\boldsymbol A\boldsymbol A\): entry \((i,j)\) is row \(i\) times column \(j\).
- Compare \(\boldsymbol A\boldsymbol A\) with \(\boldsymbol A\), entry by entry.
- If all are equal, \(\boldsymbol A\) is idempotent, and each eigenvalue is 0 or 1.

**Self-check:** For each known eigenvector, \(\boldsymbol A\boldsymbol v\) is \(\boldsymbol v\) or \(\boldsymbol 0\).

**Example** Added

Data set 2. Row 1 of \(\boldsymbol I-\boldsymbol H\) is \((0.3,-0.4,-0.1,0.2)\).

1

\[\text{entry }(1,1)=0.3\cdot0.3+(-0.4)(-0.4)+(-0.1)(-0.1)+0.2\cdot0.2=0.09+0.16+0.01+0.04=0.30\]

Entry \((1,1)\) of \(\boldsymbol I-\boldsymbol H\) is 0.3.

2

\[\text{entry }(1,2)=0.3(-0.4)+(-0.4)(0.7)+(-0.1)(-0.2)+0.2(-0.1)=-0.12-0.28+0.02-0.02=-0.40\]

Entry \((1,2)\) of \(\boldsymbol I-\boldsymbol H\) is \(-0.4\).

The other 14 entries also agree. This matches \(\lambda=1,1,0,0\) from Section 04.2. ▲

Satisfaction data of the slides: \(n=46\) patients, \(p=3\) (Lecture 6 · p.9). Sections 04.4–04.5 give 42 eigenvalues of 1 and 4 of 0.

[figure]
Figure 04.1 · The 46 eigenvalues of \(\boldsymbol I-\boldsymbol H\) for the satisfaction data (\(n=46\), \(p=3\)), largest first: 42 blue bars at 1, 4 orange dots at 0.

**Why** Slides p.39–40

1

\[\lambda_i\boldsymbol v_i=(\boldsymbol I-\boldsymbol H)\boldsymbol v_i\]

Definition of eigenvalue.

2

\[=(\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)\boldsymbol v_i\]

Idempotent.

3

\[=(\boldsymbol I-\boldsymbol H)[\lambda_i\boldsymbol v_i]\]

Definition again.

4

\[=\lambda_i(\boldsymbol I-\boldsymbol H)\boldsymbol v_i\]

\(\lambda_i\) is a number.

5

\[=\lambda_i^2\boldsymbol v_i\]

Definition again.

6

\[\lambda_i\boldsymbol v_i-\lambda_i^2\boldsymbol v_i=\boldsymbol 0\]

7

\[\lambda_i(1-\lambda_i)\boldsymbol v_i=\boldsymbol 0\]

8

\[\lambda_i\in\{0,1\}\]

\(\boldsymbol v_i\neq\boldsymbol 0\), so \(\lambda_i(1-\lambda_i)=0\). ∎

### 04.5 \(\sum_i\lambda_i=n-(p+1)\): count the 1s with the trace Slides p.41–42

Each \(\lambda_i\) is 0 or 1, so their sum counts the 1s.

**What** Slides p.41–42

Lecture 7 · p.41–42 Now, if all the \(\lambda_i\) are 0 or 1, then we can show that exactly \(n-(p+1)\) of them are equal to 1 by showing that \(\sum_i\lambda_i=n-(p+1)\): \[\sum_i\lambda_i=\mathrm{tr}(\boldsymbol D)=\mathrm{tr}(\boldsymbol D\boldsymbol\Gamma\boldsymbol\Gamma^T)=\mathrm{tr}(\boldsymbol\Gamma^T\boldsymbol D\boldsymbol\Gamma)=\mathrm{tr}(\boldsymbol I-\boldsymbol H)\] and \[\mathrm{tr}(\boldsymbol I-\boldsymbol H)=\mathrm{tr}(\boldsymbol I)-\mathrm{tr}(\boldsymbol H)=n-\mathrm{tr}(\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T)=n-\mathrm{tr}((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X)=n-(p+1)\] and we are done.

Trace \(\mathrm{tr}(\boldsymbol A)\): the sum of the diagonal entries of a square matrix.

Trace rules: \(\mathrm{tr}(\boldsymbol A-\boldsymbol B)=\mathrm{tr}(\boldsymbol A)-\mathrm{tr}(\boldsymbol B)\); \(\mathrm{tr}(\boldsymbol A\boldsymbol B)=\mathrm{tr}(\boldsymbol B\boldsymbol A)\) when both products are square.

The second rule turns \(\mathrm{tr}(\boldsymbol H)\) into the trace of the \((p+1)\times(p+1)\) identity, which is \(p+1\).

**How** Added

Find the number of \(\lambda_i\) equal to 1

- Find \(n\).
- Find \(p\). Do not count the intercept.
- Calculate \(n-(p+1)\): the number of \(\lambda_i=1\).
- Calculate \(p+1\): the number of \(\lambda_i=0\).

**Self-check:** The counts add to \(n\). By hand, \(\mathrm{tr}(\boldsymbol H)=p+1\) and \(\mathrm{tr}(\boldsymbol I-\boldsymbol H)=n-(p+1)\).

**Example** Added

Data set 2 (\(n=4\), \(p=1\)).

1

\[\sum_i\lambda_i=n-(p+1)=4-(1+1)=2\]

How steps 1–3.

2

\[\mathrm{tr}(\boldsymbol H)=h_{11}+h_{22}+h_{33}+h_{44}=0.7+0.3+0.3+0.7=2=p+1\]

3

\[\mathrm{tr}(\boldsymbol I-\boldsymbol H)=0.3+0.7+0.7+0.3=2\]

4

\[(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\frac{1}{20}\left[\begin{array}{cc}14\cdot4-6\cdot6&14\cdot6-6\cdot14\\-6\cdot4+4\cdot6&-6\cdot6+4\cdot14\end{array}\right]=\frac{1}{20}\left[\begin{array}{cc}20&0\\0&20\end{array}\right]=\boldsymbol I_2\]

Trace \(1+1=2=p+1\).

Two eigenvalues are 1, two are 0, as in Section 04.2. Satisfaction data: \(46-(3+1)=42\) ones and \(3+1=4\) zeros (Figure 04.1). ▲

**Why** Slides p.41–42

1

\[\sum_i\lambda_i=\mathrm{tr}(\boldsymbol D)\]

Diagonal of \(\boldsymbol D\) is \(\lambda_1,\dots,\lambda_n\).

2

\[=\mathrm{tr}(\boldsymbol D\boldsymbol\Gamma\boldsymbol\Gamma^T)\]

\(\boldsymbol\Gamma\boldsymbol\Gamma^T=\boldsymbol I\).

3

\[=\mathrm{tr}(\boldsymbol\Gamma^T\boldsymbol D\boldsymbol\Gamma)\]

\(\mathrm{tr}(\boldsymbol A\boldsymbol B)=\mathrm{tr}(\boldsymbol B\boldsymbol A)\), \(\boldsymbol A=\boldsymbol D\boldsymbol\Gamma\), \(\boldsymbol B=\boldsymbol\Gamma^T\).

4

\[=\mathrm{tr}(\boldsymbol I-\boldsymbol H)\]

Eigen decomposition.

1

\[\mathrm{tr}(\boldsymbol I-\boldsymbol H)=\mathrm{tr}(\boldsymbol I)-\mathrm{tr}(\boldsymbol H)\]

2

\[=n-\mathrm{tr}(\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T)\]

\(\mathrm{tr}(\boldsymbol I_n)=n\); definition of \(\boldsymbol H\).

3

\[=n-\mathrm{tr}((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X)\]

\(\mathrm{tr}(\boldsymbol A\boldsymbol B)=\mathrm{tr}(\boldsymbol B\boldsymbol A)\), \(\boldsymbol A=\boldsymbol X\).

4

\[=n-\mathrm{tr}(\boldsymbol I_{p+1})\]

\((p+1)\times(p+1)\) identity.

5

\[=n-(p+1)\]

∎

### 04.6 Recap: \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\) Slides p.43

**What** Slides p.43

Lecture 7 · p.43 We showed:
1. \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\sum_{i=1}^nZ_i^2\) where \(Z_i\overset{indep}{\sim}N(0,\lambda_i)\).
• i.e., a sum of \(n\) squared independent normally distributed r.v.'s
2. \(var(Z_i)=\lambda_i\in\{0,1\}\)
• hence are either standard normally distributed or degenerate (0)
3. \(\sum_{i=1}^n\lambda_i=n-(p+1)\)
• hence exactly \(n-(p+1)\) of the \(Z_i\) are \(N(0,1)\) r.v.s
\(\Longrightarrow\ \frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\) is a sum of \(n-(p+1)\) squared standard normal r.v.s \[\Longrightarrow\ \frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\]

Drop the terms that are always 0. The rest fits the chi-squared definition.

With \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\), the result is also \(\frac{(n-(p+1))\hat\sigma^2}{\sigma^2}\sim\chi^2_{n-(p+1)}\).

**How** Added

State the distribution of \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\) for a data set

- Find \(n\) and \(p\).
- Calculate \(n-(p+1)\).
- Write \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\).
- For \(\hat\sigma^2\), divide \(\boldsymbol e^T\boldsymbol e\) by \(n-(p+1)\).

**Self-check:** The chi-squared degrees of freedom equal the divisor in \(\hat\sigma^2\).

**Example** Added

1

\[n-(p+1)=4-(1+1)=2\]

How steps 1–2.

2

\[\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{2}\]

How step 3.

3

\[\hat\sigma^2=\frac{2.70}{2}=1.35\]

How step 4.

Section 04.3: \(\frac{2.70}{\sigma^2}=Z_1^2+Z_2^2+0^2+0^2\), two standard normal terms, as \(\chi^2_2\) requires. ▲

**Why** Slides p.43

Item 1: Section 04.3. Item 2: Section 04.4. Item 3: Section 04.5. ∎

### 04.7 Practice Added

**Q1.** A regression has \(n=20\), \(p=3\). How many eigenvalues of \(\boldsymbol I-\boldsymbol H\) are 1, and how many are 0? Give the distribution of \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\).

Answer

1

\[\sum_i\lambda_i=n-(p+1)=20-(3+1)=16\]

Slides p.42.

2

\[n-16=20-16=4=p+1\]

Each \(\lambda_i\) is 0 or 1 (Slides p.40): 16 ones, 4 zeros. Slides p.43: \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{16}\).

**Q2.** Show that \(\boldsymbol H\) is idempotent and that its eigenvalues are 0 or 1. For the satisfaction data (\(n=46\), \(p=3\)), how many eigenvalues of \(\boldsymbol H\) are 1?

Answer

1

\[\boldsymbol H\boldsymbol H=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

2

\[=\boldsymbol X\boldsymbol I(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\]

\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).

3

\[=\boldsymbol H\]

The proof of Slides p.40 with \(\boldsymbol H\): \(\lambda\boldsymbol v=\boldsymbol H\boldsymbol v=\boldsymbol H\boldsymbol H\boldsymbol v=\lambda^2\boldsymbol v\), so \(\lambda\in\{0,1\}\). The count of 1s is \(\mathrm{tr}(\boldsymbol H)=p+1=3+1=4\) (Slides p.42, line 3).

**Q3.** Use \(\boldsymbol x=(0,1,2,3)^T\), \(\boldsymbol y=(2,1,4,5)^T\), and \(y_i=\beta_0+\beta_1x_i+\epsilon_i\). Calculate \(\boldsymbol e\), \(\boldsymbol e^T\boldsymbol e\), \(\hat\sigma^2\). Give the distribution of \(\frac{2\hat\sigma^2}{\sigma^2}\).

Answer

\(\boldsymbol x\) is the same as in data set 2, so \((\boldsymbol X^T\boldsymbol X)^{-1}\) is the same.

1

\[\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}2+1+4+5\\0\cdot2+1\cdot1+2\cdot4+3\cdot5\end{array}\right]=\left[\begin{array}{c}12\\24\end{array}\right]\]

2

\[\hat{\boldsymbol\beta}=\frac{1}{20}\left[\begin{array}{c}14\cdot12-6\cdot24\\-6\cdot12+4\cdot24\end{array}\right]=\frac{1}{20}\left[\begin{array}{c}168-144\\-72+96\end{array}\right]=\left[\begin{array}{c}1.2\\1.2\end{array}\right]\]

3

\[\hat{\boldsymbol y}=(1.2,\ 2.4,\ 3.6,\ 4.8)^T\]

4

\[\boldsymbol e=(2-1.2,\ 1-2.4,\ 4-3.6,\ 5-4.8)^T=(0.8,\ -1.4,\ 0.4,\ 0.2)^T\]

5

\[\boldsymbol e^T\boldsymbol e=0.64+1.96+0.16+0.04=2.80\]

6

\[\hat\sigma^2=\frac{2.80}{4-(1+1)}=\frac{2.80}{2}=1.40\]

Check with \(\boldsymbol\Gamma\) (Section 04.2): \(\tilde e_1=\frac12(0.8+1.4-0.4+0.2)=1.0\), \(\tilde e_2=\frac{1}{\sqrt{20}}(0.8+4.2+1.2-0.2)=\frac{6}{\sqrt{20}}\). \(1.0^2+\frac{36}{20}=1.0+1.8=2.80\).

\(2\hat\sigma^2=\boldsymbol e^T\boldsymbol e\), so \(\frac{2\hat\sigma^2}{\sigma^2}\sim\chi^2_{2}\) (Slides p.43).

## 05 · t tests and confidence intervals for each \(\beta_j\)

Plan · Slides p.44–51

Step 1: The t distribution of the standardized \(\hat\beta_j\) (Slides p.44).

Step 2: \(\hat{\boldsymbol\beta}\) from the normal equations (Slides p.45).

Step 3: \(\hat\sigma^2\) and each standard error (Slides p.46).

Step 4: Rules for the test and the confidence interval (Slides p.47–49).

Step 5: p-value for \(H_0:\beta_j=0\), each \(j\) (Slides p.50).

Step 6: 95% confidence interval for each \(\beta_j\) (Slides p.51).

\(V_{00}\) belongs to the intercept \(\beta_0\).

The satisfaction data (Lecture 6 · p.9)

\(n=46\) hospital patients. Response: Satisfaction. \(p=3\) covariates: Age (\(\beta_1\)), Severity (\(\beta_2\)), Stress (\(\beta_3\)).

Model: \(\text{Satisfaction}_i=\beta_0+\beta_1\text{Age}_i+\beta_2\text{Severity}_i+\beta_3\text{Stress}_i+\epsilon_i\) (Lecture 7 · p.53).

Data set 4 Added

\(n=5\), \(p=1\). Row \(i\) of \(\boldsymbol X\) is \((1,\ x_i)\).
\[x=(1,\ 2,\ 3,\ 4,\ 5),\qquad \boldsymbol y=(1,\ 3,\ 2,\ 5,\ 4)^T,\qquad \boldsymbol X=\left[\begin{array}{cc}1&1\\1&2\\1&3\\1&4\\1&5\end{array}\right]\]

### 05.1 The t distribution of the standardized \(\hat\beta_j\) Slides p.44

**What** Slides p.44

In multiple linear regression · Lecture 7 · p.44 So we have \[\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1})\implies\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\implies\frac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}\sim N(0,1)\] and when we estimate the variance with \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol{e}^T\boldsymbol{e}\), we have: \[\frac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\sim t_{n-(p+1)}\]

Line 1: subtract the mean and divide by the standard deviation (square root of the variance) to get \(N(0,1)\).

Line 2: with \(\hat\sigma^2\) for the unknown \(\sigma^2\), the distribution is \(t_{n-(p+1)}\). d.o.f.: degrees of freedom.

**How** Added

Write the t statistic for one coefficient \(\beta_j\)

- Find \(\hat\beta_j\), entry \(j\) of \(\hat{\boldsymbol\beta}\) (counted from 0).
- Find \(V_{jj}\), diagonal entry \(j\) of \((\boldsymbol X^T\boldsymbol X)^{-1}\).
- Calculate \(\hat\sigma^2=\boldsymbol e^T\boldsymbol e/(n-(p+1))\).
- Divide \(\hat\beta_j-\beta_j\) by \(\sqrt{\hat\sigma^2V_{jj}}\).
- Use \(n-(p+1)\) degrees of freedom.

**Self-check:** \(V_{jj}>0\) and \(\hat\sigma^2\ge0\), so the denominator is positive.

**Self-check:** The degrees of freedom are a positive whole number.

**Example** Added

How step 5. Sections 05.2–05.3 calculate the other inputs.

1

The satisfaction data How step 5

\[n-(p+1)=46-(3+1)=46-4=42\]

\(t_{42}\) for each \(j=0,1,2,3\).

2

Data set 4 How step 5

\[n-(p+1)=5-(1+1)=5-2=3\]

\(t_{3}\) for each \(j=0,1\). ▲

**Why** Slides p.26–43

Section 03.3 gives the plan and the chain. Section 03.4 proves property (3). Unit 04 proves property (2).

### 05.2 Calculate \(\hat{\boldsymbol\beta}\) from the normal equations Slides p.45

**What** Slides p.45

Estimation · Lecture 7 · p.45 (statistical content of the comments) \[\hat{\boldsymbol\beta}=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\] \[(\boldsymbol X^T\boldsymbol X)\hat{\boldsymbol\beta}=\boldsymbol X^T\boldsymbol y\quad\text{(more like solving system of equations)}\]

Slide p.45 shows only code; this is its statistical content. Solve the \(p+1\) normal equations directly.

The slide says "NEVER" for the other method: invert \(\boldsymbol X^T\boldsymbol X\) first, then multiply by \(\boldsymbol X^T\boldsymbol y\).

**How** Added

Calculate \(\hat{\boldsymbol\beta}\)

- Make \(\boldsymbol X\): a column of 1s, then the covariates in model order.
- Calculate \(\boldsymbol X^T\boldsymbol X\). Entry \((j,k)\): sum of column \(j\) times column \(k\), term by term.
- Calculate \(\boldsymbol X^T\boldsymbol y\). Entry \(j\): sum of column \(j\) times \(\boldsymbol y\), term by term.
- Write \((\boldsymbol X^T\boldsymbol X)\hat{\boldsymbol\beta}=\boldsymbol X^T\boldsymbol y\).
- Solve by elimination, one unknown at a time.

**Self-check:** \(\boldsymbol X^T\boldsymbol X\) is symmetric, \((p+1)\times(p+1)\), top-left entry \(n\).

**Self-check:** \(\hat{\boldsymbol\beta}\) satisfies each normal equation.

**Example** Added

Data set 4.

1

\(\boldsymbol X^T\boldsymbol X\) How step 2

\[\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}1+1+1+1+1&1+2+3+4+5\\1+2+3+4+5&1+4+9+16+25\end{array}\right]=\left[\begin{array}{cc}5&15\\15&55\end{array}\right]\]

2

\(\boldsymbol X^T\boldsymbol y\) How step 3

\[\boldsymbol X^T\boldsymbol y=\left[\begin{array}{c}1+3+2+5+4\\1\cdot1+2\cdot3+3\cdot2+4\cdot5+5\cdot4\end{array}\right]=\left[\begin{array}{c}15\\1+6+6+20+20\end{array}\right]=\left[\begin{array}{c}15\\53\end{array}\right]\]

3

Normal equations How step 4

\[5\hat\beta_0+15\hat\beta_1=15\]

\[15\hat\beta_0+55\hat\beta_1=53\]

4

Elimination How step 5

\[15\hat\beta_0+45\hat\beta_1=45\]

First equation times 3.

5

\[(55-45)\hat\beta_1=53-45\]

Second equation minus line 4.

6

\[10\hat\beta_1=8\]

7

\[\hat\beta_1=0.8\]

8

\[5\hat\beta_0+15(0.8)=15\]

Substitute \(\hat\beta_1=0.8\).

9

\[5\hat\beta_0=15-12=3\]

10

\[\hat\beta_0=0.6\]

Self-check: \(15(0.6)+55(0.8)=9+44=53\). ▲

The slides do not print \(\hat{\boldsymbol\beta}\) for the satisfaction data. Section 05.6 gets it from the interval endpoints on p.51.

**Why** Added

1

\[\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

2

\[(\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\]

3

\[(\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\]

Form \(\boldsymbol A\boldsymbol x=\boldsymbol b\). ∎

### 05.3 Calculate \(\hat\sigma^2\) and the standard errors Slides p.46

**What** Slides p.46

Inference · Lecture 7 · p.46 (statistical content of the comments) \[\hat{\boldsymbol y}=\boldsymbol X\hat{\boldsymbol\beta},\qquad \boldsymbol e=\boldsymbol y-\hat{\boldsymbol y},\qquad \hat\sigma^2=\frac{\sum_i e_i^2}{n-(p+1)}\ \text{(unbiased estimator)}\] \[\text{standard errors: }\ \hat\sigma\sqrt{V_{00}},\ \dots,\ \hat\sigma\sqrt{V_{pp}}\]

Slide p.46 shows only code; this is its statistical content.

\(\hat\sigma^2=\dfrac{\sum_i e_i^2}{n-(p+1)}=\dfrac{\boldsymbol e^T\boldsymbol e}{n-(p+1)}\).

\(\mathrm{SE}(\hat\beta_j)=\hat\sigma\sqrt{V_{jj}}=\sqrt{\hat\sigma^2V_{jj}}\), \(j=0,1,\dots,p\).

\(p\) is the number of columns of \(\boldsymbol X\) minus 1 (the intercept column).

**How** Added

Calculate \(\mathrm{SE}(\hat\beta_j)\)

- Find \(n\) (rows of \(\boldsymbol X\)) and \(p\) (columns of \(\boldsymbol X\) minus 1).
- Calculate \(\hat y_i=\hat\beta_0+\hat\beta_1x_i+\dots\) and \(e_i=y_i-\hat y_i\).
- Calculate \(\hat\sigma^2=\sum_i e_i^2/(n-(p+1))\).
- Calculate \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\). Read \(V_{00},\dots,V_{pp}\).
- Calculate \(\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\).

**Self-check:** The residuals add to 0 (model with intercept).

**Self-check:** \(\boldsymbol V\boldsymbol X^T\boldsymbol X=\boldsymbol I\).

**Example** Added

Data set 4, \(\hat\beta_0=0.6\), \(\hat\beta_1=0.8\).

1

\(n\) and \(p\) How step 1

\[n=5,\qquad p=2-1=1,\qquad n-(p+1)=3\]

2

Fitted values How step 2

\[\hat y_i=0.6+0.8x_i:\quad 0.6+0.8=1.4,\ \ 0.6+1.6=2.2,\ \ 0.6+2.4=3.0,\ \ 0.6+3.2=3.8,\ \ 0.6+4.0=4.6\]

3

Residuals How step 2

\[e_i=y_i-\hat y_i:\quad 1-1.4=-0.4,\ \ 3-2.2=0.8,\ \ 2-3.0=-1.0,\ \ 5-3.8=1.2,\ \ 4-4.6=-0.6\]

Self-check: \(-0.4+0.8-1.0+1.2-0.6=0\).

4

\(\boldsymbol e^T\boldsymbol e\) How step 3

\[\boldsymbol e^T\boldsymbol e=0.16+0.64+1.00+1.44+0.36=3.6\]

5

\(\hat\sigma^2\) How step 3

\[\hat\sigma^2=\frac{3.6}{3}=1.2\]

6

Determinant How step 4

\[\det(\boldsymbol X^T\boldsymbol X)=5\cdot55-15\cdot15=275-225=50\]

7

\(\boldsymbol V\) How step 4

\[\boldsymbol V=\frac{1}{50}\left[\begin{array}{cc}55&-15\\-15&5\end{array}\right]=\left[\begin{array}{cc}1.1&-0.3\\-0.3&0.1\end{array}\right]\]

2×2 inverse: swap diagonal, negate others, divide by determinant.

8

\(\mathrm{SE}(\hat\beta_0)\) How step 5

\[\mathrm{SE}(\hat\beta_0)=\sqrt{1.2\times1.1}=\sqrt{1.32}=1.148913\]

9

\(\mathrm{SE}(\hat\beta_1)\) How step 5

\[\mathrm{SE}(\hat\beta_1)=\sqrt{1.2\times0.1}=\sqrt{0.12}=0.3464102\]

Check \(\boldsymbol V\): \(-0.3\cdot15+0.1\cdot55=-4.5+5.5=1\). ▲

**Why** Added

1

\[\mathrm{Var}[\hat\beta_j]=\sigma^2V_{jj}\]

p.44 (Section 05.1).

2

\[\widehat{\mathrm{Var}}[\hat\beta_j]=\hat\sigma^2V_{jj}=\big[\hat\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1}\big]_{jj}\]

\(\hat\sigma^2\) for the unknown \(\sigma^2\).

3

\[\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\]

Square root of a variance. ∎

### 05.4 Inference rules for \(\beta_j\): test and confidence interval Slides p.47–49

**What** Slides p.47–49

Hypothesis test: a decision, from data, if a statement about a parameter agrees with the data.

Null hypothesis \(H_0:\beta_j=\theta_0\): the statement under test; \(\theta_0\) is a given number.

\(t^{obs}\): the t statistic calculated from the observed data.

Confidence interval (CI): an interval from the data that contains the true \(\beta_j\) with probability \(1-\alpha\).

Inference for \(\beta_j\) · Lecture 7 · p.47–49 We showed \(\dfrac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\sim t_{n-(p+1)}\).
This is identical to the result for SLR except with updated

- Standard errors: \(SE(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\)
- And the d.o.f. of the t distribution is \(n-(p+1)\)

- Note: In SLR, we had \(n-2\), equivalent to \(p=1\) covariate

1. Hypothesis tests:

- Compare \(t^{obs}\) (i.e., realization of \(\frac{\hat\beta_j-\theta_0}{SE(\hat\beta_j)}\)) to null distribution (\(t_{n-(p+1)}\))
2. Confidence intervals:

- \(\hat\beta_j\pm t_{1-\alpha/2,\,n-(p+1)}\,SE(\hat\beta_j)\)

Note on the slide text

Slides p.47–49 print \(SE(\hat\beta_j)=\hat\sigma^2V_{jj}\) (no square root) and \(t_{n-p+1}\) (no brackets). This page writes \(\sqrt{\hat\sigma^2V_{jj}}\) and \(t_{n-(p+1)}\), as on p.44 and p.46.

Quantile \(t_{1-\alpha/2,\,n-(p+1)}\): the point with area \(1-\alpha/2\) to its left under the \(t_{n-(p+1)}\) curve.

Significance level \(\alpha\): the largest accepted probability of a wrong rejection. Usual: \(\alpha=0.05\), giving \(t_{0.975,\,n-(p+1)}\).

Null distribution: the distribution of the statistic when \(H_0\) is true.

p-value: the probability, under \(H_0\), of a value as far from 0 as \(t^{obs}\) or farther. Small values are evidence against \(H_0\).

**How** Added

Test \(H_0:\beta_j=\theta_0\)

- Write \(H_0:\beta_j=\theta_0\). Usually \(\theta_0=0\).
- Calculate \(t^{obs}=(\hat\beta_j-\theta_0)/\mathrm{SE}(\hat\beta_j)\).
- Use the null distribution \(T\sim t_{n-(p+1)}\).
- Calculate the p-value \(=2P(T>|t^{obs}|)\).
- Reject \(H_0\) if the p-value is less than \(\alpha\), or equivalently if \(|t^{obs}|>t_{1-\alpha/2,\,n-(p+1)}\).

**Self-check:** \(t^{obs}\) has the sign of \(\hat\beta_j-\theta_0\). The p-value is between 0 and 1.

\(\theta_0=0\): covariate \(j\) has no linear relation with the mean response when the other covariates stay the same.

Calculate the \(100(1-\alpha)\%\) confidence interval for \(\beta_j\)

- Find \(t_{1-\alpha/2,\,n-(p+1)}\) in a t table.
- Calculate the half-width \(t_{1-\alpha/2,\,n-(p+1)}\times\mathrm{SE}(\hat\beta_j)\).
- Write \(\hat\beta_j\pm\) half-width.

**Self-check:** The midpoint is \(\hat\beta_j\). At the same \(\alpha\), the CI excludes \(\theta_0\) exactly when the test rejects \(H_0:\beta_j=\theta_0\).

**Example** Added

Data set 4: test \(H_0:\beta_1=0\) and find the 95% CI for \(\beta_1\) (\(\alpha=0.05\)).

1

Write \(H_0\) Test step 1

\[H_0:\beta_1=0,\qquad\theta_0=0\]

2

\(t^{obs}\) Test step 2

\[t^{obs}=\frac{0.8-0}{0.3464102}=2.309401\]

Inputs from Sections 05.2 and 05.3.

3

Null distribution Test step 3

\[T\sim t_{5-(1+1)}=t_3\]

4

p-value Test step 4

\[2P(T>2.309401)=2\times0.052044=0.104088\]

5

Decision Test step 5

\[0.104>0.05\ \Rightarrow\ \text{do not reject }H_0\]

Also: \(|t^{obs}|=2.309<t_{0.975,3}=3.182446\).

6

Quantile CI step 1

\[t_{0.975,\,3}=3.182446\]

7

Half-width CI step 2

\[3.182446\times0.3464102=1.102432\]

8

Interval CI step 3

\[0.8\pm1.102432=(0.8-1.102432,\ 0.8+1.102432)=(-0.302432,\ 1.902432)\]

Contains 0, as line 5 requires. ▲

**Why** Added

Let \(q=t_{1-\alpha/2,\,n-(p+1)}\). The t distribution is symmetric about 0, so \(P(-q\le T\le q)=1-\alpha\).

1

\[P\left(-q\le\frac{\hat\beta_j-\beta_j}{\mathrm{SE}(\hat\beta_j)}\le q\right)=1-\alpha\]

Section 05.1.

2

\[P\left(-q\,\mathrm{SE}(\hat\beta_j)\le\hat\beta_j-\beta_j\le q\,\mathrm{SE}(\hat\beta_j)\right)=1-\alpha\]

3

\[P\left(-\hat\beta_j-q\,\mathrm{SE}(\hat\beta_j)\le-\beta_j\le-\hat\beta_j+q\,\mathrm{SE}(\hat\beta_j)\right)=1-\alpha\]

4

\[P\left(\hat\beta_j-q\,\mathrm{SE}(\hat\beta_j)\le\beta_j\le\hat\beta_j+q\,\mathrm{SE}(\hat\beta_j)\right)=1-\alpha\]

Multiply by \(-1\); inequalities reverse. ∎

For the test, \(H_0\) gives \(\beta_j=\theta_0\), so \(T=(\hat\beta_j-\theta_0)/\mathrm{SE}(\hat\beta_j)\sim t_{n-(p+1)}\).

### 05.5 The p-value for each coefficient Slides p.50

**What** Slides p.50

Inference: p-values · Lecture 7 · p.50 (statistical content of the comments) \(H_0:\beta_j=0\), for each \(j=0,\dots,p\).
Numerators of the test statistics: \(\hat\beta_j\). Denominators of the test statistics: \(\mathrm{SE}(\hat\beta_j)\).
p-value: want \(P(|T|>|t^{obs}|)\).

Slide p.50 shows only code; this is its statistical content. With \(\theta_0=0\), \(t^{obs}=\hat\beta_j/\mathrm{SE}(\hat\beta_j)\) and \(T\sim t_{n-(p+1)}\).

The slide gives two equal forms: \(2P(T<-|t^{obs}|)\) (left tail) and \(2P(T>|t^{obs}|)\) (right tail).

**How** Added

Calculate the p-values for \(H_0:\beta_j=0\), \(j=0,\dots,p\)

- Write \(\hat\beta_0,\dots,\hat\beta_p\) (Section 05.2).
- Write \(\mathrm{SE}(\hat\beta_0),\dots,\mathrm{SE}(\hat\beta_p)\) (Section 05.3).
- Calculate \(t^{obs}_j=\hat\beta_j/\mathrm{SE}(\hat\beta_j)\).
- Calculate each p-value \(=2P(T<-|t^{obs}_j|)\), \(T\sim t_{n-(p+1)}\).
- Compare each p-value with \(\alpha\).

**Self-check:** Larger \(|t^{obs}_j|\) gives a smaller p-value.

**Self-check:** Each p-value is between 0 and 1.

**Example** Added

Data set 4 (\(T\sim t_3\), \(\alpha=0.05\)).

1

Numerators and denominators How steps 1–2

\[\hat\beta_0=0.6,\ \ \hat\beta_1=0.8,\qquad \mathrm{SE}(\hat\beta_0)=1.148913,\ \ \mathrm{SE}(\hat\beta_1)=0.3464102\]

2

\(t^{obs}_0\) How step 3

\[t^{obs}_0=\frac{0.6}{1.148913}=0.522233\]

3

\(t^{obs}_1\) How step 3

\[t^{obs}_1=\frac{0.8}{0.3464102}=2.309401\]

4

p-value for \(\beta_0\) How step 4

\[2P(T<-0.522233)=2\times0.318809=0.637618\]

5

p-value for \(\beta_1\) How step 4

\[2P(T<-2.309401)=2\times0.052044=0.104088\]

6

Decisions How step 5

\[0.638>0.05\ \text{and}\ 0.104>0.05\ \Rightarrow\ \text{do not reject }H_0:\beta_0=0\ \text{or}\ H_0:\beta_1=0\]

Self-check: larger \(|t^{obs}_1|\), smaller p-value. ▲

The area under a density curve over an interval is the probability of that interval.

[figure]
Figure 05.1 · Test of \(H_0:\beta_1=0\), data set 4. Blue: \(t_3\) density. Purple dashed: \(\pm|t^{obs}_1|=\pm2.309\). Each orange tail: area 0.052044 (including the part past \(\pm4\)). Sum: p-value 0.104088.

**Why** Added

1

\[P(|T|>|t^{obs}|)=P(T<-|t^{obs}|)+P(T>|t^{obs}|)\]

The tails do not overlap.

2

\[=P(T<-|t^{obs}|)+P(T<-|t^{obs}|)\]

Symmetry about 0.

3

\[=2P(T<-|t^{obs}|)\]

First form on p.50; symmetry gives the second. ∎

### 05.6 The 95% confidence interval for each coefficient Slides p.51

**What** Slides p.51

Slide p.51 calculates \(\hat\beta_j\pm t_{0.975,\,n-(p+1)}\,\mathrm{SE}(\hat\beta_j)\) by two methods with equal results. It prints, for the satisfaction data (\(n=46\), \(p=3\)):

Inference · Lecture 7 · p.51 (printed endpoints)
(Intercept) \(\beta_0\): \((121.911727,\ 195.0707761)\)

Age \(\beta_1\): \((-1.575093,\ -0.7081303)\)

Severity \(\beta_2\): \((-1.434831,\ 0.5508228)\)

Stress \(\beta_3\): \((-27.797859,\ 0.8575324)\)

\((L,\ U)\): lower and upper endpoints. Each uses \(\alpha=0.05\) and \(n-(p+1)=42\).

**How** Added

Get \(\hat\beta_j\) and \(\mathrm{SE}(\hat\beta_j)\) from a printed CI \((L,\ U)\)

- Midpoint \((L+U)/2\) is \(\hat\beta_j\).
- Calculate the half-width \((U-L)/2\).
- Find \(t_{0.975,\,n-(p+1)}\).
- Divide the half-width by the quantile to get \(\mathrm{SE}(\hat\beta_j)\).
- If the CI excludes 0, reject \(H_0:\beta_j=0\) at \(\alpha=0.05\).

**Self-check:** \(L<\hat\beta_j<U\) and \(\mathrm{SE}(\hat\beta_j)>0\).

**Example** Added

Age coefficient \(\beta_1\).

1

Midpoint How step 1

\[\hat\beta_1=\frac{-1.575093+(-0.7081303)}{2}=\frac{-2.2832233}{2}=-1.141612\]

2

Half-width How step 2

\[\frac{-0.7081303-(-1.575093)}{2}=\frac{0.8669627}{2}=0.4334814\]

3

Quantile How step 3

\[t_{0.975,\,42}=2.0180817\]

4

Standard error How step 4

\[\mathrm{SE}(\hat\beta_1)=\frac{0.4334814}{2.0180817}=0.214799\]

5

Test result How step 5

\[0\notin(-1.575093,\ -0.7081303)\ \Rightarrow\ \text{reject }H_0:\beta_1=0\ \text{at}\ \alpha=0.05\]

Check: \(t^{obs}=-1.141612/0.214799=-5.315\), p-value \(3.81\times10^{-6}<0.05\). ▲

Other coefficients, with \(t_{0.975,42}=2.0180817\):

Intercept \(\beta_0\)

\(\hat\beta_0=(121.911727+195.0707761)/2=316.9825031/2=158.491252\).

Half-width \(=(195.0707761-121.911727)/2=73.1590491/2=36.579525\). \(\mathrm{SE}(\hat\beta_0)=36.579525/2.0180817=18.125889\).

Excludes 0. Reject \(H_0:\beta_0=0\).

Severity \(\beta_2\)

\(\hat\beta_2=(-1.434831+0.5508228)/2=-0.8840082/2=-0.442004\).

Half-width \(=(0.5508228-(-1.434831))/2=1.9856538/2=0.992827\). \(\mathrm{SE}(\hat\beta_2)=0.992827/2.0180817=0.491966\).

Contains 0. Do not reject \(H_0:\beta_2=0\).

Stress \(\beta_3\)

\(\hat\beta_3=(-27.797859+0.8575324)/2=-26.9403266/2=-13.470163\).

Half-width \(=(0.8575324-(-27.797859))/2=28.6553914/2=14.327696\). \(\mathrm{SE}(\hat\beta_3)=14.327696/2.0180817=7.099661\).

Contains 0. Do not reject \(H_0:\beta_3=0\).

[figure]
Figure 05.2 · 95% CI for Age \(\beta_1\) (blue). Green: \(\hat\beta_1=-1.1416\). Half-width 0.4335. Purple dashed: 0. The interval lies left of 0: reject \(H_0:\beta_1=0\).

**Why** Added

Let \(q=t_{0.975,\,n-(p+1)}\).

1

\[L=\hat\beta_j-q\,\mathrm{SE}(\hat\beta_j),\qquad U=\hat\beta_j+q\,\mathrm{SE}(\hat\beta_j)\]

CI rule, Section 05.4.

2

\[L+U=2\hat\beta_j,\qquad U-L=2q\,\mathrm{SE}(\hat\beta_j)\]

3

\[\hat\beta_j=\frac{L+U}{2},\qquad \mathrm{SE}(\hat\beta_j)=\frac{(U-L)/2}{q}\]

∎

### 05.7 Practice Added

Satisfaction data: \(n=46\), \(p=3\), \(T\sim t_{42}\). From Section 05.6: \(\hat\beta_1=-1.141612\), \(\mathrm{SE}(\hat\beta_1)=0.214799\), \(\hat\beta_3=-13.470163\), \(\mathrm{SE}(\hat\beta_3)=7.099661\).

**Q1.** Test \(H_0:\beta_3=0\) (Stress) against \(H_1:\beta_3\neq0\) at \(\alpha=0.05\). Give \(t^{obs}\), the null distribution, the p-value, and the decision. Does it agree with the 95% CI on Slides p.51?

Answer Q1

1

\(t^{obs}\) Section 05.4 · Test step 2

\[t^{obs}=\frac{-13.470163-0}{7.099661}=-1.897297\]

2

Null distribution Section 05.4 · Test step 3

\[t_{46-(3+1)}=t_{42}\]

3

p-value Section 05.4 · Test step 4

\[2P(T<-1.897297)=2\times0.032339=0.064678\]

4

Decision Section 05.4 · Test step 5

\[0.0647>0.05\ \Rightarrow\ \text{do not reject }H_0\]

The 95% CI \((-27.797859,\ 0.8575324)\) contains 0. They agree. ▲

**Q2.** Calculate the 90% CI for \(\beta_3\) (Stress) with \(t_{0.95,42}=1.681952\). At \(\alpha=0.10\), do you reject \(H_0:\beta_3=0\)?

Answer Q2

1

Quantile Section 05.4 · CI step 1

\[\alpha=0.10,\quad 1-\alpha/2=0.95,\quad t_{0.95,42}=1.681952\]

2

Half-width Section 05.4 · CI step 2

\[1.681952\times7.099661=11.941289\]

3

Interval Section 05.4 · CI step 3

\[-13.470163\pm11.941289=(-25.411452,\ -1.528874)\]

Excludes 0: reject at \(\alpha=0.10\). Q1 p-value \(0.0647<0.10\) agrees. ▲

**Q3.** Test \(H_0:\beta_1=-1\) (Age) against \(H_1:\beta_1\neq-1\) at \(\alpha=0.05\).

Answer Q3

1

\(t^{obs}\) with \(\theta_0=-1\) Section 05.4 · Test step 2

\[t^{obs}=\frac{-1.141612-(-1)}{0.214799}=\frac{-0.141612}{0.214799}=-0.659276\]

2

p-value with \(T\sim t_{42}\) Section 05.4 · Test step 4

\[2P(T<-0.659276)=0.513317\]

3

Decision Section 05.4 · Test step 5

\[0.513>0.05\ \Rightarrow\ \text{do not reject }H_0:\beta_1=-1\]

The 95% CI \((-1.575093,\ -0.7081303)\) contains \(-1\). They agree. ▲

## 06 · Practice questions Q1–Q3

p.52: section title "Practice". p.53 (Q1): estimates and SEs of \(2\beta_3\) and \(\beta_2+\beta_3\). p.54 (Q2): \(\hat{\boldsymbol\beta}\) and \(\mathrm{SE}(\hat\beta_1)\) from part of \((\boldsymbol{X}'\boldsymbol{X})^{-1}\). p.55 (Q3): three proofs with \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\).

Notation

Slides p.54–55 write the transpose with a prime: \(\boldsymbol{X}'=\boldsymbol{X}^T\). Q2 and Q3 use the prime.

### 06.1 Q1: a linear combination of coefficients and its SE Slides p.52–53

**What** Slides p.52–53

Lecture 7 · p.53 · Practice Q1 Fit the following model: \[\text{Satisfaction}_i=\beta_0+\beta_1\text{Age}_i+\beta_2\text{Severity}_i+\beta_3\text{Stress}_i+\epsilon_i,\quad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] (a) Estimate \(2\beta_3\)
(b) Compute a corresponding SE
(c) Interpret your estimate of \(2\beta_3\)
(d) Repeat (a)–(c) for \(\beta_2+\beta_3\)

Data: the satisfaction data (Unit 05), \(n=46\), \(p=3\). \(\boldsymbol\beta=(\beta_0,\beta_1,\beta_2,\beta_3)^T\). Added

\(\boldsymbol{a}^T\boldsymbol\beta=a_0\beta_0+a_1\beta_1+a_2\beta_2+a_3\beta_3\). For \(2\beta_3\), \(\boldsymbol{a}=(0,0,0,2)^T\). For \(\beta_2+\beta_3\), \(\boldsymbol{a}=(0,0,1,1)^T\).

The estimate of \(\boldsymbol{a}^T\boldsymbol\beta\) is \(\boldsymbol{a}^T\hat{\boldsymbol\beta}\).

\(\mathrm{Var}(\hat{\boldsymbol\beta})\) holds \(\mathrm{Var}(\hat\beta_j)\) at diagonal position \(j\) and \(\mathrm{Cov}(\hat\beta_j,\hat\beta_k)\) at \((j,k)\), numbered 0 to \(p\).

Lecture 5 · p.29 Consider a random vector \(\boldsymbol{y}\) and constant vector \(\boldsymbol{a}\):
\(E[\boldsymbol{a}^T\boldsymbol{y}+b]=\boldsymbol{a}^TE[\boldsymbol{y}]+b\)
\(\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})=\boldsymbol{a}^T\mathrm{Var}(\boldsymbol{y})\boldsymbol{a}\)

Lecture 7 · p.44 \[\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol{X}^T\boldsymbol{X})^{-1})\] and we estimate the variance with \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol{e}^T\boldsymbol{e}\).

With \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\): \(\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})=\hat\sigma^2\boldsymbol{V}\). \(\mathrm{SE}(\hat\beta_j)\) is the square root of its diagonal entry \(j\).

**How** Added

Estimate \(\boldsymbol{a}^T\boldsymbol\beta\) and find its SE

- **Write \(\boldsymbol{a}\).** Write the target as \(a_0\beta_0+\cdots+a_p\beta_p\), with 0 for absent coefficients.
- **Estimate.** Calculate \(\boldsymbol{a}^T\hat{\boldsymbol\beta}\).
- **Estimate the variance.** Calculate \(\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=\hat\sigma^2\,\boldsymbol{a}^T\boldsymbol{V}\boldsymbol{a}\).
- **Take the square root.** \(\mathrm{SE}(\boldsymbol{a}^T\hat{\boldsymbol\beta})=\sqrt{\hat\sigma^2\,\boldsymbol{a}^T\boldsymbol{V}\boldsymbol{a}}\).
- **Interpret.** Lecture 6 · p.14: "To interpret a parameter, isolate it." Write two conditional means whose difference is \(\boldsymbol{a}^T\boldsymbol\beta\).
- **State the result.** In one sentence, say which covariates change, by how much, and which stay fixed.

**Self-check:** \(\boldsymbol{a}\) has 4 entries: intercept, Age, Severity, Stress.

**Self-check:** One nonzero entry \(c\) at position \(j\) gives SE \(=|c|\,\mathrm{SE}(\hat\beta_j)\).

**Self-check:** \(\boldsymbol{a}^T\boldsymbol{V}\boldsymbol{a}\ge0\).

**Example · The satisfaction data** Added

Values from Section 05.6 (Slides p.51 intervals, \(t_{0.975,\,42}=2.0180817\)), 7 decimals. Slides p.51

Severity: \(\hat\beta_2=-0.4420041\), \(\mathrm{SE}(\hat\beta_2)=0.4919657\).

Stress: \(\hat\beta_3=-13.4701633\), \(\mathrm{SE}(\hat\beta_3)=7.0996609\).

Part (d) needs \(\widehat{\mathrm{Cov}}(\hat\beta_2,\hat\beta_3)\), which the slides do not print. These entries of \(\hat\sigma^2\boldsymbol{V}\) come from the course file satisfaction.csv (Lecture 6 · p.9), \(\hat\sigma^2=101.1628734\). Added · not on the slides

\(\hat\sigma^2V_{22}=0.2420303\), \(\hat\sigma^2V_{33}=50.4051837\), \(\hat\sigma^2V_{23}=-1.7916031\).

Check: \(0.4919657^2=0.2420302\), \(7.0996609^2=50.4051849\). The differences come from rounding.

1

(a) Write \(\boldsymbol{a}\) and estimate How steps 1–2

\[2\beta_3=\boldsymbol{a}^T\boldsymbol\beta,\quad \boldsymbol{a}=(0,0,0,2)^T\]

2

\[\boldsymbol{a}^T\hat{\boldsymbol\beta}=0\,\hat\beta_0+0\,\hat\beta_1+0\,\hat\beta_2+2\,\hat\beta_3=2(-13.4701633)\]

3

\[2\hat\beta_3=-26.9403266\]

4

(b) Estimate the variance How step 3

\[\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=2^2\,\hat\sigma^2V_{33}\]

Only \(a_3\) is nonzero.

5

Take the square root How step 4

\[\mathrm{SE}(2\hat\beta_3)=\sqrt{2^2\,\hat\sigma^2V_{33}}=2\sqrt{\hat\sigma^2V_{33}}=2\,\mathrm{SE}(\hat\beta_3)\]

6

\[\mathrm{SE}(2\hat\beta_3)=2(7.0996609)=14.1993218\]

**(c) Interpret \(2\hat\beta_3\).** How step 5.

1

\[E[y_i\mid \text{Age}=x_1,\text{Severity}=x_2,\text{Stress}=x_3]=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_3\]

2

\[E[y_i\mid \text{Age}=x_1,\text{Severity}=x_2,\text{Stress}=x_3+2]=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3(x_3+2)\]

3

\[\text{line 2}-\text{line 1}=2\beta_3\]

Stress +2; Age and Severity fixed.

Interpretation: with age and severity fixed, we estimate that a 2-unit increase in stress is associated with a 26.94-unit decrease in mean satisfaction (SE = 14.20).

**(d) Repeat (a)–(c) for \(\beta_2+\beta_3\).** Two nonzero entries, so a covariance term enters.

1

Write \(\boldsymbol{a}\) and estimate How steps 1–2

\[\beta_2+\beta_3=\boldsymbol{a}^T\boldsymbol\beta,\quad \boldsymbol{a}=(0,0,1,1)^T\]

2

\[\hat\beta_2+\hat\beta_3=-0.4420041+(-13.4701633)\]

3

\[\hat\beta_2+\hat\beta_3=-13.9121674\]

4

Estimate the variance How step 3

\[\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=\hat\sigma^2V_{22}+\hat\sigma^2V_{33}+2\hat\sigma^2V_{23}\]

Derived under Why.

5

\[\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=0.2420303+50.4051837+2(-1.7916031)\]

6

\[\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=50.6472140-3.5832062=47.0640078\]

7

Take the square root How step 4

\[\mathrm{SE}(\hat\beta_2+\hat\beta_3)=\sqrt{47.0640078}=6.8603213\]

Negative covariance lowers 50.65 to 47.06.

1

\[E[y_i\mid \text{Age}=x_1,\text{Severity}=x_2,\text{Stress}=x_3]=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_3\]

2

\[E[y_i\mid \text{Age}=x_1,\text{Severity}=x_2+1,\text{Stress}=x_3+1]=\beta_0+\beta_1x_1+\beta_2(x_2+1)+\beta_3(x_3+1)\]

3

\[\text{line 2}-\text{line 1}=\beta_2+\beta_3\]

Severity +1, Stress +1; Age fixed.

Interpretation: with age fixed, we estimate that a 1-unit increase in both severity and stress is associated with a 13.91-unit decrease in mean satisfaction (SE = 6.86).

**Why** Added

1

\[\mathrm{Var}(\boldsymbol{a}^T\hat{\boldsymbol\beta})=\boldsymbol{a}^T\mathrm{Var}(\hat{\boldsymbol\beta})\boldsymbol{a}\]

Lecture 5 · p.29, \(\hat{\boldsymbol\beta}\) for \(\boldsymbol{y}\).

2

\[\mathrm{Var}(\boldsymbol{a}^T\hat{\boldsymbol\beta})=\boldsymbol{a}^T\sigma^2\boldsymbol{V}\boldsymbol{a}\]

Slide p.44.

3

\[\mathrm{Var}(\boldsymbol{a}^T\hat{\boldsymbol\beta})=\sigma^2\sum_{j=0}^{p}\sum_{k=0}^{p}a_ja_kV_{jk}\]

Expand entry by entry.

4

\[\mathrm{Var}(\hat\beta_2+\hat\beta_3)=\sigma^2(1\cdot1\cdot V_{22}+1\cdot1\cdot V_{23}+1\cdot1\cdot V_{32}+1\cdot1\cdot V_{33})\]

\(a_2=a_3=1\); other \(a_j=0\).

5

\[\mathrm{Var}(\hat\beta_2+\hat\beta_3)=\sigma^2(V_{22}+V_{33}+2V_{23})\]

Symmetry: \(V_{32}=V_{23}\).

6

\[\mathrm{SE}(\hat\beta_2+\hat\beta_3)=\sqrt{\hat\sigma^2(V_{22}+V_{33}+2V_{23})}\]

\(\hat\sigma^2\) for \(\sigma^2\); square root. ∎

For a sum of two or more coefficients, covariance terms enter the SE.

### 06.2 Q2: find \(\hat{\boldsymbol\beta}\) and an SE from an incomplete \((\boldsymbol{X}'\boldsymbol{X})^{-1}\) Slides p.54

**What** Slides p.54

Lecture 7 · p.54 · Practice Q2 Consider the usual multiple linear regression model.
(a) Suppose that \(n=50\) and \[(\boldsymbol{X}'\boldsymbol{X})^{-1}=\left[\begin{array}{ccc}1&.25&.25\\?&.5&-.25\\?&?&2\end{array}\right],\quad \boldsymbol{X}'\boldsymbol{y}=\left[\begin{array}{c}2\\1.2\\-3\end{array}\right],\] but unfortunately the values of the elements in the lower corner of \((\boldsymbol{X}'\boldsymbol{X})^{-1}\) have been lost. Can you still find \(\hat{\boldsymbol\beta}\)? If so, please calculate it. Otherwise, explain why not.
(b) In addition to the data above, suppose that \(\sum_{i=1}^n(y_i-\boldsymbol{x}_i'\hat{\boldsymbol\beta})^2=2.5\). What is the standard error of \(\hat\beta_1\)?

"The usual model": \(\boldsymbol{y}=\boldsymbol{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\), \(\boldsymbol{X}\) is \(n\times(p+1)\) with a first column of 1s. Added

\(\boldsymbol{x}_i'\) is row \(i\) of \(\boldsymbol{X}\), so \(\boldsymbol{x}_i'\hat{\boldsymbol\beta}=\hat y_i\). The sum in (b) is \(\boldsymbol{e}'\boldsymbol{e}\).

\((\boldsymbol{X}'\boldsymbol{X})^{-1}\) is symmetric, so \(A_{jk}=A_{kj}\) recovers each lost entry.

**How** Added

Find \(\hat{\boldsymbol\beta}\) and \(\mathrm{SE}(\hat\beta_j)\) from matrices only

- **Complete \(\boldsymbol{V}=(\boldsymbol{X}'\boldsymbol{X})^{-1}\).** Copy each upper entry to its mirror below the diagonal.
- **Calculate \(\hat{\boldsymbol\beta}=(\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y}\).** Entry \(j\): row \(j\) of \(\boldsymbol{V}\) times \(\boldsymbol{X}'\boldsymbol{y}\), entry by entry, then add.
- **Find \(p\) and the degrees of freedom.** \(\boldsymbol{V}\) is \((p+1)\times(p+1)\). Calculate \(\hat\sigma^2=\boldsymbol{e}'\boldsymbol{e}/(n-(p+1))\).
- **Calculate the SE.** \(\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\). \(V_{11}\) is in row 2, column 2.

**Self-check:** \(\boldsymbol{V}'=\boldsymbol{V}\). Each diagonal entry is positive.

**Self-check:** \(\hat{\boldsymbol\beta}\) has length \(p+1\), as \(\boldsymbol{X}'\boldsymbol{y}\).

**Example · Q2** Added

1

Complete \(\boldsymbol{V}\) How step 1

\[V_{10}=V_{01}=.25,\quad V_{20}=V_{02}=.25,\quad V_{21}=V_{12}=-.25\]

2

\[(\boldsymbol{X}'\boldsymbol{X})^{-1}=\left[\begin{array}{ccc}1&.25&.25\\.25&.5&-.25\\.25&-.25&2\end{array}\right]\]

(a): yes. Why proves the symmetry.

3

Calculate \(\hat{\boldsymbol\beta}\) How step 2

\[\hat{\boldsymbol\beta}=\left[\begin{array}{ccc}1&.25&.25\\.25&.5&-.25\\.25&-.25&2\end{array}\right]\left[\begin{array}{c}2\\1.2\\-3\end{array}\right]\]

4

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}1(2)+.25(1.2)+.25(-3)\\.25(2)+.5(1.2)+(-.25)(-3)\\.25(2)+(-.25)(1.2)+2(-3)\end{array}\right]\]

5

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}2+0.3-0.75\\0.5+0.6+0.75\\0.5-0.3-6\end{array}\right]\]

6

\[\hat{\boldsymbol\beta}=\left[\begin{array}{c}1.55\\1.85\\-5.80\end{array}\right]\]

7

Find \(p\) and \(\hat\sigma^2\) How step 3

\[p+1=3\ \Rightarrow\ p=2,\qquad n-(p+1)=50-3=47\]

8

\[\hat\sigma^2=\frac{\sum_{i=1}^n(y_i-\boldsymbol{x}_i'\hat{\boldsymbol\beta})^2}{n-(p+1)}=\frac{2.5}{47}=0.05319149\]

9

Calculate the SE How step 4

\[\mathrm{SE}(\hat\beta_1)=\sqrt{\hat\sigma^2V_{11}}=\sqrt{\frac{2.5}{47}\times0.5}\]

\(V_{11}=.5\) (row 2, column 2).

10

\[\mathrm{SE}(\hat\beta_1)=\sqrt{0.02659574}\]

11

\[\mathrm{SE}(\hat\beta_1)=0.1630820\]

**Why** Added

Rules: \((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\); \((\boldsymbol{A}^{-1})'=(\boldsymbol{A}')^{-1}\) for invertible \(\boldsymbol{A}\).

1

\[(\boldsymbol{X}'\boldsymbol{X})'=\boldsymbol{X}'(\boldsymbol{X}')'\]

First rule, \(\boldsymbol{A}=\boldsymbol{X}'\), \(\boldsymbol{B}=\boldsymbol{X}\).

2

\[(\boldsymbol{X}'\boldsymbol{X})'=\boldsymbol{X}'\boldsymbol{X}\]

\((\boldsymbol{X}')'=\boldsymbol{X}\): \(\boldsymbol{X}'\boldsymbol{X}\) is symmetric.

3

\[\big((\boldsymbol{X}'\boldsymbol{X})^{-1}\big)'=\big((\boldsymbol{X}'\boldsymbol{X})'\big)^{-1}\]

Second rule.

4

\[\big((\boldsymbol{X}'\boldsymbol{X})^{-1}\big)'=(\boldsymbol{X}'\boldsymbol{X})^{-1}\]

Line 2. ∎

### 06.3 Q3: three equations about residuals Slides p.55

Tools: \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\), \(\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}\), and \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\) (slide p.9).

**What** Slides p.55

Lecture 7 · p.55 · Practice Q3 Consider the usual multiple linear regression model.
(a) Let \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\) and \(\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}\). Show that \(\sum_{i=1}^n\hat y_ie_i=\hat{\boldsymbol{y}}'\boldsymbol{e}=0\).
(b) Show that \(\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}\).
(c) Show that for any vector \(\boldsymbol{a}=(a_1,\dots,a_p)\), \(\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=0\).

Lecture 7 · p.9 Note: \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\). Why? \[\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{H}\boldsymbol{y})=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{X}^T\boldsymbol{y}-\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{0}\]

\(\boldsymbol{u}'\boldsymbol{v}=\sum_iu_iv_i\) is one number, so \(\sum_{i=1}^n\hat y_ie_i=\hat{\boldsymbol{y}}'\boldsymbol{e}\). Added

\(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\): \(\boldsymbol{e}\) has inner product 0 with each column of \(\boldsymbol{X}\).

In (c), \(\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}\) needs \(\boldsymbol{a}\) with \(p+1\) entries; the slide writes \((a_1,\dots,a_p)\). The proof holds for any \(\boldsymbol{a}\) of the correct length.

**How** Added

Prove a residual equation with \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\)

- **Substitute a definition.** Replace \(\hat{\boldsymbol{y}}\) with \(\boldsymbol{X}\hat{\boldsymbol\beta}\), or \(\boldsymbol{e}\) with \(\boldsymbol{y}-\hat{\boldsymbol{y}}\).
- **Expand the transpose.** Use \((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\) to get a term with \(\boldsymbol{X}'\boldsymbol{e}\).
- **Substitute \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\).**
- **Simplify.** Use \(c'=c\) for each \(1\times1\) number \(c\).

**Self-check:** \(\hat{\boldsymbol{y}}'\boldsymbol{e}\), \(\boldsymbol{e}'\boldsymbol{e}\), and \(\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}\) are all \(1\times1\).

**Example · Data set 2** Added

Data set 2 (Unit 02): \(n=4\), \(p=1\), \(x=(0,1,2,3)\), \(y=(1,3,2,5)\), \(\boldsymbol{X}'\boldsymbol{X}=\left[\begin{array}{cc}4&6\\6&14\end{array}\right]\), \(\hat{\boldsymbol\beta}=(1.1,\ 1.1)'\).

\(\hat{\boldsymbol{y}}=(1.1,\ 2.2,\ 3.3,\ 4.4)'\), \(\boldsymbol{e}=(-0.1,\ 0.8,\ -1.3,\ 0.6)'\).

1

Check \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\)

\[\sum e_i=-0.1+0.8-1.3+0.6=0,\qquad \sum x_ie_i=0+0.8-2.6+1.8=0\]

2

(a) \(\hat{\boldsymbol{y}}'\boldsymbol{e}\)

\[\hat{\boldsymbol{y}}'\boldsymbol{e}=1.1(-0.1)+2.2(0.8)+3.3(-1.3)+4.4(0.6)=-0.11+1.76-4.29+2.64=0\]

3

(b) \(\boldsymbol{e}'\boldsymbol{e}\) and \(\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}\)

\[\boldsymbol{e}'\boldsymbol{e}=0.01+0.64+1.69+0.36=2.70,\qquad \boldsymbol{y}'\boldsymbol{y}=1+9+4+25=39\]

4

\[\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}=\left[\begin{array}{c}4(1.1)+6(1.1)\\6(1.1)+14(1.1)\end{array}\right]=\left[\begin{array}{c}11\\22\end{array}\right],\qquad \hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}=1.1(11)+1.1(22)=36.3\]

5

\[\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}=39-36.3=2.7=\boldsymbol{e}'\boldsymbol{e}\]

6

(c) Take \(\boldsymbol{a}=(2,-1)'\)

\[\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=2(0)+(-1)(0)=0\]

\(\boldsymbol{X}'\boldsymbol{e}=(0,0)'\), so any \(\boldsymbol{a}\) gives 0.

Figure 06.1 shows (a) for the Q1 model (46 patients). Each point is \((\hat y_i,e_i)\). All \(\hat y_i>0\), so green points (\(e_i>0\)) give positive \(\hat y_ie_i\). Added

[figure]
Figure 06.1 · Residuals against fitted values, Q1 model. 25 green points: \(\sum\hat y_ie_i=11755.906\). 21 orange points (\(e_i<0\)): −11755.906. They cancel: \(\hat{\boldsymbol{y}}'\boldsymbol{e}=0\).

Write each proof with the How steps before you open the answer.

**Why** Added

Answer Q3(a): \(\hat{\boldsymbol{y}}'\boldsymbol{e}=0\)

1

\[\sum_{i=1}^n\hat y_ie_i=\hat{\boldsymbol{y}}'\boldsymbol{e}\]

Inner product.

2

\[\hat{\boldsymbol{y}}'\boldsymbol{e}=(\boldsymbol{X}\hat{\boldsymbol\beta})'\boldsymbol{e}\]

\(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\).

3

\[\hat{\boldsymbol{y}}'\boldsymbol{e}=\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{e}\]

\((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\).

4

\[\hat{\boldsymbol{y}}'\boldsymbol{e}=\hat{\boldsymbol\beta}'\boldsymbol{0}\]

Slide p.9.

5

\[\hat{\boldsymbol{y}}'\boldsymbol{e}=0\]

∎

Answer Q3(b): \(\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}\)

Plan: write the first \(\boldsymbol{e}\) as \(\boldsymbol{y}-\hat{\boldsymbol{y}}\), remove \(\hat{\boldsymbol{y}}'\boldsymbol{e}\) by (a), then use the normal equations \(\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{X}'\boldsymbol{y}\) (slide p.4).

1

\[\boldsymbol{e}'\boldsymbol{e}=(\boldsymbol{y}-\hat{\boldsymbol{y}})'\boldsymbol{e}\]

First factor only.

2

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{e}-\hat{\boldsymbol{y}}'\boldsymbol{e}\]

3

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{e}\]

Part (a).

4

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'(\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta})\]

\(\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}\).

5

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-\boldsymbol{y}'\boldsymbol{X}\hat{\boldsymbol\beta}\]

6

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-(\boldsymbol{X}'\boldsymbol{y})'\hat{\boldsymbol\beta}\]

\(\boldsymbol{y}'\boldsymbol{X}=(\boldsymbol{X}'\boldsymbol{y})'\).

7

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-(\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta})'\hat{\boldsymbol\beta}\]

Normal equations.

8

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'(\boldsymbol{X}'\boldsymbol{X})'\hat{\boldsymbol\beta}\]

\((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\).

9

\[\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}\]

\(\boldsymbol{X}'\boldsymbol{X}\) is symmetric (Section 06.2). ∎

Answer Q3(c): \(\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=0\)

1

\[\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{a}'(\boldsymbol{X}'\boldsymbol{e})\]

Associative.

2

\[\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{a}'\boldsymbol{0}\]

Slide p.9.

3

\[\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=0\]

True for all \(\boldsymbol{a}\). Part (a) is \(\boldsymbol{a}=\hat{\boldsymbol\beta}\). ∎

### 06.4 Practice Added

**P1.** For the Q1 model, estimate \(\beta_1-\beta_2\), give its SE, and interpret it. Use \(\hat\beta_1=-1.1416117\) (midpoint of the Age interval, slide p.51), \(\hat\sigma^2V_{11}=0.0461385\), and \(\hat\sigma^2V_{12}=-0.0322300\). The last two come from satisfaction.csv, not the slides.

Answer P1

1

Write \(\boldsymbol{a}\) and estimate Section 06.1, How steps 1–2

\[\boldsymbol{a}=(0,1,-1,0)^T,\quad \hat\beta_1-\hat\beta_2=-1.1416117-(-0.4420041)=-0.6996076\]

2

Estimate the variance Section 06.1, How step 3

\[\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=\hat\sigma^2V_{11}+\hat\sigma^2V_{22}-2\hat\sigma^2V_{12}\]

\(a_1=1\), \(a_2=-1\): the cross term is negative.

3

\[\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=0.0461385+0.2420303-2(-0.0322300)=0.2881688+0.0644600=0.3526288\]

4

Take the square root Section 06.1, How step 4

\[\mathrm{SE}(\hat\beta_1-\hat\beta_2)=\sqrt{0.3526288}=0.5938256\]

5

Interpret Section 06.1, How steps 5–6

\[E[y_i\mid x_1+1,x_2,x_3]-E[y_i\mid x_1,x_2+1,x_3]=\beta_1-\beta_2\]

Group 1: Age +1. Group 2: Severity +1. Stress equal. Group 1 mean satisfaction: 0.70 lower (SE = 0.59).

**P2.** From Q1(d), construct a 95% confidence interval for \(\beta_2+\beta_3\). Use \(t_{0.975,\,42}=2.0180817\).

Answer P2

Form of slide p.49: estimate \(\pm\ t_{1-\alpha/2,\,n-(p+1)}\times\mathrm{SE}\), with \(n-(p+1)=46-4=42\).

1

\[(\hat\beta_2+\hat\beta_3)\pm t_{0.975,\,42}\,\mathrm{SE}(\hat\beta_2+\hat\beta_3)\]

2

\[-13.9121674\pm2.0180817\times6.8603213\]

From Q1(d).

3

\[-13.9121674\pm13.8446889\]

4

\[(-27.7568563,\ -0.0674785)\]

Excludes 0.

**P3.** Q2 setup. Calculate \(\mathrm{SE}(\hat\beta_2)\) and \(t^{obs}\) for \(H_0:\beta_2=0\). Test at \(\alpha=0.05\) with \(t_{0.975,\,47}=2.0117405\).

Answer P3

1

Calculate the SE Section 06.2, How step 4

\[\mathrm{SE}(\hat\beta_2)=\sqrt{\hat\sigma^2V_{22}}=\sqrt{\frac{2.5}{47}\times2}=\sqrt{0.10638298}\]

\(V_{22}=2\) (row 3, column 3).

2

\[\mathrm{SE}(\hat\beta_2)=0.3261640\]

3

Test statistic Slides p.48

\[t^{obs}=\frac{\hat\beta_2-0}{\mathrm{SE}(\hat\beta_2)}=\frac{-5.80}{0.3261640}=-17.7825\]

4

Compare with the null distribution \(t_{47}\)

\[|t^{obs}|=17.7825>2.0117405=t_{0.975,\,47}\]

Reject \(H_0:\beta_2=0\) at \(\alpha=0.05\).