Lecture 7
Multiple Linear Regression: Estimation and Inference I
This lecture finds the distributions of \(\hat{\boldsymbol\beta}\), \(\boldsymbol e\), and \(\hat\sigma^2\) and uses them for t tests and confidence intervals for each \(\beta_j\) (Lecture 7 · p.1–55).
Unit 01Recap, fitted values, residuals, and the hat matrix
All examples use data set 1 (\(n=5\)), small enough to do by hand.
Step 1: Model, scalar and matrix form (Slides p.1–3).
Step 2: Least squares estimate \(\hat{\boldsymbol\beta}\) (Slides p.4).
Step 3: Distribution of \(\hat{\boldsymbol\beta}\) (Slides p.5).
Step 4: Fitted values and the hat matrix \(\boldsymbol{H}\) (Slides p.6–7).
Step 5: Residuals and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) (Slides p.8–9).
01.1Recap: the model Slides p.1–3
p.1: title page, "Multiple Linear Regression—Estimation and Inference I". p.2: section page "Recap". p.3–5 repeat Lecture 6.
What Slides p.3
Multiple linear regression: a linear model that uses \(p\) explanatory variables to describe one response.
Response \(y_i\): the value we explain in observation \(i\), \(i=1,\dots,n\). Sample size \(n\): the number of observations.
Covariate \(x_{ij}\): the value of explanatory variable \(j\) in observation \(i\), \(j=1,\dots,p\).
Regression coefficients \(\beta_0,\dots,\beta_p\): unknown constants. \(\beta_0\) is the intercept.
\(\beta_j\): the change in the mean of \(y_i\) when \(x_{ij}\) increases by 1 and other covariates stay the same.
Error \(\epsilon_i\): the random part of \(y_i\) that the covariates do not explain.
\(\overset{iid}{\sim}\): independent and identically distributed (the errors do not affect each other and have one distribution).
Normal distribution \(N(\mu,\sigma^2)\): the bell-shaped distribution with mean \(\mu\) and variance \(\sigma^2\).
Expectation (mean) \(E[Z]\): the theoretical average of \(Z\). Variance \(\mathrm{Var}(Z)=E[(Z-E[Z])^2]\): the spread of \(Z\).
Bold letters are vectors or matrices.
\(\boldsymbol{y}\): the \(n\times1\) response vector.
\(\boldsymbol{X}\): the \(n\times(p+1)\) design matrix. Column 1 is all 1s (intercept). Column \(j+1\) is covariate \(j\). Row \(i\) is observation \(i\).
\(\boldsymbol\beta=(\beta_0,\beta_1,\dots,\beta_p)^T\): the coefficient vector. Transpose \(T\): rows become columns.
\(\boldsymbol\epsilon\): the \(n\times1\) error vector.
\(MVN(\boldsymbol\mu,\boldsymbol\Sigma)\): multivariate normal with mean vector \(\boldsymbol\mu\) and variance matrix \(\boldsymbol\Sigma\).
Variance matrix: variances on the diagonal, covariances elsewhere. Covariance: how two random variables change together.
\(\boldsymbol{I}\): the \(n\times n\) identity matrix (1s on the diagonal, 0s elsewhere).
\(\sigma^2\boldsymbol{I}\): each \(\epsilon_i\) has variance \(\sigma^2\); each pair has covariance 0.
The matrix form puts the \(n\) equations into one. All derivations use it.
How Added
- Put the response column in \(\boldsymbol{y}\).
- Count the covariates: \(p\).
- Write a column of 1s.
- Put the \(p\) covariate columns to its right. The result is \(\boldsymbol{X}\).
Self-check: \(\boldsymbol{X}\) is \(n\times(p+1)\). \(\boldsymbol{y}\) has \(n\) rows.
Self-check: Column 1 of \(\boldsymbol{X}\) is all 1s.
Example · Data set 1 Added
A made-up data set: \(n=5\), \(p=2\).
Write the model How step 2
2 covariates: \(p=2\).
Write \(\boldsymbol{y}\) and \(\boldsymbol{X}\) How steps 1, 3, 4
Self-check: \(5\times3=5\times(2+1)\); column 1 all 1s. ▲
Why · Row \(i\) of the matrix form is the scalar model Added
Entry \(i\) of \(\boldsymbol{X}\boldsymbol\beta\) is row \(i\) of \(\boldsymbol{X}\) times \(\boldsymbol\beta\): multiply entry by entry, then add.
Row \(i\) of \(\boldsymbol{X}\).
The scalar model on p.3. ∎
Adding the constant \(\boldsymbol{X}\boldsymbol\beta\) to \(\boldsymbol\epsilon\sim MVN(\boldsymbol{0},\sigma^2\boldsymbol{I})\) moves the mean and keeps the variance. This gives \(\boldsymbol{y}\sim MVN(\boldsymbol{X}\boldsymbol\beta,\sigma^2\boldsymbol{I})\).
01.2Recap: least squares estimation Slides p.4
Estimate: a value calculated from the data to approximate an unknown parameter. A hat marks an estimate.
What Slides p.4
Least squares: select \(\boldsymbol\beta\) to make the sum of squared differences between \(\boldsymbol{y}\) and \(\boldsymbol{X}\boldsymbol\beta\) smallest.
Sum of squares, scalar form: \(S(\boldsymbol\beta)=\sum_{i=1}^n(y_i-\beta_0-\beta_1x_{i1}-\cdots-\beta_px_{ip})^2\).
Gradient \(\partial S/\partial\boldsymbol\beta\): the column of partial derivatives of \(S\) for \(\beta_0,\dots,\beta_p\).
At the minimum the gradient is \(\boldsymbol{0}\). This gives the normal equations \((\boldsymbol{X}^T\boldsymbol{X})\boldsymbol\beta=\boldsymbol{X}^T\boldsymbol{y}\): \(p+1\) equations, \(p+1\) unknowns.
Inverse \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\): the matrix with \((\boldsymbol{X}^T\boldsymbol{X})^{-1}(\boldsymbol{X}^T\boldsymbol{X})=\boldsymbol{I}\).
Linearly independent columns: no column is a linear combination of the others. Then \(\boldsymbol{X}^T\boldsymbol{X}\) is invertible.
How Added
- Calculate \(\boldsymbol{X}^T\boldsymbol{X}\). Entry \((j,k)\): multiply columns \(j\) and \(k\) of \(\boldsymbol{X}\) entry by entry, then add.
- Calculate \(\boldsymbol{X}^T\boldsymbol{y}\). Entry \(j\): multiply column \(j\) and \(\boldsymbol{y}\) entry by entry, then add.
- Calculate \((\boldsymbol{X}^T\boldsymbol{X})^{-1}\).
- Multiply: \(\hat{\boldsymbol\beta}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\boldsymbol{y}\).
Self-check: \(\boldsymbol{X}^T\boldsymbol{X}\) is \((p+1)\times(p+1)\), symmetric (equal to its transpose), top-left entry \(n\).
Self-check: \((\boldsymbol{X}^T\boldsymbol{X})\hat{\boldsymbol\beta}=\boldsymbol{X}^T\boldsymbol{y}\).
Example · Data set 1 Added
Diagonal of \(\boldsymbol{X}^T\boldsymbol{X}\) How step 1
Other entries How step 1
Self-check: \(3\times3\), symmetric, top-left \(5=n\).
\(\boldsymbol{X}^T\boldsymbol{y}\) How step 2
Inverse How step 3
Diagonal matrix: invert each diagonal entry.
Multiply How step 4
Self-check: \(5\times7=35\), \(10\times2.3=23\), \(4\times0.75=3\). ▲
Fitted model: \(\hat y=7+2.3x_1+0.75x_2\). \(\boldsymbol{X}^T\boldsymbol{X}\) is diagonal because the columns of \(\boldsymbol{X}\) are orthogonal.
Inner product of two vectors: the sum of the products of their entries. Orthogonal: inner product 0.
Why · The formula for \(\hat{\boldsymbol\beta}\) Slides p.4
Rule A: \((\boldsymbol{A}\boldsymbol{B})^T=\boldsymbol{B}^T\boldsymbol{A}^T\).
Rule B: A \(1\times1\) matrix is a number and equals its transpose.
Rule C: \(\partial(\boldsymbol{a}^T\boldsymbol\beta)/\partial\boldsymbol\beta=\boldsymbol{a}\). For symmetric \(\boldsymbol{A}\), \(\partial(\boldsymbol\beta^T\boldsymbol{A}\boldsymbol\beta)/\partial\boldsymbol\beta=2\boldsymbol{A}\boldsymbol\beta\).
Rule A.
Rule B: \(\boldsymbol\beta^T\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{y}^T\boldsymbol{X}\boldsymbol\beta\).
Rule C, \(\boldsymbol{a}=\boldsymbol{X}^T\boldsymbol{y}\), \(\boldsymbol{A}=\boldsymbol{X}^T\boldsymbol{X}\).
Independent columns: the inverse exists. ∎
01.3Recap: properties of \(\hat{\boldsymbol\beta}\) Slides p.5
Sampling distribution: the distribution of \(\hat{\boldsymbol\beta}\) over many new data sets with the same \(\boldsymbol{X}\).
What Slides p.5
Note: the notation here is a little funky due to 0-indexing \(\boldsymbol\beta\): \[\mathrm{Var}(\hat{\boldsymbol\beta})=\mathrm{Var}\left(\left[\begin{array}{c}\hat\beta_0\\\hat\beta_1\\\vdots\\\hat\beta_p\end{array}\right]\right)=\sigma^2\left[\begin{array}{cccc}V_{00}&V_{01}&\cdots&V_{0p}\\V_{10}&V_{11}&\cdots&V_{1p}\\\vdots&&&\vdots\\V_{p0}&V_{p1}&\cdots&V_{pp}\end{array}\right]\]
\(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\): \(\hat{\boldsymbol\beta}\) is unbiased. Unbiased: the mean of the estimator equals the parameter.
\(\mathrm{Var}(\hat\beta_j)=\sigma^2V_{jj}\), with \(V_{jj}\) a diagonal entry of \(\boldsymbol{V}\).
0-indexing: row and column numbers of \(\boldsymbol{V}\) start at 0. \(V_{jj}\) is in row \(j+1\), column \(j+1\).
How Added
- Calculate \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\).
- Find \(V_{jj}\) in row \(j+1\), column \(j+1\).
- Write \(\mathrm{Var}(\hat\beta_j)=\sigma^2V_{jj}\) and \(\hat\beta_j\sim N(\beta_j,\sigma^2V_{jj})\).
Self-check: \(V_{jj}>0\), because a variance cannot be negative.
Self-check: Row number \(=j+1\).
Example · Data set 1 Added
\(\boldsymbol{V}\) How step 1
Section 01.2, step 6.
Diagonal entries How step 2
Distributions How step 3
Self-check: all positive. ▲
\(\sigma^2\) is unknown. Unit 02 estimates it from the residuals.
Why · The distribution of \(\hat{\boldsymbol\beta}\) Added
If \(\boldsymbol{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) and \(\boldsymbol{A}\) is constant, then \(\boldsymbol{A}\boldsymbol{y}\sim MVN(\boldsymbol{A}\boldsymbol\mu,\boldsymbol{A}\boldsymbol\Sigma\boldsymbol{A}^T)\). Let \(\boldsymbol{A}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\boldsymbol{X}^T\), so \(\hat{\boldsymbol\beta}=\boldsymbol{A}\boldsymbol{y}\).
Mean
Substitute \(\boldsymbol\mu=\boldsymbol{X}\boldsymbol\beta\).
Variance
\(\boldsymbol\Sigma=\sigma^2\boldsymbol{I}\); Rule A for \(\boldsymbol{A}^T\).
Inverse of a symmetric matrix is symmetric. ∎
01.4Fitted values and the hat matrix Slides p.6–7
p.6 repeats the title page. New material starts here.
What Slides p.7
Fitted value: \(\hat y_i=\hat\beta_0+\hat\beta_1x_{i1}+\cdots+\hat\beta_px_{ip}\), the value the estimated model gives for observation \(i\).
Hat matrix \(\boldsymbol{H}\): the \(n\times n\) matrix that "puts the hat on" \(\boldsymbol{y}\).
\(\boldsymbol{H}\) depends only on \(\boldsymbol{X}\), not on \(\boldsymbol{y}\).
Each \(\hat y_i\) is a linear combination of \(y_1,\dots,y_n\), with weights from row \(i\) of \(\boldsymbol{H}\).
How Added
- Calculate \(\hat{\boldsymbol\beta}\) (Section 01.2).
- For each \(i\), multiply row \(i\) of \(\boldsymbol{X}\) by \(\hat{\boldsymbol\beta}\) entry by entry, then add. The result is \(\hat y_i\).
- For \(\boldsymbol{H}\), entry \((i,k)\) is row \(i\) of \(\boldsymbol{X}\), times \(\boldsymbol{V}\), times row \(k\) transposed.
Self-check: \(\boldsymbol{H}\) is \(n\times n\) and symmetric.
Self-check: Row \(i\) of \(\boldsymbol{H}\) times \(\boldsymbol{y}\) equals \(\hat y_i\) from step 2.
Example · Data set 1 Added
Fitted value of observation 2 How step 2
Row 2 of \(\boldsymbol{X}\): \((1,\,-1,\,-1)\).
All fitted values How step 2
Entries of \(\boldsymbol{H}\) How step 3
\(\boldsymbol{V}\) is diagonal: three terms.
Self-check: \(5\times5\), symmetric.
Check row 1 of \(\boldsymbol{H}\boldsymbol{y}\) How self-check
Equals \(\hat y_1\) from step 3. ▲
Why · \(\hat{\boldsymbol{y}}=\boldsymbol{H}\boldsymbol{y}\) Slides p.7
Substitute \(\hat{\boldsymbol\beta}\).
Matrix multiplication is associative.
Size: \((n\times(p+1))((p+1)\times(p+1))((p+1)\times n)=n\times n\). ∎
01.5Residuals and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) Slides p.8–9
What Slides p.8–9
Residual \(e_i=y_i-\hat y_i\): the part of \(y_i\) that the estimated model does not fit.
The error \(\epsilon_i=y_i-\boldsymbol{x}_i^T\boldsymbol\beta\) uses the true \(\boldsymbol\beta\). We cannot see it.
The residual \(e_i=y_i-\boldsymbol{x}_i^T\hat{\boldsymbol\beta}\) uses \(\hat{\boldsymbol\beta}\). We can calculate it. \(\boldsymbol{x}_i^T\) is row \(i\) of \(\boldsymbol{X}\).
\(\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}\): a matrix that depends only on \(\boldsymbol{X}\), times \(\boldsymbol{y}\).
\(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\): each column of \(\boldsymbol{X}\) is orthogonal to \(\boldsymbol{e}\) (\(p+1\) equations).
Column 1 is all 1s, which gives \(\sum_{i=1}^ne_i=0\).
How Added
- Calculate \(\hat{\boldsymbol{y}}\) (Section 01.4).
- For each \(i\), calculate \(e_i=y_i-\hat y_i\).
- For each column of \(\boldsymbol{X}\), multiply it by \(\boldsymbol{e}\) entry by entry, then add.
Self-check: All \(p+1\) sums are 0. If not, \(\hat{\boldsymbol\beta}\) has an error.
Example · Data set 1 Added
Residual of observation 2 How step 2
\(\hat y_2\) from Section 01.4, step 2.
All residuals How step 2
Column 1 times \(\boldsymbol{e}\) How step 3
Column 2 times \(\boldsymbol{e}\) How step 3
Column 3 times \(\boldsymbol{e}\) How step 3
All sums 0: \(\hat{\boldsymbol\beta}\) is correct. ▲
Blue and orange segments have equal total length, \(1.05+0.65=1.7=0.15+1+0.55\). This is \(\sum e_i=0\).
Why · \(\boldsymbol{e}=(\boldsymbol{I}-\boldsymbol{H})\boldsymbol{y}\) and \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) Slides p.8–9
Substitute \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\).
Substitute \(\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{H}\boldsymbol{y}\).
∎
Line 3 above.
Substitute \(\boldsymbol{H}\).
\(\boldsymbol{X}^T\boldsymbol{X}(\boldsymbol{X}^T\boldsymbol{X})^{-1}=\boldsymbol{I}\).
∎
01.6Practice Added
Q1. In simple linear regression, \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), write \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) as two scalar equations.
Answer
Rows of \(\boldsymbol{X}^T\) are columns of \(\boldsymbol{X}\).
▲
Q2. Data: \(n=3\), \(x=(0,1,2)\), \(y=(1,3,2)\). Model: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\). Calculate \(\hat{\boldsymbol\beta}\), \(\hat{\boldsymbol{y}}\), \(\boldsymbol{e}\). Check \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\).
Answer
\(\boldsymbol{X}\) and \(\boldsymbol{X}^T\boldsymbol{X}\) Section 01.2, How step 1
\(\boldsymbol{X}^T\boldsymbol{y}\) Section 01.2, How step 2
Inverse Section 01.2, How step 3
Rows \((a,b),(c,d)\): inverse \(\frac{1}{ad-bc}\) times rows \((d,-b),(-c,a)\).
\(\hat{\boldsymbol\beta}\) Section 01.2, How step 4
\(\hat{\boldsymbol{y}}\) Section 01.4, How step 2
\(\boldsymbol{e}\) Section 01.5, How step 2
Check Section 01.5, How step 3
▲
Q3. Use \(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) to show \(\boldsymbol{X}^T\hat{\boldsymbol{y}}=\boldsymbol{X}^T\boldsymbol{y}\). Check it for data set 1.
Answer
\(\hat{\boldsymbol{y}}=\boldsymbol{y}-\boldsymbol{e}\).
\(\boldsymbol{X}^T\boldsymbol{e}=\boldsymbol{0}\) (Slides p.9).
Check with \(\hat{\boldsymbol{y}}=(3.15,\,3.95,\,7,\,8.55,\,12.35)^T\)
Section 01.2, step 5. ∎
Unit 02Estimating \(\sigma^2\) and the joint distribution of \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\)
Step 1: Maximum likelihood estimator \(\hat\sigma^2_{MLE}\) (Slides p.10–12).
Step 2: The course estimator \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\) (Slides p.13).
Step 3: Stack \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\); show the stack is MVN (Slides p.14).
Step 4: Mean of the stack (Slides p.15–16).
Step 5: Variance matrix of the stack, block by block (Slides p.17–22).
Step 6: Distributions of \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\), and their independence (Slides p.23–25).
Error variance \(\sigma^2\): the unknown variance of each \(\epsilon_i\).
Units 02, 04, and 06 use data set 2. Added
\(n=4\), \(p=1\), \(x=(0,1,2,3)\), \(\boldsymbol y=(1,3,2,5)^T\).
\[\boldsymbol X=\left[\begin{array}{cc}1&0\\1&1\\1&2\\1&3\end{array}\right],\qquad \boldsymbol y=\left[\begin{array}{c}1\\3\\2\\5\end{array}\right]\]2×2 inverse: swap diagonal, negate others, divide by determinant.
Entries of \(\boldsymbol H\): \(h_{ij}=\left[\begin{array}{cc}1&x_i\end{array}\right]\boldsymbol V\left[\begin{array}{c}1\\x_j\end{array}\right]=0.7-0.3(x_i+x_j)+0.2x_ix_j\). For example, \(h_{12}=0.7-0.3(0+1)+0.2(0)(1)=0.4\).
\[\boldsymbol H=\left[\begin{array}{cccc}0.7&0.4&0.1&-0.2\\0.4&0.3&0.2&0.1\\0.1&0.2&0.3&0.4\\-0.2&0.1&0.4&0.7\end{array}\right],\qquad \boldsymbol I-\boldsymbol H=\left[\begin{array}{cccc}0.3&-0.4&-0.1&0.2\\-0.4&0.7&-0.2&-0.1\\-0.1&-0.2&0.7&-0.4\\0.2&-0.1&-0.4&0.3\end{array}\right]\]Self-check: row 1 of \((\boldsymbol I-\boldsymbol H)\boldsymbol y\) is \(0.3(1)-0.4(3)-0.1(2)+0.2(5)=0.3-1.2-0.2+1.0=-0.1=e_1\).
02.1The MLE of \(\sigma^2\) Slides p.10–12
p.10: section page "Estimating \(\sigma^2\)". \(\mathrm{Var}(\hat{\boldsymbol\beta})\) contains \(\sigma^2\). Estimate \(\sigma^2\) before you calculate standard errors and confidence intervals.
What Slides p.11–12
Line 1 is the log density of \(\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\). Line 2 sets the partial derivative for \(\sigma^2\) to 0. The MLE of \(\boldsymbol\beta\) is \(\hat{\boldsymbol\beta}\).
\((\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})^T(\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta})=\boldsymbol e^T\boldsymbol e=\sum_{i=1}^n e_i^2\), the sum of squared residuals. \(\hat\sigma^2_{MLE}\) is their mean.
How Added
- Calculate \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e=\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta}\).
- Square each residual.
- Add the squares: \(\boldsymbol e^T\boldsymbol e\).
- Divide by \(n\).
Self-check: After step 1, the residuals add to 0 (model with intercept).
Self-check: In \(\ell\), the last term equals \(-n/2\).
Example · Data set 2 Added
Residuals How step 1
Self-check: \(-0.1+0.8-1.3+0.6=0\).
Squares How step 2
Sum How step 3
Divide by \(n\) How step 4
▲
Self-check (\(n=4\)): the last term is \(-\frac{2.70}{2(0.675)}=-2=-n/2\). Then \(\ell=-2(1.837877)-2(-0.393043)-2=-3.675754+0.786085-2=-4.8897\).
Figure 02.1 · \(\ell(\hat{\boldsymbol\beta},\sigma^2|\boldsymbol y)\) for data set 2. Orange: maximum at \(\hat\sigma^2_{MLE}=0.675\). Green: \(\hat\sigma^2=1.35\) (Section 02.2).
Why Slides p.11–12
Let \(Q=(\boldsymbol y-\boldsymbol X\boldsymbol\beta)^T(\boldsymbol y-\boldsymbol X\boldsymbol\beta)\). \(Q\) does not contain \(\sigma^2\).
MVN density (Lecture 5).
\(|\sigma^2\boldsymbol I|=(\sigma^2)^n\), \((\sigma^2\boldsymbol I)^{-1}=\frac{1}{\sigma^2}\boldsymbol I\).
Take the log. Line 1 of p.11.
With \(t=\sigma^2\): \(\frac{d}{dt}\log t=\frac1t\), \(\frac{d}{dt}\frac1t=-\frac{1}{t^2}\).
Derivative is 0 at the maximum.
Line 3 of p.12.
MLE of \(\boldsymbol\beta\) is \(\hat{\boldsymbol\beta}\) (Lecture 6). ∎
02.2The unbiased estimate of \(\sigma^2\) Slides p.13
What Slides p.13
The MLE of \(\sigma^2\) is \(\frac{1}{n}\boldsymbol e^T\boldsymbol e\), where \(\boldsymbol e=\boldsymbol y-\boldsymbol X\hat{\boldsymbol\beta}\).
Instead we usually use \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\). Why?
Both estimators use \(\boldsymbol e^T\boldsymbol e\). Only the denominator differs.
\(p+1\) is the number of entries in \(\boldsymbol\beta\). Residual degrees of freedom \(n-(p+1)\): observations minus estimated coefficients. The slide title says \(\hat\sigma^2\) is unbiased.
p.13 only asks "Why?". Unit 04 answers it with \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\). Slides p.14–25 prepare that proof.
How Added
- Calculate \(\boldsymbol e^T\boldsymbol e\) (Section 02.1, How steps 1–3).
- Count the coefficients \(p+1\), the columns of \(\boldsymbol X\).
- Calculate \(n-(p+1)\).
- Divide \(\boldsymbol e^T\boldsymbol e\) by \(n-(p+1)\) to get \(\hat\sigma^2\).
- Take the square root to get \(\hat\sigma\).
Self-check: \(\hat\sigma^2>\hat\sigma^2_{MLE}\), and \(\hat\sigma^2/\hat\sigma^2_{MLE}=n/(n-(p+1))\).
Example · Data set 2 Added
Sum of squared residuals How step 1
Section 02.1, Example, step 3.
Count the coefficients How step 2
Columns: intercept and \(x\).
Degrees of freedom How step 3
Divide How step 4
Square root How step 5
▲
Self-check: \(1.35/0.675=2=4/2\).
02.3Stack \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) into one vector Slides p.14
Joint distribution: the distribution of several random quantities together.
What Slides p.14
Recall \(\hat{\boldsymbol\beta}=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\) and \(\boldsymbol e=(\boldsymbol I-\boldsymbol H)\boldsymbol y\).
Let's consider the vector \(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\), which is a linear combination of \(\boldsymbol y\):
\[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]=\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ \boldsymbol I-\boldsymbol H\end{array}\right]\boldsymbol y\]And since \(\boldsymbol y\sim N(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\), then \(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\) is MVN-distributed.
The stack is \((p+1+n)\times1\). The matrix is \((p+1+n)\times n\). Linear combination of \(\boldsymbol y\): a sum of \(y_1,\dots,y_n\), each times a constant.
How Added
- Write the vector as \(\boldsymbol M\boldsymbol y+\boldsymbol d\), with \(\boldsymbol M\) and \(\boldsymbol d\) free of \(\boldsymbol y\).
- Confirm \(\boldsymbol y\sim MVN(\boldsymbol X\boldsymbol\beta,\sigma^2\boldsymbol I)\).
- Use MVN linearity (Lecture 5): \(\boldsymbol M\boldsymbol y+\boldsymbol d\) is MVN.
Self-check: \(\boldsymbol M\) has \(n\) columns.
Self-check: The first \(p+1\) entries of \(\boldsymbol M\boldsymbol y\) are \(\hat{\boldsymbol\beta}\); the rest are \(\boldsymbol e\).
Here \(\boldsymbol M=\left[\begin{array}{c}(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\\ \boldsymbol I-\boldsymbol H\end{array}\right]\), \(\boldsymbol d=\boldsymbol 0\). The slides use \(\boldsymbol C\) for a block in Section 02.5, so this page uses \(\boldsymbol M\).
Example · Data set 2 Added
Top block How step 1
Row 1: \(0.7-0.3x_j\). Row 2: \(-0.3+0.2x_j\).
Stack the two blocks How step 1
\(6\times4\): \(p+1+n=6\), \(n=4\). Self-check 1 passes.
Row 1 times \(\boldsymbol y\)
Row 2 times \(\boldsymbol y\)
Rows 3 to 6 times \(\boldsymbol y\)
Row 3: \(0.3-1.2-0.2+1.0=-0.1\). Row 4: \(-0.4+2.1-0.4-0.5=0.8\). Row 5: \(-0.1-0.6+1.4-2.0=-1.3\). Row 6: \(0.2-0.3-0.8+1.5=0.6\).
\(\hat\beta_0=1.1\), \(\hat\beta_1=1.1\), then \(\boldsymbol e\). Self-check 2 passes. ▲
A mean vector and a variance matrix fully specify an MVN.
02.4The mean of the stacked vector Slides p.15–16
\(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\) is known. Only \(E[\boldsymbol e]\) is new.
What Slides p.15–16
Recall \(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\). Now,
\[\begin{array}{rl}E[\boldsymbol e]&=E[(\boldsymbol I-\boldsymbol H)\boldsymbol y]\\&=(\boldsymbol I-\boldsymbol H)E[\boldsymbol y]\\&=(\boldsymbol I-\boldsymbol H)\boldsymbol X\boldsymbol\beta\\&=\boldsymbol X\boldsymbol\beta-\boldsymbol H\boldsymbol X\boldsymbol\beta\\&=\boldsymbol X\boldsymbol\beta-\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X\boldsymbol\beta\\&=\boldsymbol X\boldsymbol\beta-\boldsymbol X\boldsymbol\beta=\boldsymbol 0\end{array}\]So \(E\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]=\left[\begin{array}{c}\boldsymbol\beta\\ \boldsymbol 0\end{array}\right]\).
On average, a residual is 0. The key step is \(\boldsymbol H\boldsymbol X=\boldsymbol X\), which gives \((\boldsymbol I-\boldsymbol H)\boldsymbol X=\boldsymbol 0\).
How Added
- Move the constant matrix out: \(E[\boldsymbol M\boldsymbol y]=\boldsymbol M E[\boldsymbol y]\).
- Substitute \(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\) to get \(\boldsymbol M\boldsymbol X\boldsymbol\beta\).
- Calculate \(\boldsymbol M\boldsymbol X\). For \(\boldsymbol M=\boldsymbol I-\boldsymbol H\), the result is \(\boldsymbol 0\).
Self-check: \(\boldsymbol M\boldsymbol X\) uses only \(\boldsymbol X\). For \(\boldsymbol M=\boldsymbol I-\boldsymbol H\), each entry is 0.
Example · Data set 2 Added
One sample cannot give \(E[\boldsymbol e]\). How step 3 uses only \(\boldsymbol X\), so you can check it.
Column 1 of \(\boldsymbol X\) is \((1,1,1,1)^T\) How step 3
Column 2 of \(\boldsymbol X\) is \((0,1,2,3)^T\) How step 3
True for each \(\boldsymbol\beta\). ▲
Why Slides p.15–16
Substitute \(\boldsymbol e\).
\(\boldsymbol I-\boldsymbol H\) is constant (Lecture 5).
\(E[\boldsymbol y]=\boldsymbol X\boldsymbol\beta\).
Substitute \(\boldsymbol H\).
\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).
With \(E[\hat{\boldsymbol\beta}]=\boldsymbol\beta\): \(E\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]=\left[\begin{array}{c}\boldsymbol\beta\\ \boldsymbol 0\end{array}\right]\). ∎
02.5The variance matrix of the stacked vector Slides p.17–18
What Slides p.17–18
How Added
- Use linearity (Lecture 5): \(\mathrm{Var}(\boldsymbol M\boldsymbol y)=\boldsymbol M\,\mathrm{Var}(\boldsymbol y)\,\boldsymbol M^T\).
- Substitute \(\mathrm{Var}(\boldsymbol y)=\sigma^2\boldsymbol I\) to get \(\sigma^2\boldsymbol M\boldsymbol M^T\).
- Split \(\boldsymbol M\) into top and bottom blocks. \(\boldsymbol M^T\) then has left and right blocks.
- Multiply each row block by each column block to get \(\boldsymbol A\), \(\boldsymbol B\), \(\boldsymbol C\), \(\boldsymbol D\).
Self-check: \(\boldsymbol A\) is \((p+1)\times(p+1)\). \(\boldsymbol D\) is \(n\times n\).
Example · Data set 2 Added
One entry of each block of \(\boldsymbol M\boldsymbol M^T\), with \(\boldsymbol M\) from Section 02.3.
Block \(\boldsymbol A\), entry (2,2): row 2 of \(\boldsymbol M\) times row 2 How step 4
Equals \(V_{11}=0.2\).
Block \(\boldsymbol B\), entry (1,1): row 1 of \(\boldsymbol M\) times row 3 How step 4
Block \(\boldsymbol C\), entry (1,2): row 3 of \(\boldsymbol M\) times row 2 How step 4
Block \(\boldsymbol D\), entry (1,1): row 3 of \(\boldsymbol M\) times row 3 How step 4
Equals entry (1,1) of \(\boldsymbol I-\boldsymbol H\). ▲
Why Slides p.17–18
Let \(\boldsymbol L=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\).
Stacked form of p.14.
Linearity (Lecture 5).
\(\mathrm{Var}(\boldsymbol y)=\sigma^2\boldsymbol I\); transpose turns a stack into a pair.
\(\boldsymbol L^T=\boldsymbol X((\boldsymbol X^T\boldsymbol X)^{-1})^T=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\).
Block multiplication.
p.18 writes \(\boldsymbol B=\boldsymbol L(\boldsymbol I-\boldsymbol H)\); \((\boldsymbol I-\boldsymbol H)^T=\boldsymbol I-\boldsymbol H\) (Section 02.6). ∎
02.6Simplify the blocks \(\boldsymbol A\), \(\boldsymbol B\), \(\boldsymbol D\), and the Aside Slides p.19–22
What Slides p.19–22
Aside:
\[\begin{array}{rl}\boldsymbol H\boldsymbol H^T&=[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T][\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T]^T\\&=[\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T][\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T]\\&=\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T=\boldsymbol H\end{array}\] \[\begin{array}{rl}\boldsymbol D&=(\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)^T\\&=(\boldsymbol I\boldsymbol I^T-\boldsymbol I\boldsymbol H^T-\boldsymbol H\boldsymbol I^T+\boldsymbol H\boldsymbol H^T)\\&=(\boldsymbol I-2\boldsymbol H+\boldsymbol H)\\&=(\boldsymbol I-\boldsymbol H)\end{array}\]How Added
- Write each \(\boldsymbol H\) as \(\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\).
- Find each adjacent pair \((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X\) or \(\boldsymbol X^T\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\).
- Replace each pair with \(\boldsymbol I\).
- Use \(\boldsymbol H^T=\boldsymbol H\) and \(\boldsymbol I^T=\boldsymbol I\).
- Replace \(\boldsymbol H\boldsymbol H\) or \(\boldsymbol H\boldsymbol H^T\) with \(\boldsymbol H\) (Aside).
Self-check: One entry of the product is the same before and after.
Example · Data set 2 Added
Block \(\boldsymbol A\): first find \(\boldsymbol L\boldsymbol X\)
\(\boldsymbol A=\boldsymbol I\boldsymbol V=\boldsymbol V=\left[\begin{array}{cc}0.7&-0.3\\-0.3&0.2\end{array}\right]\).
Block \(\boldsymbol B=\boldsymbol L(\boldsymbol I-\boldsymbol H)\): row 2 of \(\boldsymbol L\) times column 2 of \(\boldsymbol I-\boldsymbol H\)
The other 7 entries are also 0.
Aside: entry (1,2) of \(\boldsymbol H\boldsymbol H^T\), row 1 of \(\boldsymbol H\) times row 2
Equals \(h_{12}=0.4\).
Block \(\boldsymbol D\): entry (1,2), row 1 of \(\boldsymbol I-\boldsymbol H\) times row 2
Equals entry (1,2) of \(\boldsymbol I-\boldsymbol H\). ▲
Why Slides p.19–22
Block \(\boldsymbol A\):
\((\boldsymbol X^T\boldsymbol X)^{-1}(\boldsymbol X^T\boldsymbol X)=\boldsymbol I\).
∎
Block \(\boldsymbol B\):
Substitute \(\boldsymbol H\).
\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).
∎
Aside \(\boldsymbol H\boldsymbol H^T=\boldsymbol H\). Rule: \((\boldsymbol P\boldsymbol Q\boldsymbol S)^T=\boldsymbol S^T\boldsymbol Q^T\boldsymbol P^T\).
Transpose of a product: reverse the order.
Inverse of symmetric \(\boldsymbol X^T\boldsymbol X\) is symmetric. Shows \(\boldsymbol H^T=\boldsymbol H\).
Associative.
∎
Block \(\boldsymbol D\):
Transpose each term.
\(\boldsymbol I^T=\boldsymbol I\), \(\boldsymbol H^T=\boldsymbol H\).
Aside: \(\boldsymbol H\boldsymbol H^T=\boldsymbol H\).
∎
02.7Implications Slides p.23–25
What Slides p.23–25
- \(\hat{\boldsymbol\beta}\sim N(\boldsymbol\beta,\sigma^2(\boldsymbol X^T\boldsymbol X)^{-1})\)
- \(\boldsymbol e\sim N(\boldsymbol 0,\sigma^2(\boldsymbol I-\boldsymbol H))\)
- \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) are independent
Figure 02.2 · Variance matrix of \(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\) for data set 2, without \(\sigma^2\). Top-left: \((\boldsymbol X^T\boldsymbol X)^{-1}\). Bottom-right: \(\boldsymbol I-\boldsymbol H\). Grey blocks: zeros.
Each result uses one MVN property (Lecture 5).
How Added
- Calculate \(\hat\sigma^2=\boldsymbol e^T\boldsymbol e/(n-(p+1))\) (Section 02.2).
- Calculate \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\).
- Estimate \(\mathrm{Var}(\hat\beta_j)\) as \(\hat\sigma^2V_{jj}\).
- Take its square root: the standard error of \(\hat\beta_j\).
- Estimate \(\mathrm{Var}(e_i)\) as \(\hat\sigma^2(1-h_{ii})\).
Self-check: Each variance is positive. Each \(1-h_{ii}\) is between 0 and 1.
Example · Data set 2 Added
Estimate \(\sigma^2\) How step 1
Section 02.2.
Read \(V_{11}\) How step 2
Second diagonal entry (0-indexing).
Variance of \(\hat\beta_1\) How step 3
Standard error How step 4
Variances of \(e_1\) and \(e_2\) How step 5
▲
The end points \(x=0\) and \(x=3\) have larger \(h_{ii}\), so their residuals have smaller variance.
Why Slides p.23–25
Section 02.3 (p.14).
Section 02.4 (p.16).
Section 02.5 (p.18).
Section 02.6 (p.19–22).
Mean and variance specify an MVN.
Marginal distribution property.
Zero covariance gives independence. ∎
\(\hat\sigma^2\) uses only \(\boldsymbol e\), so \(\hat\sigma^2\) is also independent of \(\hat{\boldsymbol\beta}\). Unit 03 uses this for the t statistic.
02.8Practice Added
Q1. A regression has \(n=12\), \(p=2\), and \(\boldsymbol e^T\boldsymbol e=45\). Calculate \(\hat\sigma^2_{MLE}\), \(\hat\sigma^2\), and \(\hat\sigma\).
Answer
\(\hat\sigma^2_{MLE}=\frac{1}{n}\boldsymbol e^T\boldsymbol e=\frac{45}{12}=3.75\).
\(n-(p+1)=12-(2+1)=9\), so \(\hat\sigma^2=\frac{45}{9}=5\).
\(\hat\sigma=\sqrt{5}=2.2361\).
Self-check: \(5/3.75=1.3333=12/9\).
Q2. Show that the lower-left block \(\boldsymbol C=(\boldsymbol I-\boldsymbol H)\boldsymbol X(\boldsymbol X^T\boldsymbol X)^{-1}\) of \(\mathrm{Var}\left(\left[\begin{array}{c}\hat{\boldsymbol\beta}\\ \boldsymbol e\end{array}\right]\right)/\sigma^2\) is \(\boldsymbol 0\).
Answer
Substitute \(\boldsymbol H\).
\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).
∎ Or: \(\boldsymbol C=\boldsymbol B^T=\boldsymbol 0^T=\boldsymbol 0\).
Q3. Data set 2: \(h_{33}=0.3\), \(\hat\sigma^2=1.35\). Give the distribution of \(e_3\). Estimate \(\mathrm{Var}(e_3)\). Is \(e_3\) independent of \(\hat\beta_1\)?
Answer
Entry 3 of \(\boldsymbol e\sim N(\boldsymbol 0,\sigma^2(\boldsymbol I-\boldsymbol H))\): \(e_3\sim N(0,\ \sigma^2(1-h_{33}))=N(0,\ 0.7\,\sigma^2)\).
\(\widehat{\mathrm{Var}}(e_3)=1.35\times0.7=0.945\).
Yes. Their covariance is an entry of \(\sigma^2\boldsymbol B=\boldsymbol 0\), and the joint distribution is MVN.
Unit 03The t statistic in MLR and property (3): independence
Step 1: Symbols and distributions (Added).
Step 2: The SLR t statistic: three facts and one combination rule (Slides p.26).
Step 3: The same plan in MLR, and the properties still to prove (Slides p.27).
Step 4: Prove property (3) (Slides p.28). Unit 04 proves property (2).
03.1Symbols and distributions Added
03.2Recall: the t statistic in SLR Slides p.26
What Slides p.26
And we used the following: (we showed 1 but only claimed 2 & 3)
1. \(Z=\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}\sim N(0,1)\)
2. \(V=\dfrac{1}{\sigma^2}\sum_{i=1}^ne_i^2\sim\chi^2_{(n-2)}\)
3. \(Z\) and \(V\) are independent
Then if (1) \(Z\sim N(0,1)\), (2) \(V\sim\chi^2_\nu\) and (3) \(V\) independent of \(Z\): \(\Longrightarrow\dfrac{Z}{\sqrt{V/\nu}}\sim t_\nu\) \[\Longrightarrow\frac{Z}{\sqrt{V/\nu}}=\frac{\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\sqrt{\dfrac{1}{\sigma^2}\sum_{i=1}^ne_i^2/(n-2)}}\sim t_{n-2}\] \[\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{n-2}\]
With \(\hat\sigma\) for the unknown \(\sigma\), the denominator is random and the distribution is not \(N(0,1)\).
Lecture 3 proved fact 1 only. This lecture proves facts 2 and 3 for MLR. SLR is the case \(p=1\).
The random \(\hat\sigma\) adds variation, so large values of the statistic occur more often.
How Added
- Write \(Z\): (estimator − parameter) / standard deviation with the true \(\sigma\). Confirm \(Z\sim N(0,1)\).
- Write \(V\): residual sum of squares / \(\sigma^2\). Confirm \(V\sim\chi^2_\nu\). Record \(\nu\).
- Confirm that \(Z\) and \(V\) are independent.
- Put them into \(Z/\sqrt{V/\nu}\). Cancel \(\sigma\). Write \(\sum e_i^2/\nu\) as \(\hat\sigma^2\). The result is \(t_\nu\).
Self-check: (1) The result has \(\hat\sigma\), not \(\sigma\). (2) The t degrees of freedom equal those of \(V\). (3) The denominator is \(\mathrm{SE}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}\).
Example · Brain and head data Numbers from Lecture 3 · p.45
\(y\): brain weight. \(x\): head size. Lecture 3 p.45 prints:
Find \(n\) from the degrees of freedom How step 2
Find \(\sqrt{S_{xx}}\) from the SE How step 4
Approximate: the printed values are rounded.
Write the statistic and its distribution How step 4
Substitute \(\beta_1=0\)
Self-check: Lecture 3 p.44–45 prints 19.957. The difference comes from rounded inputs.
Why Slides p.26
substitute \(Z\) and \(V\), \(\nu=n-2\)
\(\hat\sigma=\sqrt{\sum_{i=1}^ne_i^2/(n-2)}\)
combination rule; needs facts 1, 2, 3 ∎
03.3The matching result in MLR Slides p.27
What Slides p.27
but \(\sigma\) is unknown in practice, so we need the distribution where we've replaced \(\sigma\) by \(\hat\sigma\).
We have shown:
1. \(\dfrac{\hat\beta_1-\beta_1}{\sqrt{\sigma^2V_{jj}}}\sim N(0,1)\)
If we can further show:
2. \(\dfrac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\)
3. \(\hat{\boldsymbol\beta}\) and \(\hat{\boldsymbol e}\) are independent
Then by the same argument we have: \[\frac{\dfrac{\hat\beta_j-\beta_j}{\sqrt{\sigma^2V_{jj}}}}{\sqrt{\left(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\right)/(n-(p+1))}}\sim t_{n-(p+1)}\] \[\frac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\sim t_{n-(p+1)}\] Let's (finally) prove properties (2) & (3)!
Item 1 prints \(\hat\beta_1-\beta_1\); it means \(\hat\beta_j-\beta_j\). In item 3, \(\hat{\boldsymbol e}\) is \(\boldsymbol e\).
For \(p=1\): \(n-(p+1)=n-2\) and \(\sqrt{\sigma^2V_{11}}=\sigma/\sqrt{S_{xx}}\), the SLR result.
How Added
- Write \(\boldsymbol X\). Calculate \(\boldsymbol X^T\boldsymbol X\) and \(\boldsymbol X^T\boldsymbol y\).
- Calculate \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\). Find \(V_{jj}\).
- Calculate \(\hat{\boldsymbol\beta}=\boldsymbol V\boldsymbol X^T\boldsymbol y\).
- Calculate \(\hat{\boldsymbol y}=\boldsymbol X\hat{\boldsymbol\beta}\) and \(\boldsymbol e=\boldsymbol y-\hat{\boldsymbol y}\).
- Calculate \(\boldsymbol e^T\boldsymbol e\) and \(\hat\sigma^2=\boldsymbol e^T\boldsymbol e/(n-(p+1))\).
- Calculate \(\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\) and \(\dfrac{\hat\beta_j-\beta_j}{\mathrm{SE}(\hat\beta_j)}\).
- Use the \(t_{n-(p+1)}\) distribution.
Self-check: (1) \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). (2) \(V_{jj}>0\). (3) Degrees of freedom: \(n\) minus the columns of \(\boldsymbol X\).
Example · Data set 3 Added
\(n=6\), \(p=2\). Part 1 does the How steps. Part 2 assumes true values and shows \(Z/\sqrt{V/\nu}\) equals the t statistic.
Part 1: the t statistic for \(\hat\beta_1\).
Write \(\boldsymbol X\), \(\boldsymbol X^T\boldsymbol X\), and \(\boldsymbol X^T\boldsymbol y\) How step 1
Diagonal: \(1\cdot6=6\); \(1+1+0+0+1+1=4\); \(6\times1=6\). Off-diagonal: \(\sum x_{i1}=0\), \(\sum x_{i2}=0\), \(\sum x_{i1}x_{i2}=1-1+0+0-1+1=0\).
Calculate \(\boldsymbol V\) and \(V_{11}\) How step 2
Diagonal matrix: invert each diagonal entry.
Calculate \(\hat{\boldsymbol\beta}\) How step 3
Fitted values and residuals How step 4
Calculate \(\boldsymbol e^T\boldsymbol e\) and \(\hat\sigma^2\) How step 5
SE and the statistic for a stated \(\beta_1=1\) How step 6
State the distribution How step 7
Self-check: \(\boldsymbol X^T\boldsymbol e=(-1-1+2+2-1-1,\ 1+1+0+0-1-1,\ 1-1-2+2+1-1)^T=(0,0,0)^T\).
Part 2: the same statistic as \(Z/\sqrt{V/\nu}\). Assume the true values \(\beta_1=1\) and \(\sigma=1\).
\(Z\) with the true \(\sigma=1\) Slides p.27, property (1)
\(V\) with the true \(\sigma=1\) Slides p.27, property (2)
Combine with \(\nu=3\) Slides p.27, combination rule
Self-check: 1, as in Part 1, step 6, but Part 1 needs no true \(\sigma\).
\(Z=2\) uses the true \(\sigma=1\). The t statistic uses \(\hat\sigma=2\).
Why Slides p.27
Slides p.5
one element of an MVN vector; variance is the diagonal element
standardize
substitute \(V\), \(\nu=n-(p+1)\)
\(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\) (Slides p.13)
combination rule; needs properties (1), (2), (3) ∎
03.4Property (3): \(\hat{\boldsymbol\beta}\) and \(\boldsymbol e\) are independent Slides p.28
What Slides p.28
How Added
- Confirm the two vectors together are MVN. Here both are linear functions of \(\boldsymbol y\) (Slides p.14).
- Write the variance matrix. Find the block \(\mathrm{Cov}(\hat{\boldsymbol\beta},\boldsymbol e)\).
- Confirm this block is 0. Here it is \(\sigma^2\boldsymbol B\), \(\boldsymbol B=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T(\boldsymbol I-\boldsymbol H)=\boldsymbol 0\) (Slides p.21).
- Write that the two vectors are independent (Lecture 5 · p.40).
- Apply this to each function of \(\boldsymbol e\) only, for example \(\boldsymbol e^T\boldsymbol e\) and \(\hat\sigma^2\).
Self-check: (1) Zero covariance gives independence only for jointly normal vectors. (2) \(\boldsymbol B\boldsymbol y=(\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol e\), so \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\).
Example · Data set 3 Added
How step 3 with \(\boldsymbol e=(-1,-1,2,2,-1,-1)^T\) from Section 03.3.
Calculate \(\boldsymbol X^T\boldsymbol e\) How step 3
Calculate \(\boldsymbol B\boldsymbol y\) Slides p.21
Functions of \(\boldsymbol e\) only How step 5
Independent of \(\hat{\boldsymbol\beta}=(5,2,1)^T\) by property (3).
One data set cannot show independence; it is a property of the distribution. The proof shows it in general.
Why Slides p.28
Slides p.14
linear function of MVN \(\boldsymbol y\) (Lecture 5 · p.40, Linearity)
\(\boldsymbol A=(\boldsymbol X^T\boldsymbol X)^{-1}\), \(\boldsymbol B=\boldsymbol 0\), \(\boldsymbol D=\boldsymbol I-\boldsymbol H\) (Slides p.21–22)
top-right block
jointly MVN, zero covariance (Lecture 5 · p.40, Independence)
function of \(\boldsymbol e\) only
\(Z\) is a function of \(\hat{\boldsymbol\beta}\) only ∎
03.5Practice Added
Q1. MLR with \(n=6\), \(p=2\). One data set gives \(\hat\beta_1=2\), \(V_{11}=0.25\), \(\boldsymbol e^T\boldsymbol e=6\). The true value is \(\beta_1=1.5\).
(a) Calculate \(\hat\sigma^2\). (b) Calculate \(\dfrac{\hat\beta_1-\beta_1}{\sqrt{\hat\sigma^2V_{11}}}\). (c) State its distribution.
Answer
Source: the Section 03.3 design with \(\boldsymbol y=(1,4,3,9,5,8)^T\): \(\hat{\boldsymbol\beta}=(5,2,2)^T\), \(\boldsymbol e=(0,-1,0,2,0,-1)^T\).
Q2. List the three properties that give \(\dfrac{\hat\beta_j-\beta_j}{\sqrt{\hat\sigma^2V_{jj}}}\sim t_{n-(p+1)}\). Why does \(\sigma\) not appear? Why does the \(t\) result need all three?
Answer
\(Z\) has \(\sigma\) in its denominator; \(\sqrt{V/\nu}\) has \(1/\sigma\). They cancel, and \(\sqrt{\boldsymbol e^T\boldsymbol e/(n-(p+1))}=\hat\sigma\) stays. The t definition needs an \(N(0,1)\) numerator, a \(\chi^2_\nu\) denominator, and their independence.
Q3. Use the joint distribution \[\left[\begin{array}{c}\hat{\boldsymbol\beta}\\\boldsymbol e\end{array}\right]\sim N\left(\left[\begin{array}{c}\boldsymbol\beta\\\boldsymbol 0\end{array}\right],\ \sigma^2\left[\begin{array}{cc}(\boldsymbol X^T\boldsymbol X)^{-1}&\boldsymbol 0\\\boldsymbol 0&(\boldsymbol I-\boldsymbol H)\end{array}\right]\right),\] to show that \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\) is independent of \(\hat{\boldsymbol\beta}\).
Answer
top-right block
jointly MVN, zero covariance
function of \(\boldsymbol e\) only ∎
Unit 04Property (2): \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\)
Step 1: Goal and problem: the \(e_i\) are not independent (Slides p.29–30).
Step 2: Eigen decomposition turns \(\boldsymbol e\) into \(\tilde{\boldsymbol e}\) with independent entries (Slides p.31–36).
Step 3: \(\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=\boldsymbol e^T\boldsymbol e\) (Slides p.37–38).
Step 4: Each \(\lambda_i\) is 0 or 1 (Slides p.39–40).
Step 5: \(\sum\lambda_i=n-(p+1)\) (Slides p.41–42).
Step 6: Combine (Slides p.43).
Results used: \(\boldsymbol e\sim N(\boldsymbol 0,\sigma^2(\boldsymbol I-\boldsymbol H))\) (Section 02.7). \(\boldsymbol I-\boldsymbol H\) is symmetric and \((\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)=\boldsymbol I-\boldsymbol H\) (Section 02.6).
From Unit 02: \(\boldsymbol x=(0,1,2,3)^T\), \(\boldsymbol y=(1,3,2,5)^T\), \(n=4\), \(p=1\).
\(\boldsymbol X^T\boldsymbol X=\left[\begin{array}{cc}4&6\\6&14\end{array}\right]\), \((\boldsymbol X^T\boldsymbol X)^{-1}=\frac{1}{20}\left[\begin{array}{cc}14&-6\\-6&4\end{array}\right]\).
\(\hat{\boldsymbol\beta}=(1.1,\ 1.1)^T\), \(\boldsymbol e=(-0.1,\ 0.8,\ -1.3,\ 0.6)^T\), \(\boldsymbol e^T\boldsymbol e=2.70\).
\(h_{11}=0.7\), \(h_{22}=0.3\), \(h_{33}=0.3\), \(h_{44}=0.7\).
\[\boldsymbol I-\boldsymbol H=\left[\begin{array}{cccc}0.3&-0.4&-0.1&0.2\\-0.4&0.7&-0.2&-0.1\\-0.1&-0.2&0.7&-0.4\\0.2&-0.1&-0.4&0.3\end{array}\right]\]04.1The plan of the proof Slides p.29–30
What Slides p.29–30
The chi-squared definition needs terms that are \(N(0,1)\) and independent. Each \(e_i/\sigma\) is normal, but the terms are not independent. The proof finds independent variables with the same sum of squares.
How Added
- Write the sum as \(\sum_i Z_i^2\). Identify each \(Z_i\).
- Confirm each \(Z_i\) is normal with mean 0 and variance 1.
- Confirm the \(Z_i\) are independent. For an MVN vector, all off-diagonal variance entries are 0.
- Count the terms: \(\nu\).
Self-check: If an off-diagonal entry is not 0, change the variables first.
Example Added
Data set 2: test \(e_1/\sigma\) and \(e_2/\sigma\) against steps 2 and 3.
Entry \((1,1)\) of \(\mathrm{Var}[\boldsymbol e]\).
Entry \((1,2)\) of \(\mathrm{Var}[\boldsymbol e]\).
Steps 2 and 3 fail. \(\sum_i(e_i/\sigma)^2\) is not directly a chi-squared sum. ▲
Why Slides p.29–30
All \(e_i\) come from one \(\boldsymbol y\) through \(\boldsymbol e=(\boldsymbol I-\boldsymbol H)\boldsymbol y\). The off-diagonal covariances \(\sigma^2(\boldsymbol I-\boldsymbol H)_{ij}\) are usually not 0.
Also \(\boldsymbol X^T\boldsymbol e=\boldsymbol 0\). In data set 2, \(-0.1+0.8-1.3+0.6=0\): three residuals fix the fourth. ∎
04.2Eigen decomposition and the distribution of \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\) Slides p.31–36
What Slides p.31–36
\(\mathrm{Var}[\boldsymbol e]\) is not diagonal. \(\mathrm{Var}[\tilde{\boldsymbol e}]=\sigma^2\boldsymbol D\) is diagonal, so the \(\tilde e_i\) are independent.
How Added
- Calculate \(\boldsymbol I-\boldsymbol H\).
- Find \(n\) mutually orthogonal eigenvectors. Read each \(\lambda\) from \((\boldsymbol I-\boldsymbol H)\boldsymbol v=\lambda\boldsymbol v\).
- Divide each eigenvector by its length \(\sqrt{\boldsymbol v^T\boldsymbol v}\).
- Put them in the rows of \(\boldsymbol\Gamma\). Put their eigenvalues on the diagonal of \(\boldsymbol D\), same order.
- Calculate \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\).
Self-check: (1) \(\boldsymbol\Gamma\boldsymbol\Gamma^T=\boldsymbol I\). (2) Each row satisfies \((\boldsymbol I-\boldsymbol H)\boldsymbol v=\lambda\boldsymbol v\).
Example Added
Step 2 How step 2: try \(\boldsymbol v_1=(1,-1,-1,1)^T\), \(\boldsymbol v_2=(1,-3,3,-1)^T\), \(\boldsymbol v_3=(1,1,1,1)^T\), \(\boldsymbol v_4=(-3,-1,1,3)^T\).
\(\lambda_1=\lambda_2=1\), \(\lambda_3=\lambda_4=0\).
Step 3 How step 3: \(\boldsymbol v_1^T\boldsymbol v_1=1+1+1+1=4\), \(\boldsymbol v_2^T\boldsymbol v_2=1+9+9+1=20\), \(\boldsymbol v_3^T\boldsymbol v_3=4\), \(\boldsymbol v_4^T\boldsymbol v_4=9+1+1+9=20\). Divide by \(\sqrt4=2\) or \(\sqrt{20}\).
Step 4 How step 4:
Step 5 How step 5: rows of \(\boldsymbol\Gamma\) times \(\boldsymbol e=(-0.1,0.8,-1.3,0.6)^T\).
Self-check (1): each row has length 1; distinct rows have inner product 0, for example rows 1 and 3: \(\frac14(1-1-1+1)=0\). \(\tilde e_3=\tilde e_4=0\) because \(\lambda_i=0\) gives variance 0. ▲
Why Slides p.31–36
Definition of \(\tilde{\boldsymbol e}\).
\(\boldsymbol\Gamma\) is constant.
Slides p.25.
Lecture 5 · p.40, Linearity.
Slides p.25.
Eigen decomposition.
\(\boldsymbol\Gamma\boldsymbol\Gamma^T=\boldsymbol I\).
The slide asks why \(\tilde{\boldsymbol e}\) is normal. It is a linear transformation of the MVN \(\boldsymbol e\), so it is MVN (Lecture 5 · p.40, Linearity). \(\sigma^2\boldsymbol D\) has zero off-diagonal entries, so the \(\tilde e_i\) are independent (Lecture 5 · p.40, Independence), with variances \(\sigma^2\lambda_i\). ∎
04.3Why \(\tilde{\boldsymbol e}\) is useful: \(\tilde{\boldsymbol e}^T\tilde{\boldsymbol e}=\boldsymbol e^T\boldsymbol e\) Slides p.37–38
What Slides p.37–38
\(\boldsymbol e^T\boldsymbol e\) is the squared length of \(\boldsymbol e\), and \(\boldsymbol\Gamma\) keeps lengths. \(Z_i=\tilde e_i/\sigma\) has variance \(\lambda_i\).
How Added
- Get \(\boldsymbol e\) and \(\tilde{\boldsymbol e}=\boldsymbol\Gamma\boldsymbol e\) (Section 04.2).
- Add the squares of the entries of \(\boldsymbol e\).
- Add the squares of the entries of \(\tilde{\boldsymbol e}\).
- Compare the two sums.
Self-check: Both sums equal \(\boldsymbol e^T\boldsymbol e\).
Example Added
How step 2.
How step 3.
\(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\frac{2.70}{\sigma^2}=Z_1^2+Z_2^2+Z_3^2+Z_4^2\), with observed \(Z_3=Z_4=0\). ▲
Why Slides p.37–38
Definition of \(\tilde{\boldsymbol e}\).
\((\boldsymbol A\boldsymbol B)^T=\boldsymbol B^T\boldsymbol A^T\).
\(\boldsymbol\Gamma^T\boldsymbol\Gamma=\boldsymbol I\).
\(Z_i=\tilde e_i/\sigma\overset{indep}{\sim}N(0,\lambda_i)\). ∎
Two conditions remain: each variance is 1, and there are \(n-(p+1)\) terms.
04.4Each \(\lambda_i\) is 0 or 1 Slides p.39–40
What Slides p.39–40
• Recall: need a sum of independent standard normally distributed r.v.'s (var=1)
• And: we need to sum up exactly \(n-(p+1)\) of these r.v.s
So: to show \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{(n-(p+1))}\), we will show that \(n-(p+1)\) of the \(\lambda_i\)'s are equal to 1, and the remainder are all 0.
Let's first show that all of the \(\lambda_i\)'s are equal to either 0 or 1:
• We know that \((\boldsymbol I-\boldsymbol H)(\boldsymbol I-\boldsymbol H)=(\boldsymbol I-\boldsymbol H)\); i.e. \((\boldsymbol I-\boldsymbol H)\) is idempotent.
• The eigenvalues (elements \(\lambda_i\) of \(\boldsymbol D\)) of an idempotent matrix are equal to 0 or 1.
• Why? By definition of eigenvalue \(\lambda_i\) and corresponding eigenvector \(\boldsymbol v_i\), we know: \(\lambda_i\boldsymbol v_i=(\boldsymbol I-\boldsymbol H)\boldsymbol v_i\)
Thus all \(\lambda_i\) are either 0 or 1.
\(\lambda_i=1\) gives \(Z_i\sim N(0,1)\). \(\lambda_i=0\) gives \(Z_i=0\), which adds 0 to the sum.
How Added
- Calculate \(\boldsymbol A\boldsymbol A\): entry \((i,j)\) is row \(i\) times column \(j\).
- Compare \(\boldsymbol A\boldsymbol A\) with \(\boldsymbol A\), entry by entry.
- If all are equal, \(\boldsymbol A\) is idempotent, and each eigenvalue is 0 or 1.
Self-check: For each known eigenvector, \(\boldsymbol A\boldsymbol v\) is \(\boldsymbol v\) or \(\boldsymbol 0\).
Example Added
Data set 2. Row 1 of \(\boldsymbol I-\boldsymbol H\) is \((0.3,-0.4,-0.1,0.2)\).
Entry \((1,1)\) of \(\boldsymbol I-\boldsymbol H\) is 0.3.
Entry \((1,2)\) of \(\boldsymbol I-\boldsymbol H\) is \(-0.4\).
The other 14 entries also agree. This matches \(\lambda=1,1,0,0\) from Section 04.2. ▲
Satisfaction data of the slides: \(n=46\) patients, \(p=3\) (Lecture 6 · p.9). Sections 04.4–04.5 give 42 eigenvalues of 1 and 4 of 0.
Why Slides p.39–40
Definition of eigenvalue.
Idempotent.
Definition again.
\(\lambda_i\) is a number.
Definition again.
\(\boldsymbol v_i\neq\boldsymbol 0\), so \(\lambda_i(1-\lambda_i)=0\). ∎
04.5\(\sum_i\lambda_i=n-(p+1)\): count the 1s with the trace Slides p.41–42
Each \(\lambda_i\) is 0 or 1, so their sum counts the 1s.
What Slides p.41–42
The second rule turns \(\mathrm{tr}(\boldsymbol H)\) into the trace of the \((p+1)\times(p+1)\) identity, which is \(p+1\).
How Added
- Find \(n\).
- Find \(p\). Do not count the intercept.
- Calculate \(n-(p+1)\): the number of \(\lambda_i=1\).
- Calculate \(p+1\): the number of \(\lambda_i=0\).
Self-check: The counts add to \(n\). By hand, \(\mathrm{tr}(\boldsymbol H)=p+1\) and \(\mathrm{tr}(\boldsymbol I-\boldsymbol H)=n-(p+1)\).
Example Added
Data set 2 (\(n=4\), \(p=1\)).
How steps 1–3.
Trace \(1+1=2=p+1\).
Two eigenvalues are 1, two are 0, as in Section 04.2. Satisfaction data: \(46-(3+1)=42\) ones and \(3+1=4\) zeros (Figure 04.1). ▲
Why Slides p.41–42
Diagonal of \(\boldsymbol D\) is \(\lambda_1,\dots,\lambda_n\).
\(\boldsymbol\Gamma\boldsymbol\Gamma^T=\boldsymbol I\).
\(\mathrm{tr}(\boldsymbol A\boldsymbol B)=\mathrm{tr}(\boldsymbol B\boldsymbol A)\), \(\boldsymbol A=\boldsymbol D\boldsymbol\Gamma\), \(\boldsymbol B=\boldsymbol\Gamma^T\).
Eigen decomposition.
\(\mathrm{tr}(\boldsymbol I_n)=n\); definition of \(\boldsymbol H\).
\(\mathrm{tr}(\boldsymbol A\boldsymbol B)=\mathrm{tr}(\boldsymbol B\boldsymbol A)\), \(\boldsymbol A=\boldsymbol X\).
\((p+1)\times(p+1)\) identity.
∎
04.6Recap: \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\) Slides p.43
What Slides p.43
1. \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e=\sum_{i=1}^nZ_i^2\) where \(Z_i\overset{indep}{\sim}N(0,\lambda_i)\).
• i.e., a sum of \(n\) squared independent normally distributed r.v.'s
2. \(var(Z_i)=\lambda_i\in\{0,1\}\)
• hence are either standard normally distributed or degenerate (0)
3. \(\sum_{i=1}^n\lambda_i=n-(p+1)\)
• hence exactly \(n-(p+1)\) of the \(Z_i\) are \(N(0,1)\) r.v.s
\(\Longrightarrow\ \frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\) is a sum of \(n-(p+1)\) squared standard normal r.v.s \[\Longrightarrow\ \frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\]
Drop the terms that are always 0. The rest fits the chi-squared definition.
With \(\hat\sigma^2=\frac{1}{n-(p+1)}\boldsymbol e^T\boldsymbol e\), the result is also \(\frac{(n-(p+1))\hat\sigma^2}{\sigma^2}\sim\chi^2_{n-(p+1)}\).
How Added
- Find \(n\) and \(p\).
- Calculate \(n-(p+1)\).
- Write \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{n-(p+1)}\).
- For \(\hat\sigma^2\), divide \(\boldsymbol e^T\boldsymbol e\) by \(n-(p+1)\).
Self-check: The chi-squared degrees of freedom equal the divisor in \(\hat\sigma^2\).
Example Added
How steps 1–2.
How step 3.
How step 4.
Section 04.3: \(\frac{2.70}{\sigma^2}=Z_1^2+Z_2^2+0^2+0^2\), two standard normal terms, as \(\chi^2_2\) requires. ▲
Why Slides p.43
Item 1: Section 04.3. Item 2: Section 04.4. Item 3: Section 04.5. ∎
04.7Practice Added
Q1. A regression has \(n=20\), \(p=3\). How many eigenvalues of \(\boldsymbol I-\boldsymbol H\) are 1, and how many are 0? Give the distribution of \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\).
Answer
Slides p.42.
Each \(\lambda_i\) is 0 or 1 (Slides p.40): 16 ones, 4 zeros. Slides p.43: \(\frac{1}{\sigma^2}\boldsymbol e^T\boldsymbol e\sim\chi^2_{16}\).
Q2. Show that \(\boldsymbol H\) is idempotent and that its eigenvalues are 0 or 1. For the satisfaction data (\(n=46\), \(p=3\)), how many eigenvalues of \(\boldsymbol H\) are 1?
Answer
\((\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol X=\boldsymbol I\).
The proof of Slides p.40 with \(\boldsymbol H\): \(\lambda\boldsymbol v=\boldsymbol H\boldsymbol v=\boldsymbol H\boldsymbol H\boldsymbol v=\lambda^2\boldsymbol v\), so \(\lambda\in\{0,1\}\). The count of 1s is \(\mathrm{tr}(\boldsymbol H)=p+1=3+1=4\) (Slides p.42, line 3).
Q3. Use \(\boldsymbol x=(0,1,2,3)^T\), \(\boldsymbol y=(2,1,4,5)^T\), and \(y_i=\beta_0+\beta_1x_i+\epsilon_i\). Calculate \(\boldsymbol e\), \(\boldsymbol e^T\boldsymbol e\), \(\hat\sigma^2\). Give the distribution of \(\frac{2\hat\sigma^2}{\sigma^2}\).
Answer
\(\boldsymbol x\) is the same as in data set 2, so \((\boldsymbol X^T\boldsymbol X)^{-1}\) is the same.
Check with \(\boldsymbol\Gamma\) (Section 04.2): \(\tilde e_1=\frac12(0.8+1.4-0.4+0.2)=1.0\), \(\tilde e_2=\frac{1}{\sqrt{20}}(0.8+4.2+1.2-0.2)=\frac{6}{\sqrt{20}}\). \(1.0^2+\frac{36}{20}=1.0+1.8=2.80\).
\(2\hat\sigma^2=\boldsymbol e^T\boldsymbol e\), so \(\frac{2\hat\sigma^2}{\sigma^2}\sim\chi^2_{2}\) (Slides p.43).
Unit 05t tests and confidence intervals for each \(\beta_j\)
Step 1: The t distribution of the standardized \(\hat\beta_j\) (Slides p.44).
Step 2: \(\hat{\boldsymbol\beta}\) from the normal equations (Slides p.45).
Step 3: \(\hat\sigma^2\) and each standard error (Slides p.46).
Step 4: Rules for the test and the confidence interval (Slides p.47–49).
Step 5: p-value for \(H_0:\beta_j=0\), each \(j\) (Slides p.50).
Step 6: 95% confidence interval for each \(\beta_j\) (Slides p.51).
\(V_{00}\) belongs to the intercept \(\beta_0\).
05.1The t distribution of the standardized \(\hat\beta_j\) Slides p.44
What Slides p.44
Line 1: subtract the mean and divide by the standard deviation (square root of the variance) to get \(N(0,1)\).
Line 2: with \(\hat\sigma^2\) for the unknown \(\sigma^2\), the distribution is \(t_{n-(p+1)}\). d.o.f.: degrees of freedom.
How Added
- Find \(\hat\beta_j\), entry \(j\) of \(\hat{\boldsymbol\beta}\) (counted from 0).
- Find \(V_{jj}\), diagonal entry \(j\) of \((\boldsymbol X^T\boldsymbol X)^{-1}\).
- Calculate \(\hat\sigma^2=\boldsymbol e^T\boldsymbol e/(n-(p+1))\).
- Divide \(\hat\beta_j-\beta_j\) by \(\sqrt{\hat\sigma^2V_{jj}}\).
- Use \(n-(p+1)\) degrees of freedom.
Self-check: \(V_{jj}>0\) and \(\hat\sigma^2\ge0\), so the denominator is positive.
Self-check: The degrees of freedom are a positive whole number.
Example Added
How step 5. Sections 05.2–05.3 calculate the other inputs.
The satisfaction data How step 5
\(t_{42}\) for each \(j=0,1,2,3\).
Data set 4 How step 5
\(t_{3}\) for each \(j=0,1\). ▲
Why Slides p.26–43
Section 03.3 gives the plan and the chain. Section 03.4 proves property (3). Unit 04 proves property (2).
05.2Calculate \(\hat{\boldsymbol\beta}\) from the normal equations Slides p.45
What Slides p.45
Slide p.45 shows only code; this is its statistical content. Solve the \(p+1\) normal equations directly.
The slide says "NEVER" for the other method: invert \(\boldsymbol X^T\boldsymbol X\) first, then multiply by \(\boldsymbol X^T\boldsymbol y\).
How Added
- Make \(\boldsymbol X\): a column of 1s, then the covariates in model order.
- Calculate \(\boldsymbol X^T\boldsymbol X\). Entry \((j,k)\): sum of column \(j\) times column \(k\), term by term.
- Calculate \(\boldsymbol X^T\boldsymbol y\). Entry \(j\): sum of column \(j\) times \(\boldsymbol y\), term by term.
- Write \((\boldsymbol X^T\boldsymbol X)\hat{\boldsymbol\beta}=\boldsymbol X^T\boldsymbol y\).
- Solve by elimination, one unknown at a time.
Self-check: \(\boldsymbol X^T\boldsymbol X\) is symmetric, \((p+1)\times(p+1)\), top-left entry \(n\).
Self-check: \(\hat{\boldsymbol\beta}\) satisfies each normal equation.
Example Added
Data set 4.
\(\boldsymbol X^T\boldsymbol X\) How step 2
\(\boldsymbol X^T\boldsymbol y\) How step 3
Normal equations How step 4
Elimination How step 5
First equation times 3.
Second equation minus line 4.
Substitute \(\hat\beta_1=0.8\).
Self-check: \(15(0.6)+55(0.8)=9+44=53\). ▲
The slides do not print \(\hat{\boldsymbol\beta}\) for the satisfaction data. Section 05.6 gets it from the interval endpoints on p.51.
Why Added
Form \(\boldsymbol A\boldsymbol x=\boldsymbol b\). ∎
05.3Calculate \(\hat\sigma^2\) and the standard errors Slides p.46
What Slides p.46
Slide p.46 shows only code; this is its statistical content.
\(p\) is the number of columns of \(\boldsymbol X\) minus 1 (the intercept column).
How Added
- Find \(n\) (rows of \(\boldsymbol X\)) and \(p\) (columns of \(\boldsymbol X\) minus 1).
- Calculate \(\hat y_i=\hat\beta_0+\hat\beta_1x_i+\dots\) and \(e_i=y_i-\hat y_i\).
- Calculate \(\hat\sigma^2=\sum_i e_i^2/(n-(p+1))\).
- Calculate \(\boldsymbol V=(\boldsymbol X^T\boldsymbol X)^{-1}\). Read \(V_{00},\dots,V_{pp}\).
- Calculate \(\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\).
Self-check: The residuals add to 0 (model with intercept).
Self-check: \(\boldsymbol V\boldsymbol X^T\boldsymbol X=\boldsymbol I\).
Example Added
Data set 4, \(\hat\beta_0=0.6\), \(\hat\beta_1=0.8\).
\(n\) and \(p\) How step 1
Fitted values How step 2
Residuals How step 2
Self-check: \(-0.4+0.8-1.0+1.2-0.6=0\).
\(\boldsymbol e^T\boldsymbol e\) How step 3
\(\hat\sigma^2\) How step 3
Determinant How step 4
\(\boldsymbol V\) How step 4
2×2 inverse: swap diagonal, negate others, divide by determinant.
\(\mathrm{SE}(\hat\beta_0)\) How step 5
\(\mathrm{SE}(\hat\beta_1)\) How step 5
Check \(\boldsymbol V\): \(-0.3\cdot15+0.1\cdot55=-4.5+5.5=1\). ▲
Why Added
p.44 (Section 05.1).
\(\hat\sigma^2\) for the unknown \(\sigma^2\).
Square root of a variance. ∎
05.4Inference rules for \(\beta_j\): test and confidence interval Slides p.47–49
What Slides p.47–49
This is identical to the result for SLR except with updated
- Standard errors: \(SE(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\)
- And the d.o.f. of the t distribution is \(n-(p+1)\)
- Note: In SLR, we had \(n-2\), equivalent to \(p=1\) covariate
- Compare \(t^{obs}\) (i.e., realization of \(\frac{\hat\beta_j-\theta_0}{SE(\hat\beta_j)}\)) to null distribution (\(t_{n-(p+1)}\))
- \(\hat\beta_j\pm t_{1-\alpha/2,\,n-(p+1)}\,SE(\hat\beta_j)\)
Slides p.47–49 print \(SE(\hat\beta_j)=\hat\sigma^2V_{jj}\) (no square root) and \(t_{n-p+1}\) (no brackets). This page writes \(\sqrt{\hat\sigma^2V_{jj}}\) and \(t_{n-(p+1)}\), as on p.44 and p.46.
How Added
- Write \(H_0:\beta_j=\theta_0\). Usually \(\theta_0=0\).
- Calculate \(t^{obs}=(\hat\beta_j-\theta_0)/\mathrm{SE}(\hat\beta_j)\).
- Use the null distribution \(T\sim t_{n-(p+1)}\).
- Calculate the p-value \(=2P(T>|t^{obs}|)\).
- Reject \(H_0\) if the p-value is less than \(\alpha\), or equivalently if \(|t^{obs}|>t_{1-\alpha/2,\,n-(p+1)}\).
Self-check: \(t^{obs}\) has the sign of \(\hat\beta_j-\theta_0\). The p-value is between 0 and 1.
\(\theta_0=0\): covariate \(j\) has no linear relation with the mean response when the other covariates stay the same.
- Find \(t_{1-\alpha/2,\,n-(p+1)}\) in a t table.
- Calculate the half-width \(t_{1-\alpha/2,\,n-(p+1)}\times\mathrm{SE}(\hat\beta_j)\).
- Write \(\hat\beta_j\pm\) half-width.
Self-check: The midpoint is \(\hat\beta_j\). At the same \(\alpha\), the CI excludes \(\theta_0\) exactly when the test rejects \(H_0:\beta_j=\theta_0\).
Example Added
Data set 4: test \(H_0:\beta_1=0\) and find the 95% CI for \(\beta_1\) (\(\alpha=0.05\)).
Write \(H_0\) Test step 1
\(t^{obs}\) Test step 2
Inputs from Sections 05.2 and 05.3.
Null distribution Test step 3
p-value Test step 4
Decision Test step 5
Also: \(|t^{obs}|=2.309<t_{0.975,3}=3.182446\).
Quantile CI step 1
Half-width CI step 2
Interval CI step 3
Contains 0, as line 5 requires. ▲
Why Added
Let \(q=t_{1-\alpha/2,\,n-(p+1)}\). The t distribution is symmetric about 0, so \(P(-q\le T\le q)=1-\alpha\).
Section 05.1.
Multiply by \(-1\); inequalities reverse. ∎
For the test, \(H_0\) gives \(\beta_j=\theta_0\), so \(T=(\hat\beta_j-\theta_0)/\mathrm{SE}(\hat\beta_j)\sim t_{n-(p+1)}\).
05.5The p-value for each coefficient Slides p.50
What Slides p.50
Numerators of the test statistics: \(\hat\beta_j\). Denominators of the test statistics: \(\mathrm{SE}(\hat\beta_j)\).
p-value: want \(P(|T|>|t^{obs}|)\).
Slide p.50 shows only code; this is its statistical content. With \(\theta_0=0\), \(t^{obs}=\hat\beta_j/\mathrm{SE}(\hat\beta_j)\) and \(T\sim t_{n-(p+1)}\).
The slide gives two equal forms: \(2P(T<-|t^{obs}|)\) (left tail) and \(2P(T>|t^{obs}|)\) (right tail).
How Added
- Write \(\hat\beta_0,\dots,\hat\beta_p\) (Section 05.2).
- Write \(\mathrm{SE}(\hat\beta_0),\dots,\mathrm{SE}(\hat\beta_p)\) (Section 05.3).
- Calculate \(t^{obs}_j=\hat\beta_j/\mathrm{SE}(\hat\beta_j)\).
- Calculate each p-value \(=2P(T<-|t^{obs}_j|)\), \(T\sim t_{n-(p+1)}\).
- Compare each p-value with \(\alpha\).
Self-check: Larger \(|t^{obs}_j|\) gives a smaller p-value.
Self-check: Each p-value is between 0 and 1.
Example Added
Data set 4 (\(T\sim t_3\), \(\alpha=0.05\)).
Numerators and denominators How steps 1–2
\(t^{obs}_0\) How step 3
\(t^{obs}_1\) How step 3
p-value for \(\beta_0\) How step 4
p-value for \(\beta_1\) How step 4
Decisions How step 5
Self-check: larger \(|t^{obs}_1|\), smaller p-value. ▲
The area under a density curve over an interval is the probability of that interval.
Figure 05.1 · Test of \(H_0:\beta_1=0\), data set 4. Blue: \(t_3\) density. Purple dashed: \(\pm|t^{obs}_1|=\pm2.309\). Each orange tail: area 0.052044 (including the part past \(\pm4\)). Sum: p-value 0.104088.
Why Added
The tails do not overlap.
Symmetry about 0.
First form on p.50; symmetry gives the second. ∎
05.6The 95% confidence interval for each coefficient Slides p.51
What Slides p.51
Slide p.51 calculates \(\hat\beta_j\pm t_{0.975,\,n-(p+1)}\,\mathrm{SE}(\hat\beta_j)\) by two methods with equal results. It prints, for the satisfaction data (\(n=46\), \(p=3\)):
\((L,\ U)\): lower and upper endpoints. Each uses \(\alpha=0.05\) and \(n-(p+1)=42\).
How Added
- Midpoint \((L+U)/2\) is \(\hat\beta_j\).
- Calculate the half-width \((U-L)/2\).
- Find \(t_{0.975,\,n-(p+1)}\).
- Divide the half-width by the quantile to get \(\mathrm{SE}(\hat\beta_j)\).
- If the CI excludes 0, reject \(H_0:\beta_j=0\) at \(\alpha=0.05\).
Self-check: \(L<\hat\beta_j<U\) and \(\mathrm{SE}(\hat\beta_j)>0\).
Example Added
Age coefficient \(\beta_1\).
Midpoint How step 1
Half-width How step 2
Quantile How step 3
Standard error How step 4
Test result How step 5
Check: \(t^{obs}=-1.141612/0.214799=-5.315\), p-value \(3.81\times10^{-6}<0.05\). ▲
Other coefficients, with \(t_{0.975,42}=2.0180817\):
Figure 05.2 · 95% CI for Age \(\beta_1\) (blue). Green: \(\hat\beta_1=-1.1416\). Half-width 0.4335. Purple dashed: 0. The interval lies left of 0: reject \(H_0:\beta_1=0\).
Why Added
Let \(q=t_{0.975,\,n-(p+1)}\).
CI rule, Section 05.4.
∎
05.7Practice Added
Satisfaction data: \(n=46\), \(p=3\), \(T\sim t_{42}\). From Section 05.6: \(\hat\beta_1=-1.141612\), \(\mathrm{SE}(\hat\beta_1)=0.214799\), \(\hat\beta_3=-13.470163\), \(\mathrm{SE}(\hat\beta_3)=7.099661\).
Q1. Test \(H_0:\beta_3=0\) (Stress) against \(H_1:\beta_3\neq0\) at \(\alpha=0.05\). Give \(t^{obs}\), the null distribution, the p-value, and the decision. Does it agree with the 95% CI on Slides p.51?
Answer Q1
\(t^{obs}\) Section 05.4 · Test step 2
Null distribution Section 05.4 · Test step 3
p-value Section 05.4 · Test step 4
Decision Section 05.4 · Test step 5
The 95% CI \((-27.797859,\ 0.8575324)\) contains 0. They agree. ▲
Q2. Calculate the 90% CI for \(\beta_3\) (Stress) with \(t_{0.95,42}=1.681952\). At \(\alpha=0.10\), do you reject \(H_0:\beta_3=0\)?
Answer Q2
Quantile Section 05.4 · CI step 1
Half-width Section 05.4 · CI step 2
Interval Section 05.4 · CI step 3
Excludes 0: reject at \(\alpha=0.10\). Q1 p-value \(0.0647<0.10\) agrees. ▲
Q3. Test \(H_0:\beta_1=-1\) (Age) against \(H_1:\beta_1\neq-1\) at \(\alpha=0.05\).
Answer Q3
\(t^{obs}\) with \(\theta_0=-1\) Section 05.4 · Test step 2
p-value with \(T\sim t_{42}\) Section 05.4 · Test step 4
Decision Section 05.4 · Test step 5
The 95% CI \((-1.575093,\ -0.7081303)\) contains \(-1\). They agree. ▲
Unit 06Practice questions Q1–Q3
p.52: section title "Practice". p.53 (Q1): estimates and SEs of \(2\beta_3\) and \(\beta_2+\beta_3\). p.54 (Q2): \(\hat{\boldsymbol\beta}\) and \(\mathrm{SE}(\hat\beta_1)\) from part of \((\boldsymbol{X}'\boldsymbol{X})^{-1}\). p.55 (Q3): three proofs with \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\).
Slides p.54–55 write the transpose with a prime: \(\boldsymbol{X}'=\boldsymbol{X}^T\). Q2 and Q3 use the prime.
06.1Q1: a linear combination of coefficients and its SE Slides p.52–53
What Slides p.52–53
(b) Compute a corresponding SE
(c) Interpret your estimate of \(2\beta_3\)
(d) Repeat (a)–(c) for \(\beta_2+\beta_3\)
\(E[\boldsymbol{a}^T\boldsymbol{y}+b]=\boldsymbol{a}^TE[\boldsymbol{y}]+b\)
\(\mathrm{Var}(\boldsymbol{a}^T\boldsymbol{y})=\boldsymbol{a}^T\mathrm{Var}(\boldsymbol{y})\boldsymbol{a}\)
With \(\boldsymbol{V}=(\boldsymbol{X}^T\boldsymbol{X})^{-1}\): \(\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})=\hat\sigma^2\boldsymbol{V}\). \(\mathrm{SE}(\hat\beta_j)\) is the square root of its diagonal entry \(j\).
How Added
- Write \(\boldsymbol{a}\). Write the target as \(a_0\beta_0+\cdots+a_p\beta_p\), with 0 for absent coefficients.
- Estimate. Calculate \(\boldsymbol{a}^T\hat{\boldsymbol\beta}\).
- Estimate the variance. Calculate \(\boldsymbol{a}^T\widehat{\mathrm{Var}}(\hat{\boldsymbol\beta})\boldsymbol{a}=\hat\sigma^2\,\boldsymbol{a}^T\boldsymbol{V}\boldsymbol{a}\).
- Take the square root. \(\mathrm{SE}(\boldsymbol{a}^T\hat{\boldsymbol\beta})=\sqrt{\hat\sigma^2\,\boldsymbol{a}^T\boldsymbol{V}\boldsymbol{a}}\).
- Interpret. Lecture 6 · p.14: "To interpret a parameter, isolate it." Write two conditional means whose difference is \(\boldsymbol{a}^T\boldsymbol\beta\).
- State the result. In one sentence, say which covariates change, by how much, and which stay fixed.
Self-check: \(\boldsymbol{a}\) has 4 entries: intercept, Age, Severity, Stress.
Self-check: One nonzero entry \(c\) at position \(j\) gives SE \(=|c|\,\mathrm{SE}(\hat\beta_j)\).
Self-check: \(\boldsymbol{a}^T\boldsymbol{V}\boldsymbol{a}\ge0\).
Example · The satisfaction data Added
Values from Section 05.6 (Slides p.51 intervals, \(t_{0.975,\,42}=2.0180817\)), 7 decimals. Slides p.51
Part (d) needs \(\widehat{\mathrm{Cov}}(\hat\beta_2,\hat\beta_3)\), which the slides do not print. These entries of \(\hat\sigma^2\boldsymbol{V}\) come from the course file satisfaction.csv (Lecture 6 · p.9), \(\hat\sigma^2=101.1628734\). Added · not on the slides
Check: \(0.4919657^2=0.2420302\), \(7.0996609^2=50.4051849\). The differences come from rounding.
(a) Write \(\boldsymbol{a}\) and estimate How steps 1–2
(b) Estimate the variance How step 3
Only \(a_3\) is nonzero.
Take the square root How step 4
(c) Interpret \(2\hat\beta_3\). How step 5.
Stress +2; Age and Severity fixed.
(d) Repeat (a)–(c) for \(\beta_2+\beta_3\). Two nonzero entries, so a covariance term enters.
Write \(\boldsymbol{a}\) and estimate How steps 1–2
Estimate the variance How step 3
Derived under Why.
Take the square root How step 4
Negative covariance lowers 50.65 to 47.06.
Severity +1, Stress +1; Age fixed.
Why Added
Lecture 5 · p.29, \(\hat{\boldsymbol\beta}\) for \(\boldsymbol{y}\).
Slide p.44.
Expand entry by entry.
\(a_2=a_3=1\); other \(a_j=0\).
Symmetry: \(V_{32}=V_{23}\).
\(\hat\sigma^2\) for \(\sigma^2\); square root. ∎
For a sum of two or more coefficients, covariance terms enter the SE.
06.2Q2: find \(\hat{\boldsymbol\beta}\) and an SE from an incomplete \((\boldsymbol{X}'\boldsymbol{X})^{-1}\) Slides p.54
What Slides p.54
(a) Suppose that \(n=50\) and \[(\boldsymbol{X}'\boldsymbol{X})^{-1}=\left[\begin{array}{ccc}1&.25&.25\\?&.5&-.25\\?&?&2\end{array}\right],\quad \boldsymbol{X}'\boldsymbol{y}=\left[\begin{array}{c}2\\1.2\\-3\end{array}\right],\] but unfortunately the values of the elements in the lower corner of \((\boldsymbol{X}'\boldsymbol{X})^{-1}\) have been lost. Can you still find \(\hat{\boldsymbol\beta}\)? If so, please calculate it. Otherwise, explain why not.
(b) In addition to the data above, suppose that \(\sum_{i=1}^n(y_i-\boldsymbol{x}_i'\hat{\boldsymbol\beta})^2=2.5\). What is the standard error of \(\hat\beta_1\)?
How Added
- Complete \(\boldsymbol{V}=(\boldsymbol{X}'\boldsymbol{X})^{-1}\). Copy each upper entry to its mirror below the diagonal.
- Calculate \(\hat{\boldsymbol\beta}=(\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y}\). Entry \(j\): row \(j\) of \(\boldsymbol{V}\) times \(\boldsymbol{X}'\boldsymbol{y}\), entry by entry, then add.
- Find \(p\) and the degrees of freedom. \(\boldsymbol{V}\) is \((p+1)\times(p+1)\). Calculate \(\hat\sigma^2=\boldsymbol{e}'\boldsymbol{e}/(n-(p+1))\).
- Calculate the SE. \(\mathrm{SE}(\hat\beta_j)=\sqrt{\hat\sigma^2V_{jj}}\). \(V_{11}\) is in row 2, column 2.
Self-check: \(\boldsymbol{V}'=\boldsymbol{V}\). Each diagonal entry is positive.
Self-check: \(\hat{\boldsymbol\beta}\) has length \(p+1\), as \(\boldsymbol{X}'\boldsymbol{y}\).
Example · Q2 Added
Complete \(\boldsymbol{V}\) How step 1
(a): yes. Why proves the symmetry.
Calculate \(\hat{\boldsymbol\beta}\) How step 2
Find \(p\) and \(\hat\sigma^2\) How step 3
Calculate the SE How step 4
\(V_{11}=.5\) (row 2, column 2).
Why Added
Rules: \((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\); \((\boldsymbol{A}^{-1})'=(\boldsymbol{A}')^{-1}\) for invertible \(\boldsymbol{A}\).
First rule, \(\boldsymbol{A}=\boldsymbol{X}'\), \(\boldsymbol{B}=\boldsymbol{X}\).
\((\boldsymbol{X}')'=\boldsymbol{X}\): \(\boldsymbol{X}'\boldsymbol{X}\) is symmetric.
Second rule.
Line 2. ∎
06.3Q3: three equations about residuals Slides p.55
Tools: \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\), \(\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}\), and \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\) (slide p.9).
What Slides p.55
(a) Let \(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\) and \(\boldsymbol{e}=\boldsymbol{y}-\hat{\boldsymbol{y}}\). Show that \(\sum_{i=1}^n\hat y_ie_i=\hat{\boldsymbol{y}}'\boldsymbol{e}=0\).
(b) Show that \(\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}\).
(c) Show that for any vector \(\boldsymbol{a}=(a_1,\dots,a_p)\), \(\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=0\).
How Added
- Substitute a definition. Replace \(\hat{\boldsymbol{y}}\) with \(\boldsymbol{X}\hat{\boldsymbol\beta}\), or \(\boldsymbol{e}\) with \(\boldsymbol{y}-\hat{\boldsymbol{y}}\).
- Expand the transpose. Use \((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\) to get a term with \(\boldsymbol{X}'\boldsymbol{e}\).
- Substitute \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\).
- Simplify. Use \(c'=c\) for each \(1\times1\) number \(c\).
Self-check: \(\hat{\boldsymbol{y}}'\boldsymbol{e}\), \(\boldsymbol{e}'\boldsymbol{e}\), and \(\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}\) are all \(1\times1\).
Example · Data set 2 Added
Check \(\boldsymbol{X}'\boldsymbol{e}=\boldsymbol{0}\)
(a) \(\hat{\boldsymbol{y}}'\boldsymbol{e}\)
(b) \(\boldsymbol{e}'\boldsymbol{e}\) and \(\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}\)
(c) Take \(\boldsymbol{a}=(2,-1)'\)
\(\boldsymbol{X}'\boldsymbol{e}=(0,0)'\), so any \(\boldsymbol{a}\) gives 0.
Figure 06.1 shows (a) for the Q1 model (46 patients). Each point is \((\hat y_i,e_i)\). All \(\hat y_i>0\), so green points (\(e_i>0\)) give positive \(\hat y_ie_i\). Added
Write each proof with the How steps before you open the answer.
Why Added
Answer Q3(a): \(\hat{\boldsymbol{y}}'\boldsymbol{e}=0\)
Inner product.
\(\hat{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol\beta}\).
\((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\).
Slide p.9.
∎
Answer Q3(b): \(\boldsymbol{e}'\boldsymbol{e}=\boldsymbol{y}'\boldsymbol{y}-\hat{\boldsymbol\beta}'\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}\)
Plan: write the first \(\boldsymbol{e}\) as \(\boldsymbol{y}-\hat{\boldsymbol{y}}\), remove \(\hat{\boldsymbol{y}}'\boldsymbol{e}\) by (a), then use the normal equations \(\boldsymbol{X}'\boldsymbol{X}\hat{\boldsymbol\beta}=\boldsymbol{X}'\boldsymbol{y}\) (slide p.4).
First factor only.
Part (a).
\(\boldsymbol{e}=\boldsymbol{y}-\boldsymbol{X}\hat{\boldsymbol\beta}\).
\(\boldsymbol{y}'\boldsymbol{X}=(\boldsymbol{X}'\boldsymbol{y})'\).
Normal equations.
\((\boldsymbol{A}\boldsymbol{B})'=\boldsymbol{B}'\boldsymbol{A}'\).
\(\boldsymbol{X}'\boldsymbol{X}\) is symmetric (Section 06.2). ∎
Answer Q3(c): \(\boldsymbol{a}'\boldsymbol{X}'\boldsymbol{e}=0\)
Associative.
Slide p.9.
True for all \(\boldsymbol{a}\). Part (a) is \(\boldsymbol{a}=\hat{\boldsymbol\beta}\). ∎
06.4Practice Added
P1. For the Q1 model, estimate \(\beta_1-\beta_2\), give its SE, and interpret it. Use \(\hat\beta_1=-1.1416117\) (midpoint of the Age interval, slide p.51), \(\hat\sigma^2V_{11}=0.0461385\), and \(\hat\sigma^2V_{12}=-0.0322300\). The last two come from satisfaction.csv, not the slides.
Answer P1
Write \(\boldsymbol{a}\) and estimate Section 06.1, How steps 1–2
Estimate the variance Section 06.1, How step 3
\(a_1=1\), \(a_2=-1\): the cross term is negative.
Take the square root Section 06.1, How step 4
Interpret Section 06.1, How steps 5–6
Group 1: Age +1. Group 2: Severity +1. Stress equal. Group 1 mean satisfaction: 0.70 lower (SE = 0.59).
P2. From Q1(d), construct a 95% confidence interval for \(\beta_2+\beta_3\). Use \(t_{0.975,\,42}=2.0180817\).
Answer P2
Form of slide p.49: estimate \(\pm\ t_{1-\alpha/2,\,n-(p+1)}\times\mathrm{SE}\), with \(n-(p+1)=46-4=42\).
From Q1(d).
Excludes 0.
P3. Q2 setup. Calculate \(\mathrm{SE}(\hat\beta_2)\) and \(t^{obs}\) for \(H_0:\beta_2=0\). Test at \(\alpha=0.05\) with \(t_{0.975,\,47}=2.0117405\).
Answer P3
Calculate the SE Section 06.2, How step 4
\(V_{22}=2\) (row 3, column 3).
Test statistic Slides p.48
Compare with the null distribution \(t_{47}\)
Reject \(H_0:\beta_2=0\) at \(\alpha=0.05\).