Lecture 5: Random Vectors and Multivariate Normal

Unit 01Recap and multiple linear regression in matrix form

01.1Title and section pages Slides p.1–2

Page 1: "Lecture 5: Random Vectors & Multivariate normal".

Page 2: section title "Recap" (pages 3–5).

01.2Recap: simple linear regression model Slides p.3

What Slides p.3

Recap: Simple Linear Regression · Lecture 5 · p.3 \[y_i=\beta_0+\beta_1x_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] Or: \[y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\]

Simple linear regression is a linear model with one explanatory variable.

The response \(y_i\) is the quantity to explain in observation \(i\), \(i=1,\dots,n\). The sample size \(n\) is the number of observations.

The explanatory variable (covariate) \(x_i\) explains \(y_i\). The model treats \(x_i\) as a known, fixed number.

The intercept \(\beta_0\) and the slope \(\beta_1\) are parameters: fixed, unknown numbers of the population.

The error \(\epsilon_i\) is the random distance of \(y_i\) from the line \(\beta_0+\beta_1x_i\).

A random variable is a number whose value comes from a random result.

The expectation (mean) \(E[\cdot]\) is the theoretical average of a random variable.

The variance \(Var(\cdot)\) is the expected squared distance from the mean. It measures spread.

\(N(\mu,\sigma^2)\) is the bell-shaped normal distribution with mean \(\mu\) and variance \(\sigma^2\). The standard deviation \(\sigma\) is the square root of the variance.

\(\overset{iid}{\sim}\) means "independent and identically distributed": the errors do not affect each other and have one distribution.

\(\overset{indep}{\sim}\) means "independent". The \(y_i\) have different means, because the mean changes with \(x_i\).

The first form is "line plus error". The second form gives the distribution of each \(y_i\).

How Added

Write a data set as a simple linear regression model

Find the response \(y\).
Find the explanatory variable \(x\).
Count the observations \(n\).
Write \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\) and \(i=1,\dots,n\).

Self-check: The unknown parameters are \(\beta_0\), \(\beta_1\) and \(\sigma^2\). The data give \(x_i\), \(y_i\) and \(n\).

Example · brainhead data Added

The course file brainhead.csv gives the head size and brain weight of 236 persons. All examples in this unit use it. The first 3 rows:

Row 1: head size 4512 cm³, brain weight 1530 g.

Row 2: head size 3738 cm³, brain weight 1297 g.

Row 3: head size 4261 cm³, brain weight 1335 g.

Response How step 1

\[y_i=\text{brain weight of person } i\text{ (g)}\]

Explanatory variable How step 2

\[x_i=\text{head size of person } i\text{ (cm³)}\]

Sample size How step 3

\[n=236\]

Model How step 4

\[y_i=\beta_0+\beta_1x_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\qquad i=1,\dots,236\]

Lecture 4 · p.7 gives \(\hat\beta_0=335.43231\), \(\hat\beta_1=0.26082\) and \(\hat\sigma=72.35\). A hat \(\hat{\ }\) marks an estimate: a number calculated from the data.

The examples use more digits, calculated from brainhead.csv (Added, not on the slides): \(\hat\beta_0=335.4323\), \(\hat\beta_1=0.2608207\), \(\hat\sigma=72.35066\), \(\bar x=3637.864\), \(\bar y=1284.263\), \(\sum_{i=1}^n(x_i-\bar x)^2=30647233.66\).

Figure 01.1 · Brainhead data (\(n=236\)) and the fitted line \(\hat y=335.43+0.2608x\) (green). Orange: the first 3 observations. Each orange dashed line is a residual (vertical distance to the line).

Why · the two forms are the same Added

\[y_i=\beta_0+\beta_1x_i+\epsilon_i\]

\[E[y_i]=\beta_0+\beta_1x_i+E[\epsilon_i]\]

\(\beta_0+\beta_1x_i\) is a constant.

\[E[y_i]=\beta_0+\beta_1x_i\]

\(E[\epsilon_i]=0\).

\[Var(y_i)=Var(\epsilon_i)=\sigma^2\]

A constant does not change the variance.

\[y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\]

Normal plus constant is normal; independent \(\epsilon_i\). ∎

01.3Recap: estimating the mean response Slides p.4

What Slides p.4

Estimating the Mean Response · Lecture 5 · p.4 \[\hat\mu_0=\hat\beta_0+\hat\beta_1x_0=\bar y+\hat\beta_1(x_0-\bar x)\] \[\frac{\hat\mu_0-\mu_0}{\hat\sigma\sqrt{\dfrac1n+\dfrac{(x_0-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}}\sim t_{n-2}\] 100(1-α)% CI is: \[\hat\mu_0\pm t_{1-\alpha/2,\,n-2}\,\hat\sigma\sqrt{\frac1n+\frac{(x_0-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}\]

The mean response \(\mu_0=\beta_0+\beta_1x_0\) is the population mean of \(y\) at \(x=x_0\). It is an unknown parameter.

\(\hat\mu_0\) is the estimate of \(\mu_0\).

The sample means \(\bar x=\frac1n\sum_{i=1}^nx_i\) and \(\bar y\) are the averages of the \(x_i\) and the \(y_i\).

\(\hat\sigma=\sqrt{\sum_ie_i^2/(n-2)}\) estimates \(\sigma\). The residual \(e_i=y_i-\hat y_i\) is the observed value minus the fitted value.

\(t_{n-2}\) is the t distribution with \(n-2\) degrees of freedom. It is bell-shaped like \(N(0,1)\), with thicker tails.

\(t_{1-\alpha/2,\,n-2}\) is the \(1-\alpha/2\) quantile of \(t_{n-2}\): the point with area \(1-\alpha/2\) to its left.

A confidence interval (CI) is calculated from the data. In repeated samples, \(100(1-\alpha)\%\) of these intervals contain the true \(\mu_0\).

How Added

Calculate the \(100(1-\alpha)\%\) CI for \(\mu_0\)

Calculate \(\hat\mu_0=\hat\beta_0+\hat\beta_1x_0\).
Calculate \(\dfrac1n+\dfrac{(x_0-\bar x)^2}{\sum_i(x_i-\bar x)^2}\).
Take the square root of step 2. Multiply it by \(\hat\sigma\).
Find the quantile \(t_{1-\alpha/2,\,n-2}\).
Multiply step 4 by step 3. This is the half-width.
Write \(\hat\mu_0\pm\) the half-width.

Self-check: The midpoint equals \(\hat\mu_0\). The interval is wider when \(x_0\) is farther from \(\bar x\).

Example · brainhead data, \(x_0=4000\) cm³ Added

Find the 95% CI (\(\alpha=0.05\)) for the mean brain weight at head size 4000 cm³.

Point estimate How step 1

\[\hat\mu_0=335.4323+0.2608207\times4000\]

\[\hat\mu_0=335.4323+1043.2828=1378.7151\]

Term in the square root How step 2

\[x_0-\bar x=4000-3637.864=362.136\]

\[(x_0-\bar x)^2=362.136^2=131142.48\]

\[\frac{131142.48}{30647233.66}=0.0042791\]

\[\frac1{236}=0.0042373\]

\[0.0042373+0.0042791=0.0085164\]

Standard error How step 3

\[\sqrt{0.0085164}=0.0922843\]

\[72.35066\times0.0922843=6.67683\]

Quantile How step 4

\[t_{1-0.05/2,\,236-2}=t_{0.975,\,234}=1.970154\]

Half-width How step 5

\[1.970154\times6.67683=13.15438\]

Interval How step 6

\[1378.7151\pm13.1544=(1378.7151-13.1544,\ \ 1378.7151+13.1544)=(1365.5607,\ \ 1391.8695)\]

Self-check: \((1365.5607+1391.8695)/2=1378.7151=\hat\mu_0\).

Result: 95% CI for the mean brain weight at 4000 cm³ is (1365.56, 1391.87) g. ▲

Why · the second form is correct Added

\[\hat\mu_0=\hat\beta_0+\hat\beta_1x_0\]

\[\hat\mu_0=\bar y-\hat\beta_1\bar x+\hat\beta_1x_0\]

Put in \(\hat\beta_0=\bar y-\hat\beta_1\bar x\).

\[\hat\mu_0=\bar y+\hat\beta_1(x_0-\bar x)\]

∎

01.4Recap: predicting a new response Slides p.5

What Slides p.5

Predicting a New Response · Lecture 5 · p.5 \[\hat y_{new}=\hat\beta_0+\hat\beta_1x_{new}\] \[\frac{\hat y_{new}-y_{new}}{\hat\sigma\sqrt{1+\dfrac1n+\dfrac{(x_{new}-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}}\sim t_{n-2}\] 100(1-α)% prediction interval is: \[\hat y_{new}\pm t_{1-\alpha/2,\,n-2}\,\hat\sigma\sqrt{1+\frac1n+\frac{(x_{new}-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}\]

The new response \(y_{new}=\beta_0+\beta_1x_{new}+\epsilon_{new}\) is the response of a new, unobserved person. It is a random variable, not a parameter.

\(\hat y_{new}\) is the prediction of \(y_{new}\). It uses the same equation as \(\hat\mu_0\).

A prediction interval (PI) contains \(y_{new}\) in \(100(1-\alpha)\%\) of repeated samples.

How Added

Calculate the \(100(1-\alpha)\%\) PI for \(y_{new}\)

Calculate \(\hat y_{new}=\hat\beta_0+\hat\beta_1x_{new}\).
Calculate \(1+\dfrac1n+\dfrac{(x_{new}-\bar x)^2}{\sum_i(x_i-\bar x)^2}\).
Take the square root of step 2. Multiply it by \(\hat\sigma\).
Find the quantile \(t_{1-\alpha/2,\,n-2}\).
Multiply step 4 by step 3. This is the half-width.
Write \(\hat y_{new}\pm\) the half-width.

Self-check: At one point, the PI and the CI have the same midpoint. The PI is always wider.

Example · brainhead data, \(x_{new}=4000\) cm³ Added

Find the 95% PI for the brain weight of one new person with head size 4000 cm³.

Prediction How step 1

\[\hat y_{new}=335.4323+0.2608207\times4000=1378.7151\]

Term in the square root How step 2

\[1+0.0085164=1.0085164\]

0.0085164 from Section 01.3, step 2.

Standard error How step 3

\[\sqrt{1.0085164}=1.0042492\]

\[72.35066\times1.0042492=72.65809\]

Quantile How step 4

\[t_{0.975,\,234}=1.970154\]

Half-width How step 5

\[1.970154\times72.65809=143.14763\]

Interval How step 6

\[1378.7151\pm143.1476=(1378.7151-143.1476,\ \ 1378.7151+143.1476)=(1235.5675,\ \ 1521.8627)\]

Self-check: Midpoint 1378.7151, same as the CI. Half-width is \(143.1476/13.1544=10.88\) times the CI half-width.

Result: 95% PI for one new person at 4000 cm³ is (1235.57, 1521.86) g. ▲

Why · the square root has an extra 1 Added

The prediction error \(\hat y_{new}-y_{new}\) has two independent sources: the estimated line and the new error \(\epsilon_{new}\).

\[Var(\hat y_{new}-y_{new})=Var(\hat y_{new})+Var(y_{new})\]

\(y_{new}\) is independent of the data.

\[=\sigma^2\left(\frac1n+\frac{(x_{new}-\bar x)^2}{\sum_i(x_i-\bar x)^2}\right)+\sigma^2\]

\(Var(\hat y_{new})\) as on page 4; \(Var(y_{new})=\sigma^2\).

\[=\sigma^2\left(1+\frac1n+\frac{(x_{new}-\bar x)^2}{\sum_i(x_i-\bar x)^2}\right)\]

Put \(\hat\sigma\) for \(\sigma\): ratio is \(t_{n-2}\). ∎

01.5From simple to multiple linear regression Slides p.6–7

Slides p.6 Page 6 repeats the title page. New material starts.

What Slides p.7

Linear Regression · Lecture 5 · p.7 Simple Linear Regression: \[y_i=\beta_0+\beta_1x_i+\epsilon_i,\quad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\iff y_i\,|\,x_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\] Multiple Linear Regression: \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i,\quad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] \[\iff y_i\,|\,x_i\overset{indep}{\sim}N(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip},\ \sigma^2)\]

Multiple linear regression is a linear model with \(p\) explanatory variables. With \(p=1\), it is simple linear regression.

\(x_{ij}\) is the value of variable \(j\) in observation \(i\). Index \(i\) counts rows (observations); \(j=1,\dots,p\) counts columns (variables).

\(\beta_j\) is the coefficient of variable \(j\). With the intercept \(\beta_0\), the model has \(p+1\) coefficients.

\(y_i\,|\,x_i\) is the conditional distribution of \(y_i\) when \(x_i=(x_{i1},\dots,x_{ip})\) is fixed.

\(\iff\) means "if and only if": the two sides give the same model.

How Added

Write a data set as a multiple linear regression model

Find the response \(y\). Count the observations \(n\).
List the explanatory variables. Count them: \(p\).
Number the variables \(1,\dots,p\).
Call the value of variable \(j\) in row \(i\) \(x_{ij}\).
Write \(y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\), with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\) and \(i=1,\dots,n\).

Self-check: The right side has \(p+1\) coefficients. For \(j\ge1\), \(\beta_j\) multiplies a variable with second index \(j\).

Example · brainhead data Added

The only explanatory variable is head size: \(p=1\).

Response and sample size How step 1

\[y_i=\text{brain weight},\qquad n=236\]

Variables How steps 2–3

\[\text{variable 1}=\text{head size},\qquad p=1\]

Values How step 4

\[x_{11}=4512,\qquad x_{21}=3738,\qquad x_{31}=4261\]

Model How step 5

\[y_i=\beta_0+\beta_1x_{i1}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\qquad i=1,\dots,236\]

This is the model of Section 01.2, with \(x_i\) renamed \(x_{i1}\). ▲

Why · the two forms are the same Added

\[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\]

\[E[y_i\,|\,x_i]=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+E[\epsilon_i]\]

For fixed \(x_i\), the linear part is constant.

\[E[y_i\,|\,x_i]=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\]

\(E[\epsilon_i]=0\).

\[Var(y_i\,|\,x_i)=Var(\epsilon_i)=\sigma^2\]

\[y_i\,|\,x_i\overset{indep}{\sim}N(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip},\ \sigma^2)\]

Normal plus constant is normal; independent \(\epsilon_i\). ∎

01.6Matrix form of multiple linear regression Slides p.8

What Slides p.8

Multiple Linear Regression · Lecture 5 · p.8 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] We can write this: \[\underbrace{\left[\begin{array}{c}y_1\\y_2\\\vdots\\y_n\end{array}\right]}_{\mathbf{y}}=\underbrace{\left[\begin{array}{ccccc}1&x_{11}&x_{12}&\dots&x_{1p}\\1&x_{21}&x_{22}&\dots&x_{2p}\\\vdots&\vdots&\vdots&\vdots&\vdots\\1&x_{n1}&x_{n2}&\dots&x_{np}\end{array}\right]}_{\mathbf{X}}\underbrace{\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]}_{\boldsymbol\beta}+\underbrace{\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\vdots\\\epsilon_n\end{array}\right]}_{\boldsymbol\epsilon}\]

A matrix is a set of numbers in rows and columns. An \(r\times c\) matrix has \(r\) rows and \(c\) columns (its dimension).

A column vector is an \(r\times1\) matrix. The slides write vectors in bold lowercase (\(\mathbf{y},\boldsymbol\beta,\boldsymbol\epsilon\)) and matrices in bold uppercase (\(\mathbf{X}\)).

\(\mathbf{y}\) (\(n\times1\)) is the response vector, with components \(y_i\).

\(\mathbf{X}\) (\(n\times(p+1)\)) is the design matrix. Row \(i\) is observation \(i\). Column 1 is all 1s. Column \(j+1\) is variable \(j\).

\(\boldsymbol\beta\) (\((p+1)\times1\)) is the coefficient vector \(\beta_0,\beta_1,\dots,\beta_p\).

\(\boldsymbol\epsilon\) (\(n\times1\)) is the error vector, with components \(\epsilon_i\).

Matrix multiplication: component \(i\) of \(\mathbf{X}\boldsymbol\beta\) is the sum of the term-by-term products of row \(i\) of \(\mathbf{X}\) and \(\boldsymbol\beta\). Columns of \(\mathbf{X}\) must equal rows of \(\boldsymbol\beta\).

An \(n\times(p+1)\) matrix times a \((p+1)\times1\) vector gives an \(n\times1\) vector.

Matrix addition adds entries in the same position. The two matrices must have the same dimension.

Figure 01.2 · Block dimensions in \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\). The \(p+1\) columns of \(\mathbf{X}\) match the \(p+1\) rows of \(\boldsymbol\beta\). The product is \(n\times1\), like \(\boldsymbol\epsilon\) and \(\mathbf{y}\). Dark green: column of 1s.

How Added

Write multiple linear regression as \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\)

Write \(\mathbf{y}\): put \(y_1,\dots,y_n\) in one column, in observation order.
Write \(\mathbf{X}\): row \(i\) is \([\,1,\ x_{i1},\ \dots,\ x_{ip}\,]\), in the row order of \(\mathbf{y}\).
Write \(\boldsymbol\beta\): put \(\beta_0,\beta_1,\dots,\beta_p\) in one column.
Write \(\boldsymbol\epsilon\): put \(\epsilon_1,\dots,\epsilon_n\) in one column.

Self-check: ① \(\mathbf{X}\) is \(n\times(p+1)\), and column 1 is all 1s. ② Row \(i\) of \(\mathbf{X}\) times \(\boldsymbol\beta\) gives \(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\).

Example · first 3 rows of the brainhead data Added

Use only the first 3 observations: \(n=3\), \(p=1\).

Response vector How step 1

\[\mathbf{y}=\left[\begin{array}{c}1530\\1297\\1335\end{array}\right]\qquad(3\times1)\]

Design matrix How step 2

\[\mathbf{X}=\left[\begin{array}{cc}1&4512\\1&3738\\1&4261\end{array}\right]\qquad(3\times2=n\times(p+1))\]

Coefficient vector How step 3

\[\boldsymbol\beta=\left[\begin{array}{c}\beta_0\\\beta_1\end{array}\right]\qquad(2\times1)\]

Error vector How step 4

\[\boldsymbol\epsilon=\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\qquad(3\times1)\]

Matrix equation

\[\left[\begin{array}{c}1530\\1297\\1335\end{array}\right]=\left[\begin{array}{cc}1&4512\\1&3738\\1&4261\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\]

Multiply each row (self-check ②):

Row 1: \(1530=1\cdot\beta_0+4512\cdot\beta_1+\epsilon_1\), that is, \(y_1=\beta_0+\beta_1x_{11}+\epsilon_1\).

Row 2: \(1297=1\cdot\beta_0+3738\cdot\beta_1+\epsilon_2\), that is, \(y_2=\beta_0+\beta_1x_{21}+\epsilon_2\).

Row 3: \(1335=1\cdot\beta_0+4261\cdot\beta_1+\epsilon_3\), that is, \(y_3=\beta_0+\beta_1x_{31}+\epsilon_3\).

The 1 in column 1 puts the intercept \(\beta_0\) in each row.

\(\boldsymbol\epsilon\) is unknown, because \(\boldsymbol\beta\) is unknown. As a check, use \(\hat\beta_0=335.4323\) and \(\hat\beta_1=0.2608207\). \(\mathbf{X}\hat{\boldsymbol\beta}\) gives the fitted values; \(\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}\) gives the residuals.

Row 1

\[1\times335.4323+4512\times0.2608207=335.4323+1176.8230=1512.2553\]

\[e_1=1530-1512.2553=17.7447\]

Row 2

\[1\times335.4323+3738\times0.2608207=335.4323+974.9478=1310.3801\]

\[e_2=1297-1310.3801=-13.3801\]

Row 3

\[1\times335.4323+4261\times0.2608207=335.4323+1111.3570=1446.7893\]

\[e_3=1335-1446.7893=-111.7893\]

These residuals are the orange dashed lines in Figure 01.1. A positive residual is a point above the line.

For all 236 rows: \(\mathbf{y}\) is \(236\times1\), \(\mathbf{X}\) is \(236\times2\), \(\boldsymbol\beta\) is \(2\times1\), \(\boldsymbol\epsilon\) is \(236\times1\). ▲

Why · row \(i\) of the matrix equation is equation \(i\) Added

\[[\mathbf{y}]_i=[\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon]_i\]

\([\cdot]_i\): component \(i\). Equal vectors have equal components.

\[y_i=[\mathbf{X}\boldsymbol\beta]_i+\epsilon_i\]

Matrix addition, entry by entry.

\[y_i=1\cdot\beta_0+x_{i1}\beta_1+x_{i2}\beta_2+\cdots+x_{ip}\beta_p+\epsilon_i\]

Row \(i\) of \(\mathbf{X}\) is \([1,x_{i1},\dots,x_{ip}]\).

\[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\]

True for each \(i=1,\dots,n\). ∎

01.7Practice Added

Q1. Rows 4 to 6 of brainhead.csv have head sizes 3777, 4177, 3585 and brain weights 1282, 1590, 1300. Use only these rows and the model \(y_i=\beta_0+\beta_1x_{i1}+\epsilon_i\). Write \(\mathbf{y}\), \(\mathbf{X}\), \(\boldsymbol\beta\), \(\boldsymbol\epsilon\) and their dimensions.

Answer

Step 1: \(\mathbf{y}=\left[\begin{array}{c}1282\\1590\\1300\end{array}\right]\), \(3\times1\).

Step 2: \(\mathbf{X}=\left[\begin{array}{cc}1&3777\\1&4177\\1&3585\end{array}\right]\), \(3\times2\).

Step 3: \(\boldsymbol\beta=\left[\begin{array}{c}\beta_0\\\beta_1\end{array}\right]\), \(2\times1\).

Step 4: \(\boldsymbol\epsilon=\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\), \(3\times1\).

Self-check: Row 2 gives \(1590=\beta_0+4177\beta_1+\epsilon_2\).

Q2. A study has \(n=50\) observations and the model \(y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3}+\epsilon_i\). Give the dimensions of \(\mathbf{y}\), \(\mathbf{X}\), \(\boldsymbol\beta\), \(\boldsymbol\epsilon\). Write row 7 of \(\mathbf{X}\).

Answer

Step 1: 3 explanatory variables: \(p=3\), \(p+1=4\).

Step 2: \(\mathbf{y}\) is \(50\times1\). \(\mathbf{X}\) is \(50\times4\). \(\boldsymbol\beta\) is \(4\times1\). \(\boldsymbol\epsilon\) is \(50\times1\).

Step 3: Row 7 is \([\,1,\ x_{71},\ x_{72},\ x_{73}\,]\).

Self-check: 4 columns of \(\mathbf{X}\), 4 rows of \(\boldsymbol\beta\); the product is \(50\times1\).

Q3. Use \(\hat\beta_0=335.4323\) and \(\hat\beta_1=0.2608207\). For Q1, calculate component 2 of \(\mathbf{X}\hat{\boldsymbol\beta}\) and of \(\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}\).

Answer

Step 1: Row 2 of \(\mathbf{X}\) is \([1,\ 4177]\).

Step 2: \([\mathbf{X}\hat{\boldsymbol\beta}]_2=1\times335.4323+4177\times0.2608207=335.4323+1089.4481=1424.8804\).

Step 3: \([\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}]_2=1590-1424.8804=165.1196\).

Self-check: The residual is positive: 1590 is above 1424.8804.

Unit 02Matrix review: transpose, inverse, trace

Slides p.9–19 The 11 pages show one slide. Each page adds one rule.

Notation used in this unit · Added

Entry \([\mathbf{C}]_{ij}\): the number in row \(i\), column \(j\) of \(\mathbf{C}\). Also \(a_{ij}=[\mathbf{A}]_{ij}\), \(b_{ij}=[\mathbf{B}]_{ij}\).

Multiplication in entry form: if \(\mathbf{A}\) is \(n\times m\) and \(\mathbf{B}\) is \(m\times p\), then \(\mathbf{AB}\) is \(n\times p\), with \([\mathbf{AB}]_{ij}=\sum_{k=1}^{m}a_{ik}b_{kj}\).

Diagonal: the entries with row number equal to column number, \(a_{11},a_{22},\dots\).

Many examples use this design matrix (\(n=3\), \(p=1\), \(x_1=1\), \(x_2=2\), \(x_3=4\)):

\[\mathbf{X}=\left[\begin{array}{cc}1 & 1\\ 1 & 2\\ 1 & 4\end{array}\right]\qquad(3\times2)\]

02.1Transpose Slides p.9

What Slides p.9

Lecture 5 · p.9 \([\mathbf{C}^T]_{ij}=[\mathbf{C}]_{ji}\)

Transpose \(\mathbf{C}^T\): row \(i\) of \(\mathbf{C}\) becomes column \(i\). If \(\mathbf{C}\) is \(n\times m\), \(\mathbf{C}^T\) is \(m\times n\).

How Added

Find \(\mathbf{C}^T\)

Find the dimension. If \(\mathbf{C}\) is \(n\times m\), \(\mathbf{C}^T\) is \(m\times n\).
Write each row as a column. Row \(i\) of \(\mathbf{C}\) becomes column \(i\) of \(\mathbf{C}^T\).

Self-check: For one entry, confirm \([\mathbf{C}^T]_{ij}=[\mathbf{C}]_{ji}\). Diagonal entries do not move.

Example · \(\mathbf{X}^T\) Added

The rows \((1,1)\), \((1,2)\), \((1,4)\) of \(\mathbf{X}\) become columns:

\[\mathbf{X}^T=\left[\begin{array}{ccc}1 & 1 & 1\\ 1 & 2 & 4\end{array}\right]\qquad(2\times3)\]

Self-check: \([\mathbf{X}^T]_{23}=4=[\mathbf{X}]_{32}\).

Figure 02.1 · Each row of \(\mathbf{X}\) becomes the column with the same number in \(\mathbf{X}^T\). The orange cells hold the same number, 4, with the indices changed.

Why Added

This rule is the definition of the transpose. It needs no proof. Later transpose rules follow from it, entry by entry.

02.2Symmetric matrix Slides p.10

What Slides p.10

Lecture 5 · p.10 \(\mathbf{C}\) is symmetric if \(\mathbf{C}^T=\mathbf{C}\)

Square matrix: an \(n\times n\) matrix.

Symmetric matrix: a matrix equal to its transpose, that is, \([\mathbf{C}]_{ij}=[\mathbf{C}]_{ji}\) for all \(i,j\).

Only a square matrix can be symmetric, because \(\mathbf{C}^T\) must have the dimension of \(\mathbf{C}\).

How Added

Test if \(\mathbf{C}\) is symmetric

Check the dimension. \(\mathbf{C}\) must be \(n\times n\).
Compare pairs. For each \(i\neq j\), compare \([\mathbf{C}]_{ij}\) with \([\mathbf{C}]_{ji}\). All equal: symmetric.

Self-check: Write \(\mathbf{C}^T\). It must be identical to \(\mathbf{C}\).

Example · \(\mathbf{X}^T\mathbf{X}\) Added

A \(2\times3\) matrix times a \(3\times2\) matrix gives a \(2\times2\) matrix.

Entry (1,1): row 1 of \(\mathbf{X}^T\), \((1,1,1)\), times column 1 of \(\mathbf{X}\), \((1,1,1)\)

\[[\mathbf{X}^T\mathbf{X}]_{11}=1\cdot1+1\cdot1+1\cdot1=3\]

Entry (1,2): row 1 of \(\mathbf{X}^T\), \((1,1,1)\), times column 2 of \(\mathbf{X}\), \((1,2,4)\)

\[[\mathbf{X}^T\mathbf{X}]_{12}=1\cdot1+1\cdot2+1\cdot4=7\]

Entry (2,1): row 2 of \(\mathbf{X}^T\), \((1,2,4)\), times column 1 of \(\mathbf{X}\), \((1,1,1)\)

\[[\mathbf{X}^T\mathbf{X}]_{21}=1\cdot1+2\cdot1+4\cdot1=7\]

Entry (2,2): row 2 of \(\mathbf{X}^T\) times column 2 of \(\mathbf{X}\)

\[[\mathbf{X}^T\mathbf{X}]_{22}=1^2+2^2+4^2\]

\[[\mathbf{X}^T\mathbf{X}]_{22}=1+4+16\]

\[[\mathbf{X}^T\mathbf{X}]_{22}=21\]

\[\mathbf{X}^T\mathbf{X}=\left[\begin{array}{cc}3 & 7\\ 7 & 21\end{array}\right]\]

Self-check: The only off-diagonal pair is \(7\) and \(7\). The matrix is symmetric.

Why · this example is symmetric Added

Steps 2 and 3 calculate the same sum \(\sum_i x_i\), with each product in the other order. Slides p.11 proves the general case.

02.3Transpose of a product Slides p.11

What Slides p.11

Lecture 5 · p.11 \((\mathbf{AB})^T=\mathbf{B}^T\mathbf{A}^T\)

Transpose each factor and multiply in reverse order. With \(\mathbf{A}\) \(n\times m\) and \(\mathbf{B}\) \(m\times p\), \(\mathbf{B}^T\mathbf{A}^T\) is \(p\times n\), like \((\mathbf{AB})^T\).

How Added

Expand \((\mathbf{AB})^T\)

Reverse the factors. Write \(\mathbf{A},\mathbf{B}\) as \(\mathbf{B},\mathbf{A}\).
Add \(T\) to each factor. The result is \(\mathbf{B}^T\mathbf{A}^T\).
Simplify \((\mathbf{C}^T)^T=\mathbf{C}\). Two transposes return the indices to their start.

Self-check: The dimension of the new product is the dimension of \(\mathbf{AB}\) in reverse.

Example · a numerical check and \((\mathbf{X}^T\mathbf{X})^T\) Added

Section 02.4 uses these matrices again.

\[\mathbf{A}=\left[\begin{array}{cc}3 & 1\\ 2 & 1\end{array}\right],\qquad \mathbf{B}=\left[\begin{array}{cc}1 & 2\\ 0 & 1\end{array}\right]\]

\[\mathbf{AB}=\left[\begin{array}{cc}3\cdot1+1\cdot0 & 3\cdot2+1\cdot1\\ 2\cdot1+1\cdot0 & 2\cdot2+1\cdot1\end{array}\right]\]

\[\mathbf{AB}=\left[\begin{array}{cc}3 & 7\\ 2 & 5\end{array}\right]\]

\[(\mathbf{AB})^T=\left[\begin{array}{cc}3 & 2\\ 7 & 5\end{array}\right]\]

Slides p.9

\[\mathbf{B}^T\mathbf{A}^T=\left[\begin{array}{cc}1 & 0\\ 2 & 1\end{array}\right]\left[\begin{array}{cc}3 & 2\\ 1 & 1\end{array}\right]\]

\[\mathbf{B}^T\mathbf{A}^T=\left[\begin{array}{cc}1\cdot3+0\cdot1 & 1\cdot2+0\cdot1\\ 2\cdot3+1\cdot1 & 2\cdot2+1\cdot1\end{array}\right]\]

\[\mathbf{B}^T\mathbf{A}^T=\left[\begin{array}{cc}3 & 2\\ 7 & 5\end{array}\right]\]

Self-check: Steps 3 and 6 agree.

Apply the rule to \(\mathbf{X}^T\mathbf{X}\) for any \(\mathbf{X}\):

\[(\mathbf{X}^T\mathbf{X})^T=\mathbf{X}^T(\mathbf{X}^T)^T\]

Slides p.11 with \(\mathbf{A}=\mathbf{X}^T\), \(\mathbf{B}=\mathbf{X}\)

\[(\mathbf{X}^T\mathbf{X})^T=\mathbf{X}^T\mathbf{X}\]

\((\mathbf{X}^T)^T=\mathbf{X}\)

By Slides p.10, \(\mathbf{X}^T\mathbf{X}\) is always symmetric.

Why Added

\[[(\mathbf{AB})^T]_{ij}=[\mathbf{AB}]_{ji}\]

Slides p.9

\[=\sum_k a_{jk}b_{ki}\]

multiplication formula

\[=\sum_k b_{ki}a_{jk}\]

\[=\sum_k [\mathbf{B}^T]_{ik}[\mathbf{A}^T]_{kj}\]

Slides p.9: \(b_{ki}=[\mathbf{B}^T]_{ik}\), \(a_{jk}=[\mathbf{A}^T]_{kj}\)

\[=[\mathbf{B}^T\mathbf{A}^T]_{ij}\]

multiplication formula ∎

02.4Inverse Slides p.12

What Slides p.12

Lecture 5 · p.12 If square matrix (i.e. \(n\times n\)) \(\mathbf{B}\) is nonsingular, then \(\mathbf{BB}^{-1}=\mathbf{B}^{-1}\mathbf{B}=\mathbf{I}\)

Identity matrix \(\mathbf{I}\): a square matrix with 1 on the diagonal and 0 elsewhere. \(\mathbf{CI}=\mathbf{IC}=\mathbf{C}\). For \(2\times2\), \(\mathbf{I}=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]\).

Inverse \(\mathbf{B}^{-1}\): the square matrix whose product with \(\mathbf{B}\), on either side, is \(\mathbf{I}\).

Nonsingular: \(\mathbf{B}^{-1}\) exists.

The inverse acts like the reciprocal \(1/b\) of a number \(b\neq0\). Only a square matrix can have an inverse.

How Added

Find and check \(\mathbf{B}^{-1}\)

Confirm that \(\mathbf{B}\) is square.
Get a candidate inverse. For \(2\times2\), use the optional formula below.
Check the candidate. \(\mathbf{BB}^{-1}\) and \(\mathbf{B}^{-1}\mathbf{B}\) must both equal \(\mathbf{I}\).

Self-check: If one product is not \(\mathbf{I}\), the candidate is not the inverse.

Optional — not on the slides

The determinant of a \(2\times2\) matrix is \(\det\left[\begin{array}{cc}a & b\\ c & d\end{array}\right]=ad-bc\). The matrix is nonsingular if and only if \(ad-bc\neq0\).

Then the inverse is \(\frac{1}{ad-bc}\left[\begin{array}{cc}d & -b\\ -c & a\end{array}\right]\).

Example · \(\mathbf{A}^{-1}\) and \(\mathbf{B}^{-1}\) Added

Use \(\mathbf{A}=\left[\begin{array}{cc}3 & 1\\ 2 & 1\end{array}\right]\) and \(\mathbf{B}=\left[\begin{array}{cc}1 & 2\\ 0 & 1\end{array}\right]\) from 02.3.

Determinant of \(\mathbf{A}\)

\[3\cdot1-1\cdot2=1\]

Candidate \(\mathbf{A}^{-1}\)

\[\mathbf{A}^{-1}=\frac{1}{1}\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]=\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\]

optional formula

Determinant of \(\mathbf{B}\)

\[1\cdot1-2\cdot0=1\]

Candidate \(\mathbf{B}^{-1}\)

\[\mathbf{B}^{-1}=\frac{1}{1}\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]=\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]\]

optional formula

Check \(\mathbf{AA}^{-1}\):

\[\mathbf{AA}^{-1}=\left[\begin{array}{cc}3 & 1\\ 2 & 1\end{array}\right]\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\]

\[=\left[\begin{array}{cc}3\cdot1+1\cdot(-2) & 3\cdot(-1)+1\cdot3\\ 2\cdot1+1\cdot(-2) & 2\cdot(-1)+1\cdot3\end{array}\right]\]

\[=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\]

Check \(\mathbf{A}^{-1}\mathbf{A}\):

\[\mathbf{A}^{-1}\mathbf{A}=\left[\begin{array}{cc}1\cdot3+(-1)\cdot2 & 1\cdot1+(-1)\cdot1\\ (-2)\cdot3+3\cdot2 & (-2)\cdot1+3\cdot1\end{array}\right]\]

\[=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\]

For \(\mathbf{B}\): \(\mathbf{BB}^{-1}=\left[\begin{array}{cc}1\cdot1+2\cdot0 & 1\cdot(-2)+2\cdot1\\ 0\cdot1+1\cdot0 & 0\cdot(-2)+1\cdot1\end{array}\right]=\mathbf{I}\).

Self-check: All products equal \(\mathbf{I}\): \(\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\), \(\mathbf{B}^{-1}=\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]\).

Why Added

Slides p.12 is the definition. To test a candidate, multiply it with the matrix and compare with \(\mathbf{I}\). Sections 02.5 and 02.6 use this test.

02.5Inverse of a product Slides p.13

What Slides p.13

Lecture 5 · p.13 \((\mathbf{AB})^{-1}=\mathbf{B}^{-1}\mathbf{A}^{-1}\), if both square and nonsingular

If \(\mathbf{A}\) and \(\mathbf{B}\) are \(n\times n\) and nonsingular, \(\mathbf{AB}\) is nonsingular. Its inverse is the product of the inverses in reverse order.

How Added

Find \((\mathbf{AB})^{-1}\)

Check the conditions. \(\mathbf{A}\) and \(\mathbf{B}\) are square and nonsingular.
Find each inverse. Find \(\mathbf{A}^{-1}\) and \(\mathbf{B}^{-1}\).
Multiply in reverse order. Calculate \(\mathbf{B}^{-1}\mathbf{A}^{-1}\).

Self-check: The result times \(\mathbf{AB}\) must equal \(\mathbf{I}\).

Example · \((\mathbf{AB})^{-1}\) Added

From 02.3, \(\mathbf{AB}=\left[\begin{array}{cc}3 & 7\\ 2 & 5\end{array}\right]\). \(\mathbf{A}^{-1}\) and \(\mathbf{B}^{-1}\) are from 02.4.

\[\mathbf{B}^{-1}\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\]

\[=\left[\begin{array}{cc}1\cdot1+(-2)(-2) & 1\cdot(-1)+(-2)\cdot3\\ 0\cdot1+1\cdot(-2) & 0\cdot(-1)+1\cdot3\end{array}\right]\]

\[=\left[\begin{array}{cc}5 & -7\\ -2 & 3\end{array}\right]\]

Check against \(\mathbf{AB}\):

\[(\mathbf{AB})(\mathbf{B}^{-1}\mathbf{A}^{-1})=\left[\begin{array}{cc}3\cdot5+7\cdot(-2) & 3\cdot(-7)+7\cdot3\\ 2\cdot5+5\cdot(-2) & 2\cdot(-7)+5\cdot3\end{array}\right]\]

\[=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\]

Self-check: \(\det(\mathbf{AB})=3\cdot5-7\cdot2=1\). The optional formula gives \(\left[\begin{array}{cc}5 & -7\\ -2 & 3\end{array}\right]\) again.

Why Added

Use the test from 02.4:

\[(\mathbf{AB})(\mathbf{B}^{-1}\mathbf{A}^{-1})=\mathbf{A}(\mathbf{BB}^{-1})\mathbf{A}^{-1}\]

Regroup the product (associativity).

\[=\mathbf{AIA}^{-1}\]

Slides p.12

\[=\mathbf{AA}^{-1}\]

\[=\mathbf{I}\]

Slides p.12

The other order: \((\mathbf{B}^{-1}\mathbf{A}^{-1})(\mathbf{AB})=\mathbf{B}^{-1}(\mathbf{A}^{-1}\mathbf{A})\mathbf{B}=\mathbf{B}^{-1}\mathbf{IB}=\mathbf{B}^{-1}\mathbf{B}=\mathbf{I}\). ∎

The reverse order puts \(\mathbf{B}\) next to \(\mathbf{B}^{-1}\).

02.6Inverse of a transpose Slides p.14

What Slides p.14

Lecture 5 · p.14 \((\mathbf{A}^T)^{-1}=(\mathbf{A}^{-1})^T\)

For a nonsingular square \(\mathbf{A}\), the transpose and the inverse can be done in either order.

How Added

Find \((\mathbf{A}^T)^{-1}\)

Find the inverse. Find \(\mathbf{A}^{-1}\).
Transpose it. Write the rows of \(\mathbf{A}^{-1}\) as columns.

Self-check: The result times \(\mathbf{A}^T\) must equal \(\mathbf{I}\).

Example · \((\mathbf{A}^T)^{-1}\) Added

From 02.4

\[\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\]

\[(\mathbf{A}^{-1})^T=\left[\begin{array}{cc}1 & -2\\ -1 & 3\end{array}\right]\]

Slides p.9

Other order: \(\mathbf{A}^T=\left[\begin{array}{cc}3 & 2\\ 1 & 1\end{array}\right]\), determinant \(3\cdot1-2\cdot1=1\)

\[(\mathbf{A}^T)^{-1}=\frac{1}{1}\left[\begin{array}{cc}1 & -2\\ -1 & 3\end{array}\right]\]

optional formula

\[(\mathbf{A}^T)^{-1}=\left[\begin{array}{cc}1 & -2\\ -1 & 3\end{array}\right]\]

Self-check: Steps 2 and 4 agree.

Why Added

Use the test from 02.4:

\[\mathbf{A}^T(\mathbf{A}^{-1})^T=(\mathbf{A}^{-1}\mathbf{A})^T\]

Slides p.11, right to left

\[=\mathbf{I}^T\]

Slides p.12

\[=\mathbf{I}\]

\(\mathbf{I}\) is symmetric

The other order: \((\mathbf{A}^{-1})^T\mathbf{A}^T=(\mathbf{AA}^{-1})^T=\mathbf{I}^T=\mathbf{I}\). ∎

It follows that the inverse of a symmetric nonsingular matrix is symmetric (Practice Q3).

02.7Definition of the trace Slides p.15

What Slides p.15

Lecture 5 · p.15 \(tr(\mathbf{A})=\sum_j^n a_{jj}\) for \(n\times n\) matrix

Trace \(tr(\mathbf{A})\): the sum of the diagonal entries of a square matrix. Only a square matrix has a trace.

How Added

Find \(tr(\mathbf{A})\)

Confirm that \(\mathbf{A}\) is square.
List the diagonal. Write \(a_{11},\dots,a_{nn}\).
Add the diagonal entries.

Self-check: The trace is one number. Off-diagonal entries are not in the sum.

Example · \(tr(\mathbf{A})\), \(tr(\mathbf{B})\), \(tr(\mathbf{X}^T\mathbf{X})\) Added

\[tr(\mathbf{A})=3+1=4\]

\[tr(\mathbf{B})=1+1=2\]

\[tr(\mathbf{X}^T\mathbf{X})=3+21=24\]

\(\mathbf{X}^T\mathbf{X}\) from 02.2

Why Added

Slides p.15 is a definition. It needs no proof.

02.8Trace of a sum Slides p.16

What Slides p.16

Lecture 5 · p.16 \(tr(\mathbf{A}+\mathbf{B})=tr(\mathbf{A})+tr(\mathbf{B})\)

Matrix addition in entry form: \([\mathbf{A}+\mathbf{B}]_{ij}=a_{ij}+b_{ij}\).

How Added

Split \(tr(\mathbf{A}+\mathbf{B})\)

Confirm the dimensions. \(\mathbf{A}\) and \(\mathbf{B}\) are both \(n\times n\).
Add the two traces. You do not need \(\mathbf{A}+\mathbf{B}\).

Self-check: The trace of \(\mathbf{A}+\mathbf{B}\) gives the same number.

Example Added

\[\mathbf{A}+\mathbf{B}=\left[\begin{array}{cc}3+1 & 1+2\\ 2+0 & 1+1\end{array}\right]=\left[\begin{array}{cc}4 & 3\\ 2 & 2\end{array}\right]\]

\[tr(\mathbf{A}+\mathbf{B})=4+2=6\]

\[tr(\mathbf{A})+tr(\mathbf{B})=4+2=6\]

traces from 02.7

Why Added

\[tr(\mathbf{A}+\mathbf{B})=\sum_j^n[\mathbf{A}+\mathbf{B}]_{jj}\]

Slides p.15

\[=\sum_j^n(a_{jj}+b_{jj})\]

matrix addition

\[=\sum_j^n a_{jj}+\sum_j^n b_{jj}\]

\[=tr(\mathbf{A})+tr(\mathbf{B})\]

Slides p.15 ∎

02.9Trace of a scalar multiple Slides p.17

What Slides p.17

Lecture 5 · p.17 \(tr(c\mathbf{A})=c\,tr(\mathbf{A})\)

Scalar \(c\): one ordinary number. \(c\mathbf{A}\) multiplies each entry by \(c\): \([c\mathbf{A}]_{ij}=c\,a_{ij}\).

How Added

Simplify \(tr(c\mathbf{A})\)

Move \(c\) outside. Write \(c\,tr(\mathbf{A})\).
Find \(tr(\mathbf{A})\). Multiply it by \(c\).

Self-check: The trace of \(c\mathbf{A}\) gives the same number.

Example · \(c=3\) Added

\[3\mathbf{A}=\left[\begin{array}{cc}3\cdot3 & 3\cdot1\\ 3\cdot2 & 3\cdot1\end{array}\right]=\left[\begin{array}{cc}9 & 3\\ 6 & 3\end{array}\right]\]

\[tr(3\mathbf{A})=9+3=12\]

\[3\,tr(\mathbf{A})=3\times4=12\]

Why Added

\[tr(c\mathbf{A})=\sum_j^n c\,a_{jj}\]

Slides p.15 and scalar multiplication

\[=c\sum_j^n a_{jj}\]

\[=c\,tr(\mathbf{A})\]

Slides p.15 ∎

02.8 and 02.9 together say that the trace is linear.

02.10Trace of a transpose Slides p.18

What Slides p.18

Lecture 5 · p.18 \(tr(\mathbf{A}^T)=tr(\mathbf{A})\)

The transpose does not move the diagonal entries.

How Added

Simplify \(tr(\mathbf{A}^T)\)

Remove the transpose. Write \(tr(\mathbf{A})\).

Self-check: The diagonal of \(\mathbf{A}^T\) is the diagonal of \(\mathbf{A}\).

Example Added

\(\mathbf{A}^T=\left[\begin{array}{cc}3 & 2\\ 1 & 1\end{array}\right]\). \(tr(\mathbf{A}^T)=3+1=4=tr(\mathbf{A})\).

Why Added

\[tr(\mathbf{A}^T)=\sum_j^n[\mathbf{A}^T]_{jj}\]

Slides p.15

\[=\sum_j^n[\mathbf{A}]_{jj}\]

Slides p.9 with \(i=j\)

\[=tr(\mathbf{A})\]

Slides p.15 ∎

02.11Trace of a product in either order Slides p.19

What Slides p.19

Lecture 5 · p.19 \(tr(\mathbf{AB})=tr(\mathbf{BA})\)

Let \(\mathbf{A}\) be \(n\times m\) and \(\mathbf{B}\) be \(m\times n\). \(\mathbf{AB}\) (\(n\times n\)) and \(\mathbf{BA}\) (\(m\times m\)) have equal traces. \(\mathbf{A}\) and \(\mathbf{B}\) need not be square.

How Added

Use \(tr(\mathbf{AB})=tr(\mathbf{BA})\) to make a calculation shorter

Check the dimensions. \(\mathbf{A}\) is \(n\times m\) and \(\mathbf{B}\) is \(m\times n\).
Select the smaller product. Compare \(n\) and \(m\).
Calculate only its diagonal. Add the diagonal entries.

Self-check: The trace of the other product is equal.

Example · 1 · square \(\mathbf{A}\) and \(\mathbf{B}\) Added

From 02.3, \(\mathbf{AB}=\left[\begin{array}{cc}3 & 7\\ 2 & 5\end{array}\right]\).

\[\mathbf{BA}=\left[\begin{array}{cc}1\cdot3+2\cdot2 & 1\cdot1+2\cdot1\\ 0\cdot3+1\cdot2 & 0\cdot1+1\cdot1\end{array}\right]=\left[\begin{array}{cc}7 & 3\\ 2 & 1\end{array}\right]\]

\[tr(\mathbf{AB})=3+5=8\]

\[tr(\mathbf{BA})=7+1=8\]

\(\mathbf{AB}\neq\mathbf{BA}\), but both traces are 8.

Example · 2 · non-square factors \(\mathbf{X}\) and \(\mathbf{X}^T\) Added

\(\mathbf{XX}^T\) is \(3\times3\); \(\mathbf{X}^T\mathbf{X}\) is \(2\times2\). Entry \((i,j)\) of \(\mathbf{XX}^T\) is row \(i\) of \(\mathbf{X}\) times row \(j\) of \(\mathbf{X}\): \(1+x_ix_j\).

Row 1: \(1+1\cdot1\), \(1+1\cdot2\), \(1+1\cdot4\)

\[(2,\ 3,\ 5)\]

Row 2: \(1+2\cdot1\), \(1+2\cdot2\), \(1+2\cdot4\)

\[(3,\ 5,\ 9)\]

Row 3: \(1+4\cdot1\), \(1+4\cdot2\), \(1+4\cdot4\)

\[(5,\ 9,\ 17)\]

The \(3\times3\) side

\[tr(\mathbf{XX}^T)=2+5+17=24\]

The \(2\times2\) side, from 02.7

\[tr(\mathbf{X}^T\mathbf{X})=3+21=24\]

Self-check: The sides are equal. The \(2\times2\) side needs only 2 diagonal entries.

Figure 02.2 · Blue: the diagonals. \(\mathbf{XX}^T\) and \(\mathbf{X}^T\mathbf{X}\) differ in dimension and entries. Both diagonal sums are 24. Both matrices are symmetric (02.3).

Why Added

Let \(\mathbf{A}\) be \(n\times m\) and \(\mathbf{B}\) be \(m\times n\).

\[tr(\mathbf{AB})=\sum_{i=1}^{n}[\mathbf{AB}]_{ii}\]

Slides p.15

\[=\sum_{i=1}^{n}\sum_{k=1}^{m}a_{ik}b_{ki}\]

multiplication formula

\[=\sum_{k=1}^{m}\sum_{i=1}^{n}b_{ki}a_{ik}\]

Change the order of finite sums and factors.

\[=\sum_{k=1}^{m}[\mathbf{BA}]_{kk}\]

multiplication formula

\[=tr(\mathbf{BA})\]

Slides p.15 ∎

Lecture 7 uses this rule.

02.12Practice Added

Q1. Let \(\mathbf{X}\) be any \(n\times(p+1)\) matrix. With only the rules on Lecture 5 p.9–19, show that \(\mathbf{X}^T\mathbf{X}\) is symmetric and that \(tr(\mathbf{X}^T\mathbf{X})=tr(\mathbf{XX}^T)\). Give the dimensions of \(\mathbf{X}^T\mathbf{X}\) and \(\mathbf{XX}^T\).

Answer

Symmetric: \((\mathbf{X}^T\mathbf{X})^T=\mathbf{X}^T(\mathbf{X}^T)^T=\mathbf{X}^T\mathbf{X}\) (Slides p.11, then \((\mathbf{X}^T)^T=\mathbf{X}\)). By Slides p.10, it is symmetric.

Trace: with \(\mathbf{A}=\mathbf{X}^T\), \(\mathbf{B}=\mathbf{X}\), Slides p.19 gives \(tr(\mathbf{X}^T\mathbf{X})=tr(\mathbf{XX}^T)\).

Dimensions: \(\mathbf{X}^T\mathbf{X}\) is \((p+1)\times(p+1)\). \(\mathbf{XX}^T\) is \(n\times n\).

For the \(\mathbf{X}\) of this unit (\(n=3\), \(p=1\)), both traces are 24 (02.11).

Q2. Let \(\mathbf{A}=\left[\begin{array}{cc}4 & 1\\ 3 & 1\end{array}\right]\). (a) Verify that \(\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -1\\ -3 & 4\end{array}\right]\). (b) Find \((\mathbf{A}^T)^{-1}\). (c) Compute \(tr(\mathbf{A}^T\mathbf{A})\).

Answer

(a) \(\mathbf{AA}^{-1}=\left[\begin{array}{cc}4\cdot1+1\cdot(-3) & 4\cdot(-1)+1\cdot4\\ 3\cdot1+1\cdot(-3) & 3\cdot(-1)+1\cdot4\end{array}\right]=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\).

\(\mathbf{A}^{-1}\mathbf{A}=\left[\begin{array}{cc}1\cdot4+(-1)\cdot3 & 1\cdot1+(-1)\cdot1\\ (-3)\cdot4+4\cdot3 & (-3)\cdot1+4\cdot1\end{array}\right]=\mathbf{I}\).

(b) By Slides p.14, \((\mathbf{A}^T)^{-1}=(\mathbf{A}^{-1})^T=\left[\begin{array}{cc}1 & -3\\ -1 & 4\end{array}\right]\).

(c) \(\mathbf{A}^T=\left[\begin{array}{cc}4 & 3\\ 1 & 1\end{array}\right]\). \(\mathbf{A}^T\mathbf{A}=\left[\begin{array}{cc}4\cdot4+3\cdot3 & 4\cdot1+3\cdot1\\ 1\cdot4+1\cdot3 & 1\cdot1+1\cdot1\end{array}\right]=\left[\begin{array}{cc}25 & 7\\ 7 & 2\end{array}\right]\).

\(tr(\mathbf{A}^T\mathbf{A})=25+2=27\). By Slides p.19, \(tr(\mathbf{AA}^T)=27\) also.

Q3. \(\mathbf{C}\) is \(n\times n\), symmetric and nonsingular. Show that \(\mathbf{C}^{-1}\) is symmetric.

Answer

\[(\mathbf{C}^{-1})^T=(\mathbf{C}^T)^{-1}\]

Slides p.14

\[=\mathbf{C}^{-1}\]

\(\mathbf{C}^T=\mathbf{C}\) (Slides p.10)

By Slides p.10, \(\mathbf{C}^{-1}\) is symmetric. ∎ For example, \((\mathbf{X}^T\mathbf{X})^{-1}\) is symmetric when it exists.

Check with \(\mathbf{X}^T\mathbf{X}=\left[\begin{array}{cc}3 & 7\\ 7 & 21\end{array}\right]\): determinant \(3\cdot21-7\cdot7=14\); inverse \(\frac{1}{14}\left[\begin{array}{cc}21 & -7\\ -7 & 3\end{array}\right]\). Both off-diagonal entries are \(-7/14\).

Unit 03Matrix calculus

Symbols Added

\(\mathbf{y}=(y_1,\dots,y_k)^T\) is a column vector with \(k\) components.

\(z=f(y_1,\dots,y_k)\) is a function that gives one scalar \(z\).

Partial derivative \(\frac{\partial z}{\partial y_l}\): the derivative of \(z\) with respect to \(y_l\), with all other \(y_i\) constant.

Matrix-vector product: \((\mathbf{A}\mathbf{y})_l=\sum_{j=1}^k a_{lj}y_j\) (row \(l\) of \(\mathbf{A}\) times \(\mathbf{y}\)).

03.1The derivative with respect to a vector Slides p.20

What Slides p.20

Lecture 5 · p.20 Let \(z=f(y_1,\dots,y_k)\) and \(\mathbf{y}=\left[\begin{array}{c}y_1\\ \vdots\\ y_k\end{array}\right]\), then \[\frac{\partial z}{\partial \mathbf{y}}=\left[\begin{array}{c}\frac{\partial z}{\partial y_1}\\ \frac{\partial z}{\partial y_2}\\ \vdots\\ \frac{\partial z}{\partial y_k}\end{array}\right]\]

The gradient \(\frac{\partial z}{\partial \mathbf{y}}\) is the column of the \(k\) partial derivatives. Component \(l\) shows how fast \(z\) changes when only \(y_l\) changes.

The gradient is \(k\times1\), like \(\mathbf{y}\).

How Added

Find \(\partial z/\partial \mathbf{y}\) from the definition

Write \(z\) as a formula in \(y_1,\dots,y_k\).
For each \(l=1,\dots,k\), find \(\frac{\partial z}{\partial y_l}\).
Put the \(k\) results in one column, in the order of \(l\).

Self-check: The result is \(k\times1\). Row \(l\) is the partial derivative with respect to \(y_l\).

Example Added

Let \(k=2\) and \(z=3y_1+y_1y_2\). Find \(\frac{\partial z}{\partial \mathbf{y}}\) at \(\mathbf{y}=(1,2)^T\).

Write \(z\) How step 1

\[z=3y_1+y_1y_2\]

Partial derivatives How step 2

\[\frac{\partial z}{\partial y_1}=3+y_2,\qquad \frac{\partial z}{\partial y_2}=y_1\]

One column How step 3

\[\frac{\partial z}{\partial \mathbf{y}}=\left[\begin{array}{c}3+y_2\\ y_1\end{array}\right]\]

\[\frac{\partial z}{\partial \mathbf{y}}\Big|_{\mathbf{y}=(1,2)^T}=\left[\begin{array}{c}3+2\\ 1\end{array}\right]=\left[\begin{array}{c}5\\ 1\end{array}\right]\]

Why Added

Slides p.20 is a definition. It needs no proof.

03.2Linear function: \(\partial(\mathbf{a}^T\mathbf{y})/\partial\mathbf{y}=\mathbf{a}\) Slides p.20

What Slides p.20

Lecture 5 · p.20 If \(z=\mathbf{a}^T\mathbf{y}\), where \(\mathbf{a}=(a_1,\dots,a_k)^T\) is a vector, then \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}\]

\(\mathbf{a}\) is constant. \(z=\mathbf{a}^T\mathbf{y}=a_1y_1+\dots+a_ky_k\) is a scalar and a linear function: each \(y_i\) has power 1. The gradient is \(\mathbf{a}\) at all \(\mathbf{y}\).

How Added

Use the rule for \(\partial(\mathbf{a}^T\mathbf{y})/\partial\mathbf{y}\)

Write \(z\) as a constant vector, transposed, times \(\mathbf{y}\).
Read \(\mathbf{a}\): \(a_l\) is the coefficient of \(y_l\).
Write \(\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}\).

Self-check: The result is \(k\times1\) and contains no \(y_i\). For one \(l\), \(\partial z/\partial y_l=a_l\).

Example · a linear function Added

Let \(k=3\) and \(\mathbf{a}=(1,2,3)^T\).

Write \(z\) How step 1

\[z=\mathbf{a}^T\mathbf{y}=\left[\begin{array}{ccc}1&2&3\end{array}\right]\left[\begin{array}{c}y_1\\ y_2\\ y_3\end{array}\right]\]

\[z=1\cdot y_1+2\cdot y_2+3\cdot y_3=y_1+2y_2+3y_3\]

Use the rule How steps 2–3

\[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}=\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]\]

Check with the definition 03.1 How

\[\frac{\partial z}{\partial y_1}=1,\qquad \frac{\partial z}{\partial y_2}=2,\qquad \frac{\partial z}{\partial y_3}=3\]

Basis: 03.1. For \(y_1\), the terms \(2y_2\) and \(3y_3\) are constant.

At \(\mathbf{y}=(2,-1,1)^T\), \(z=2-2+3=3\), and the gradient is still \((1,2,3)^T\). ▲

Figure · gradient of \(z=y_1+2y_2\) Added

Figure 03.1 · \(z=y_1+2y_2\). Blue lines: level lines (constant \(z\)), 1 apart in \(z\). Thick line: \(z=3\). Orange arrow: the gradient \(\mathbf{a}=(1,2)^T\) at the green point \((1,1)^T\). It is at a right angle to the level lines (equal axis scales). The gradient is the same at all points.

Why · derivation of the rule in 03.2 Added

\[z=\mathbf{a}^T\mathbf{y}=\sum_{i=1}^k a_iy_i\]

\[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k a_i\frac{\partial y_i}{\partial y_l}\]

Each \(a_i\) is a constant.

\[\frac{\partial y_i}{\partial y_l}=\begin{cases}1, & i=l\\ 0, & i\neq l\end{cases}\]

For \(i\neq l\), \(y_i\) is constant.

\[\frac{\partial z}{\partial y_l}=a_l\]

Only the term \(i=l\) stays.

\[\frac{\partial z}{\partial \mathbf{y}}=\left[\begin{array}{c}a_1\\ \vdots\\ a_k\end{array}\right]=\mathbf{a}\]

Stack \(l=1,\dots,k\) (03.1). ∎

03.3Quadratic form: \(\partial(\mathbf{y}^T\mathbf{A}\mathbf{y})/\partial\mathbf{y}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\) Slides p.21

What Slides p.21

Lecture 5 · p.21 If \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\) where \(\mathbf{A}\) is a \(k\times k\) matrix, then \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\] and if \(\mathbf{A}\) is symmetric, then: \(\frac{\partial z}{\partial \mathbf{y}}=2\mathbf{A}\mathbf{y}\)

A quadratic form is \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}y_iy_j\), a scalar. Each term is a coefficient times two components.

If \(\mathbf{A}\) is symmetric, \(\mathbf{A}^T\mathbf{y}=\mathbf{A}\mathbf{y}\), and the two terms become \(2\mathbf{A}\mathbf{y}\).

How Added

Use the rule for \(\partial(\mathbf{y}^T\mathbf{A}\mathbf{y})/\partial\mathbf{y}\)

Confirm that \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\), with \(\mathbf{A}\) a constant \(k\times k\) matrix.
Compare \(a_{ij}\) with \(a_{ji}\) for each pair. All equal: \(\mathbf{A}\) is symmetric.
If \(\mathbf{A}\) is symmetric, calculate \(2\mathbf{A}\mathbf{y}\).
If not, calculate \(\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\).
Put in the given value of \(\mathbf{y}\), if any.

Self-check: The result is \(k\times1\). Each component is linear in \(y_1,\dots,y_k\), with no constant term. One expanded partial derivative must agree.

Example · a matrix that is not symmetric Added

Let \(k=2\), \(\mathbf{A}=\left[\begin{array}{cc}1&2\\ 0&3\end{array}\right]\), \(\mathbf{y}=(1,2)^T\).

Write \(z\) How step 1

\[z=\sum_{i=1}^2\sum_{j=1}^2 a_{ij}y_iy_j=a_{11}y_1^2+a_{12}y_1y_2+a_{21}y_2y_1+a_{22}y_2^2\]

\[z=1\cdot y_1^2+2y_1y_2+0\cdot y_2y_1+3y_2^2=y_1^2+2y_1y_2+3y_2^2\]

\[z(1,2)=1+4+12=17\]

Examine symmetry How step 2

\[\mathbf{A}^T=\left[\begin{array}{cc}1&0\\ 2&3\end{array}\right]\]

\[a_{12}=2,\quad a_{21}=0,\quad a_{12}\neq a_{21}\]

Not symmetric: use the general rule.

Calculate \(\mathbf{A}\mathbf{y}\) How step 4

\[\mathbf{A}\mathbf{y}=\left[\begin{array}{c}1\cdot1+2\cdot2\\ 0\cdot1+3\cdot2\end{array}\right]=\left[\begin{array}{c}5\\ 6\end{array}\right]\]

Calculate \(\mathbf{A}^T\mathbf{y}\) How step 4

\[\mathbf{A}^T\mathbf{y}=\left[\begin{array}{c}1\cdot1+0\cdot2\\ 2\cdot1+3\cdot2\end{array}\right]=\left[\begin{array}{c}1\\ 8\end{array}\right]\]

Add How steps 4–5

\[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}=\left[\begin{array}{c}5+1\\ 6+8\end{array}\right]=\left[\begin{array}{c}6\\ 14\end{array}\right]\]

Check with the definition 03.1 How

\[\frac{\partial z}{\partial y_1}=2y_1+2y_2=2\cdot1+2\cdot2=6\]

\[\frac{\partial z}{\partial y_2}=2y_1+6y_2=2\cdot1+6\cdot2=14\]

Basis: \(z=y_1^2+2y_1y_2+3y_2^2\) from step 1.

Example · a symmetric matrix Added

Let \(k=3\), \(\mathbf{A}=\left[\begin{array}{ccc}4&0&1\\ 0&3&-1\\ 1&-1&4\end{array}\right]\), \(\mathbf{y}=(1,2,2)^T\).

Write \(z\) How step 1

\[z=\sum_{i=1}^3\sum_{j=1}^3 a_{ij}y_iy_j\]

\[z=4y_1^2+0\cdot y_1y_2+1\cdot y_1y_3+0\cdot y_2y_1+3y_2^2-1\cdot y_2y_3+1\cdot y_3y_1-1\cdot y_3y_2+4y_3^2\]

\[z=4y_1^2+3y_2^2+4y_3^2+2y_1y_3-2y_2y_3\]

\[z(1,2,2)=4+12+16+4-8=28\]

Examine symmetry How step 2

\[a_{12}=a_{21}=0,\quad a_{13}=a_{31}=1,\quad a_{23}=a_{32}=-1\]

\(\mathbf{A}^T=\mathbf{A}\): use \(2\mathbf{A}\mathbf{y}\).

Calculate \(\mathbf{A}\mathbf{y}\) How step 3

\[\mathbf{A}\mathbf{y}=\left[\begin{array}{c}4\cdot1+0\cdot2+1\cdot2\\ 0\cdot1+3\cdot2+(-1)\cdot2\\ 1\cdot1+(-1)\cdot2+4\cdot2\end{array}\right]=\left[\begin{array}{c}6\\ 4\\ 7\end{array}\right]\]

Multiply by 2 How steps 3, 5

\[\frac{\partial z}{\partial \mathbf{y}}=2\mathbf{A}\mathbf{y}=\left[\begin{array}{c}12\\ 8\\ 14\end{array}\right]\]

Check with the definition 03.1 How

\[\frac{\partial z}{\partial y_1}=8y_1+2y_3=8+4=12\]

\[\frac{\partial z}{\partial y_2}=6y_2-2y_3=12-4=8\]

\[\frac{\partial z}{\partial y_3}=8y_3+2y_1-2y_2=16+2-4=14\]

Basis: \(z\) from step 1, at \(\mathbf{y}=(1,2,2)^T\).

For a symmetric matrix, calculate \(\mathbf{A}\mathbf{y}\) one time only. ▲

Why · derivation of the rule in 03.3 Added

The product rule: \((uv)'=u'v+uv'\).

\[z=\mathbf{y}^T\mathbf{A}\mathbf{y}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}y_iy_j\]

\[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}\frac{\partial (y_iy_j)}{\partial y_l}\]

\[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}\left(\frac{\partial y_i}{\partial y_l}y_j+y_i\frac{\partial y_j}{\partial y_l}\right)\]

Product rule.

\[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}\frac{\partial y_i}{\partial y_l}y_j+\sum_{i=1}^k\sum_{j=1}^k a_{ij}y_i\frac{\partial y_j}{\partial y_l}\]

\[\frac{\partial z}{\partial y_l}=\sum_{j=1}^k a_{lj}y_j+\sum_{i=1}^k a_{il}y_i\]

03.2 Why step 3: \(i=l\), then \(j=l\), stays.

\[\sum_{j=1}^k a_{lj}y_j=(\mathbf{A}\mathbf{y})_l\]

Row \(l\) of \(\mathbf{A}\) times \(\mathbf{y}\).

\[\sum_{i=1}^k a_{il}y_i=(\mathbf{A}^T\mathbf{y})_l\]

Row \(l\) of \(\mathbf{A}^T\) has entries \(a_{il}\).

\[\frac{\partial z}{\partial y_l}=(\mathbf{A}\mathbf{y})_l+(\mathbf{A}^T\mathbf{y})_l\]

\[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\]

Stack \(l=1,\dots,k\) (03.1).

\[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}\mathbf{y}\]

Symmetric: \(\mathbf{A}^T=\mathbf{A}\).

\[\frac{\partial z}{\partial \mathbf{y}}=2\mathbf{A}\mathbf{y}\]

∎

The two terms come from the derivatives of \(y_i\) and of \(y_j\).

03.4Practice Added

Q1. Let \(\mathbf{y}=(y_1,y_2,y_3)^T\), \(\mathbf{a}=(2,0,-1)^T\), and \(z=\mathbf{a}^T\mathbf{y}+\mathbf{y}^T\mathbf{y}\). Find \(\frac{\partial z}{\partial \mathbf{y}}\) at \(\mathbf{y}=(1,1,1)^T\).

Answer

The derivative of a sum is the sum of the derivatives.

\(\mathbf{y}^T\mathbf{y}=\mathbf{y}^T\mathbf{I}\mathbf{y}\), and \(\mathbf{I}\) is symmetric.

\(\frac{\partial(\mathbf{a}^T\mathbf{y})}{\partial\mathbf{y}}=\mathbf{a}\) (03.2). \(\frac{\partial(\mathbf{y}^T\mathbf{I}\mathbf{y})}{\partial\mathbf{y}}=2\mathbf{I}\mathbf{y}=2\mathbf{y}\) (03.3).

\(\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}+2\mathbf{y}\). At \(\mathbf{y}=(1,1,1)^T\): \((2+2,\ 0+2,\ -1+2)^T=(4,2,1)^T\).

Check: \(z=2y_1-y_3+y_1^2+y_2^2+y_3^2\). \(\partial z/\partial y_1=2+2y_1=4\), \(\partial z/\partial y_2=2y_2=2\), \(\partial z/\partial y_3=-1+2y_3=1\).

Q2. Let \(\mathbf{A}=\left[\begin{array}{cc}2&1\\ 3&1\end{array}\right]\) and \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\), \(\mathbf{y}=(y_1,y_2)^T\). (a) Write \(z\) as a polynomial in \(y_1,y_2\). (b) Compute \(\frac{\partial z}{\partial \mathbf{y}}\) at \(\mathbf{y}=(1,-1)^T\) with the rule on p.21. (c) Check (b) with the partial derivatives of (a).

Answer

(a) \(z=2y_1^2+1\cdot y_1y_2+3y_2y_1+1\cdot y_2^2=2y_1^2+4y_1y_2+y_2^2\). At \((1,-1)^T\): \(z=2-4+1=-1\).

(b) \(a_{12}=1\neq a_{21}=3\): not symmetric. Use \(\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\).

\(\mathbf{A}\mathbf{y}=(2\cdot1+1\cdot(-1),\ 3\cdot1+1\cdot(-1))^T=(1,2)^T\).

\(\mathbf{A}^T=\left[\begin{array}{cc}2&3\\ 1&1\end{array}\right]\), \(\mathbf{A}^T\mathbf{y}=(2\cdot1+3\cdot(-1),\ 1\cdot1+1\cdot(-1))^T=(-1,0)^T\).

\(\frac{\partial z}{\partial \mathbf{y}}=(1+(-1),\ 2+0)^T=(0,2)^T\).

(c) \(\frac{\partial z}{\partial y_1}=4y_1+4y_2=4-4=0\). \(\frac{\partial z}{\partial y_2}=4y_1+2y_2=4-2=2\). Same as (b).

Q3. Let \(z=3y_1^2+4y_1y_2+y_2^2\). Find a symmetric \(2\times2\) matrix \(\mathbf{A}\) with \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\). Find the gradient with \(2\mathbf{A}\mathbf{y}\). Confirm it with partial derivatives.

Answer

For a symmetric \(\mathbf{A}\), \(z=a_{11}y_1^2+(a_{12}+a_{21})y_1y_2+a_{22}y_2^2=a_{11}y_1^2+2a_{12}y_1y_2+a_{22}y_2^2\). Match the coefficients.

\(a_{11}=3\), \(a_{22}=1\), \(2a_{12}=4\): \(a_{12}=a_{21}=2\). \(\mathbf{A}=\left[\begin{array}{cc}3&2\\ 2&1\end{array}\right]\).

\(2\mathbf{A}\mathbf{y}=2\left[\begin{array}{c}3y_1+2y_2\\ 2y_1+y_2\end{array}\right]=\left[\begin{array}{c}6y_1+4y_2\\ 4y_1+2y_2\end{array}\right]\).

Direct: \(\frac{\partial z}{\partial y_1}=6y_1+4y_2\), \(\frac{\partial z}{\partial y_2}=4y_1+2y_2\). They agree.

Unit 04Random vectors: mean vector, covariance matrix, and linear transformations

Given values · Added

All examples use brainhead.csv (236 persons; not printed on the slides).

Random vector \(\mathbf{y}=(y_1,y_2)^T\): \(y_1\) is head size (cm³), \(y_2\) is brain weight (g) of one person. In the slide formulas, \(n=2\).

The sample values of the 236 persons act as the true values.

Given mean vector: \(\boldsymbol\mu=(3637.864,\ 1284.263)^T\).

Given covariance matrix: \(\mathbf{V}=\left[\begin{array}{cc}130413.76&34014.61\\34014.61&14084.06\end{array}\right]\).

04.1Random vector and mean vector Slides p.22

What Slides p.22

Random Vectors · Lecture 5 · p.22 Let \(\mathbf{y}=(y_1,\dots,y_n)^T\) be a random vector. The mean of \(\mathbf{y}\) is \[E[\mathbf{y}]=\left[\begin{array}{c}E[y_1]\\ \vdots\\ E[y_n]\end{array}\right]\]

A random vector \(\mathbf{y}\) is an \(n\times1\) column of random variables \(y_1,\dots,y_n\).

The mean vector \(E[\mathbf{y}]\) has entry \(i\) equal to \(E[y_i]\). The slides write \(\boldsymbol\mu=E[\mathbf{y}]\) and \(\mu_i=E[y_i]\).

A random matrix is a matrix of random variables. Its expectation is also taken entry by entry.

How Added

Write the mean vector \(E[\mathbf{y}]\)

List the entries \(y_1,\dots,y_n\) in order.
Get \(E[y_i]\) for each entry.
Put \(E[y_1],\dots,E[y_n]\) in one column, in the same order.

Self-check: \(E[\mathbf{y}]\) has the length of \(\mathbf{y}\).

Example · brainhead Added How steps 1–3

Step 1: \(\mathbf{y}=(y_1,y_2)^T\), \(n=2\).

Step 2: mean head size 3637.864; mean brain weight 1284.263.

Step 3: \(E[\mathbf{y}]=\boldsymbol\mu=\left[\begin{array}{c}3637.864\\1284.263\end{array}\right]\). ▲

Figure 04.1 · The 236 persons (blue) and the mean vector \(\boldsymbol\mu\) (orange). The purple lines extend one standard deviation to each side: \(\sqrt{\mathbf{V}_{11}}=361.13\) (horizontal) and \(\sqrt{\mathbf{V}_{22}}=118.68\) (vertical). The upward trend shows \(cov(y_1,y_2)\gt 0\).

Why Added

Slides p.22 is a definition. It needs no proof.

04.2Covariance matrix Slides p.23

What Slides p.23

Covariance Matrix · Lecture 5 · p.23 The variance is \[var(\mathbf{y})=E[(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T]\] \[\mathbf{V}=var(\mathbf{y})=\left[\begin{array}{cccc}var(y_1)&cov(y_1,y_2)&\cdots&cov(y_1,y_n)\\cov(y_2,y_1)&var(y_2)&\cdots&cov(y_2,y_n)\\ \vdots&\vdots&\ddots&\vdots\\cov(y_n,y_1)&cov(y_n,y_2)&\cdots&var(y_n)\end{array}\right]\]

The covariance \(cov(y_i,y_j)=E[(y_i-\mu_i)(y_j-\mu_j)]\) is the expected product of the two distances from the means.

Positive covariance: the two variables tend to be large together. Negative: one tends to be large when the other is small.

\(cov(y_i,y_i)=var(y_i)\).

The covariance matrix \(\mathbf{V}=var(\mathbf{y})\) is the \(n\times n\) matrix with \(\mathbf{V}_{ij}=cov(y_i,y_j)\).

\((\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T\) is an \(n\times1\) column times a \(1\times n\) row: an \(n\times n\) random matrix.

How Added

Write the covariance matrix \(\mathbf{V}=var(\mathbf{y})\)

Find the length \(n\) of \(\mathbf{y}\). Make an empty \(n\times n\) table.
In diagonal entry \(i\), write \(var(y_i)\).
In entry \((i,j)\), \(i\ne j\), write \(cov(y_i,y_j)\).

Self-check: \(\mathbf{V}\) is \(n\times n\). Diagonal entries are \(\ge0\). \(\mathbf{V}_{ij}=\mathbf{V}_{ji}\) (04.3).

Example · brainhead Added How steps 1–3

Step 1: \(n=2\): \(\mathbf{V}\) is \(2\times2\).

Step 2: \(var(y_1)=130413.76\), \(var(y_2)=14084.06\).

Step 3: \(cov(y_1,y_2)=cov(y_2,y_1)=34014.61\). Positive: a larger head tends to come with a larger brain weight.

Result: \[\mathbf{V}=var(\mathbf{y})=\left[\begin{array}{cc}var(y_1)&cov(y_1,y_2)\\cov(y_2,y_1)&var(y_2)\end{array}\right]=\left[\begin{array}{cc}130413.76&34014.61\\34014.61&14084.06\end{array}\right]\]

Standard deviations: \(\sqrt{130413.76}=361.13\), \(\sqrt{14084.06}=118.68\). ▲

Why · the entries are variances and covariances Added

\[\mathbf{y}-\boldsymbol\mu=\left[\begin{array}{c}y_1-\mu_1\\ \vdots\\ y_n-\mu_n\end{array}\right]\]

\[\big[(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T\big]_{ij}=(y_i-\mu_i)(y_j-\mu_j)\]

Column times row.

\[\big[E[(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T]\big]_{ij}=E[(y_i-\mu_i)(y_j-\mu_j)]\]

Expectation entry by entry (04.1).

\[=cov(y_i,y_j)\]

Definition of covariance. ∎

04.3\(\mathbf{V}\) is symmetric Slides p.24

What Slides p.24

Covariance Matrix · Lecture 5 · p.24 \(\mathbf{V}\) is symmetric: \[\mathbf{V}_{ij}=\mathbf{V}_{ji}\]

Calculate only the entries on and above the diagonal.

How Added

Use the symmetry of \(\mathbf{V}\)

Calculate the diagonal entries and the entries with \(i<j\).
Copy each \(\mathbf{V}_{ij}\) to position \((j,i)\).

Self-check: Each pair \(\mathbf{V}_{ij}\), \(\mathbf{V}_{ji}\) is equal.

Example · brainhead Added How steps 1–2

Step 1: \(\mathbf{V}_{12}=cov(y_1,y_2)=34014.61\).

Step 2: \(\mathbf{V}_{21}=cov(y_2,y_1)=34014.61\). ▲

Why · \(\mathbf{V}\) is symmetric Added

\[\mathbf{V}_{ij}=cov(y_i,y_j)=E[(y_i-\mu_i)(y_j-\mu_j)]\]

Entry formula (04.2).

\[=E[(y_j-\mu_j)(y_i-\mu_i)]\]

Factor order does not change a product.

\[=cov(y_j,y_i)=\mathbf{V}_{ji}\]

∎

04.4\(\mathbf{V}\) is positive semi-definite Slides p.25

What Slides p.25

Covariance Matrix · Lecture 5 · p.25 \(\mathbf{V}\) is positive semi-definite: for all \(\mathbf{a}\) in \(\mathbb{R}^n\) \[\mathbf{a}^T\mathbf{V}\mathbf{a}\ge0\]

\(\mathbb{R}^n\) is the set of all columns of \(n\) real numbers. \(\mathbf{a}\) is a constant \(n\times1\) vector.

\(\mathbf{a}^T\mathbf{V}\mathbf{a}=\sum_i\sum_j a_i\mathbf{V}_{ij}a_j\) is one number.

A symmetric matrix is positive semi-definite when \(\mathbf{a}^T\mathbf{V}\mathbf{a}\ge0\) for every \(\mathbf{a}\).

How Added

Calculate \(\mathbf{a}^T\mathbf{V}\mathbf{a}\) for a given \(\mathbf{a}\)

Calculate the column \(\mathbf{V}\mathbf{a}\).
Multiply each entry of \(\mathbf{a}\) by the matching entry of \(\mathbf{V}\mathbf{a}\).
Add the products.

Self-check: The result is one number, \(\ge0\).

Example · brainhead, \(\mathbf{a}=(-0.3,\ 1)^T\) Added How steps 1–3

Step 1: \[\mathbf{V}\mathbf{a}=\left[\begin{array}{c}130413.76\times(-0.3)+34014.61\times1\\34014.61\times(-0.3)+14084.06\times1\end{array}\right]=\left[\begin{array}{c}-39124.128+34014.61\\-10204.383+14084.06\end{array}\right]=\left[\begin{array}{c}-5109.518\\3879.677\end{array}\right]\]

Step 2: \((-0.3)(-5109.518)=1532.8554\) and \((1)(3879.677)=3879.677\).

Step 3: \(\mathbf{a}^T\mathbf{V}\mathbf{a}=1532.8554+3879.677=5412.5324\ge0\). ▲

This is the variance of \(z=-0.3y_1+y_2\) (04.5).

Why · \(\mathbf{V}\) is positive semi-definite Added

Linearity of expectation: a constant moves into or out of \(E[\cdot]\).

\[\mathbf{a}^T\mathbf{V}\mathbf{a}=\mathbf{a}^TE[(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T]\mathbf{a}\]

Definition of \(\mathbf{V}\) (p.23).

\[=E[\mathbf{a}^T(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T\mathbf{a}]\]

\(\mathbf{a}\) is constant.

\[=E[\big(\mathbf{a}^T(\mathbf{y}-\boldsymbol\mu)\big)\big(\mathbf{a}^T(\mathbf{y}-\boldsymbol\mu)\big)]\]

A number equals its transpose.

\[=E[\big(\mathbf{a}^T(\mathbf{y}-\boldsymbol\mu)\big)^2]\]

\[\ge0\]

The expectation of a square is \(\ge0\). ∎

04.5Linear combinations and properties of random vectors Slides p.26–29

What (1) Slides p.26

Linear Combinations of Random Variables · Lecture 5 · p.26 Recall that if: \(z=\sum_{i=1}^n a_iy_i+c\) and \(u=\sum_{i=1}^n b_iy_i+d\). Then: \[E[z]=\sum_{i=1}^n a_iE[y_i]+c\] \[cov(z,u)=\sum_{i=1}^n\sum_{j=1}^n a_ib_j\,cov(y_i,y_j)\] Equivalently in matrix notation: \(z=\mathbf{a}^T\mathbf{y}+c\) and \(u=\mathbf{b}^T\mathbf{y}+d\) \[E[z]=\mathbf{a}^T\boldsymbol\mu+c\] \[cov(z,u)=\mathbf{a}^T\mathbf{V}\mathbf{b}\] where \(\boldsymbol\mu=E[\mathbf{y}]\) and \(\mathbf{V}=var(\mathbf{y})\)

A linear combination multiplies each \(y_i\) by a constant weight, adds the products, and can add a constant. \(z\) and \(u\) are random variables.

\(\mathbf{a}=(a_1,\dots,a_n)^T\) and \(\mathbf{b}=(b_1,\dots,b_n)^T\) hold the constant weights. \(c\) and \(d\) are constants.

What (2) Slides p.27–29

Properties of Random Vectors · Lecture 5 · p.27–29 Consider a random vector \(\mathbf{y}\) and constant vector \(\mathbf{a}\)

\(E[\mathbf{a}]=\mathbf{a}\) (p.27)
\(E[\mathbf{a}^T\mathbf{y}+b]=\mathbf{a}^TE[\mathbf{y}]+b\) (p.28)
\(Var(\mathbf{a}^T\mathbf{y})=\mathbf{a}^TVar(\mathbf{y})\mathbf{a}\) (p.29)

\(b\) is a constant number, like \(c\) on p.26. \(Var(\mathbf{y})=var(\mathbf{y})=\mathbf{V}\).

The three properties are special cases of p.26. The third uses \(u=z\), because \(cov(z,z)=Var(z)\).

How Added

Calculate \(E[z]\), \(Var(z)\) and \(cov(z,u)\)

Read \(\mathbf{a}\), \(c\) from \(z\) and \(\mathbf{b}\), \(d\) from \(u\). Write weight 0 for a missing \(y_i\).
\(E[z]\): calculate \(\mathbf{a}^T\boldsymbol\mu+c\).
\(cov(z,u)\): calculate \(\mathbf{V}\mathbf{b}\), then \(\mathbf{a}^T(\mathbf{V}\mathbf{b})\). Do not use \(c\) or \(d\).
\(Var(z)\): use step 3 with \(\mathbf{b}=\mathbf{a}\).

Self-check: \(\mathbf{a}\) and \(\mathbf{b}\) have the length of \(\mathbf{y}\). The sum form gives the same result. \(Var(z)\ge0\).

Example · brainhead, \(z=-0.3y_1+y_2\) and \(u=y_1-y_2\) Added How steps 1–4

Step 1: \(\mathbf{a}=(-0.3,\ 1)^T\), \(c=0\), \(\mathbf{b}=(1,\ -1)^T\), \(d=0\). By p.27, \(E[\mathbf{a}]=(-0.3,\ 1)^T\).

Step 2: \(E[z]=\mathbf{a}^T\boldsymbol\mu+c=(-0.3)(3637.864)+(1)(1284.263)+0=-1091.3592+1284.263=192.9038\).

Step 3 (matrix form): \[\mathbf{V}\mathbf{b}=\left[\begin{array}{c}130413.76-34014.61\\34014.61-14084.06\end{array}\right]=\left[\begin{array}{c}96399.15\\19930.55\end{array}\right]\] \[cov(z,u)=\mathbf{a}^T\mathbf{V}\mathbf{b}=(-0.3)(96399.15)+(1)(19930.55)=-28919.745+19930.55=-8989.195\]

Step 3 (sum form, 4 terms): \[a_1b_1cov(y_1,y_1)=(-0.3)(1)(130413.76)=-39124.128\] \[a_1b_2cov(y_1,y_2)=(-0.3)(-1)(34014.61)=10204.383\] \[a_2b_1cov(y_2,y_1)=(1)(1)(34014.61)=34014.61\] \[a_2b_2cov(y_2,y_2)=(1)(-1)(14084.06)=-14084.06\] Sum: \(-39124.128+10204.383+34014.61-14084.06=-8989.195\).

Step 4: \(Var(z)=\mathbf{a}^T\mathbf{V}\mathbf{a}=5412.5324\) (04.4). Sum form: \[a_1^2\mathbf{V}_{11}+2a_1a_2\mathbf{V}_{12}+a_2^2\mathbf{V}_{22}=(0.09)(130413.76)+2(-0.3)(1)(34014.61)+(1)(14084.06)\] \[=11737.2384-20408.766+14084.06=5412.5324\]

\(\boldsymbol\mu\) and \(\mathbf{V}\) are sufficient. The 236 data rows are not necessary. ▲

Figure 04.2 shows the 236 values \(z_i=-0.3\times(\text{head size})_i+(\text{brain weight})_i\). Example: \(z_1=-0.3\times4512+1530=-1353.6+1530=176.4\). A histogram is a bar chart of the count of values in equal-width intervals.

Figure 04.2 · Histogram of the 236 values \(z_i\) (interval width 25). Orange: \(\mathbf{a}^T\boldsymbol\mu=192.90\), the center. Purple: one standard deviation \(\sqrt{\mathbf{a}^T\mathbf{V}\mathbf{a}}=73.57\) to each side.

Why · formulas for linear combinations Added

\[E[z]=E\Big[\sum_{i=1}^n a_iy_i+c\Big]=\sum_{i=1}^n a_iE[y_i]+c\]

Linearity of expectation.

\[=\sum_{i=1}^n a_i\mu_i+c=\mathbf{a}^T\boldsymbol\mu+c\]

\(\mu_i=E[y_i]\).

\[z-E[z]=\sum_{i=1}^n a_i(y_i-\mu_i),\qquad u-E[u]=\sum_{j=1}^n b_j(y_j-\mu_j)\]

\(c\) and \(d\) cancel.

\[cov(z,u)=E\Big[\sum_{i=1}^n a_i(y_i-\mu_i)\sum_{j=1}^n b_j(y_j-\mu_j)\Big]\]

\(cov(z,u)=E[(z-E[z])(u-E[u])]\).

\[=E\Big[\sum_{i=1}^n\sum_{j=1}^n a_ib_j(y_i-\mu_i)(y_j-\mu_j)\Big]\]

\[=\sum_{i=1}^n\sum_{j=1}^n a_ib_jE[(y_i-\mu_i)(y_j-\mu_j)]=\sum_{i=1}^n\sum_{j=1}^n a_ib_j\,cov(y_i,y_j)\]

Linearity of expectation.

\[=\sum_{i=1}^n\sum_{j=1}^n a_i\mathbf{V}_{ij}b_j=\mathbf{a}^T\mathbf{V}\mathbf{b}\]

\(\mathbf{V}_{ij}=cov(y_i,y_j)\).

The properties of p.27–29:

\[E[\mathbf{a}]=\left[\begin{array}{c}E[a_1]\\ \vdots\\ E[a_n]\end{array}\right]=\left[\begin{array}{c}a_1\\ \vdots\\ a_n\end{array}\right]=\mathbf{a}\]

The expectation of a constant is the constant.

\[E[\mathbf{a}^T\mathbf{y}+b]=\mathbf{a}^T\boldsymbol\mu+b=\mathbf{a}^TE[\mathbf{y}]+b\]

p.26 with \(c=b\).

\[Var(\mathbf{a}^T\mathbf{y})=cov(\mathbf{a}^T\mathbf{y},\ \mathbf{a}^T\mathbf{y})\]

\[=\mathbf{a}^T\mathbf{V}\mathbf{a}=\mathbf{a}^TVar(\mathbf{y})\mathbf{a}\]

p.26 with \(\mathbf{b}=\mathbf{a}\), \(c=d=0\). ∎

04.6Linear transformations \(\mathbf{z}=\mathbf{A}\mathbf{y}\) Slides p.30–35

What Slides p.30–31

Linear Transformations · Lecture 5 · p.30–31 Matrix Notation: Consider a random vector \(\mathbf{z}=(z_1,\dots,z_k)^T\) of \(k\) linear combinations of random \(\mathbf{y}\): \[\begin{array}{rcl}z_1&=&a_{11}y_1+a_{12}y_2+\cdots a_{1n}y_n\\z_2&=&a_{21}y_1+a_{22}y_2+\cdots a_{2n}y_n\\&\vdots&\\z_k&=&a_{k1}y_1+a_{k2}y_2+\cdots a_{kn}y_n\end{array}\] We can equivalently write \(\mathbf{z}=\mathbf{A}\mathbf{y}\) for \([\mathbf{A}]_{ij}=a_{ij}\). Then: \[E[\mathbf{z}]=\mathbf{A}E[\mathbf{y}]\] \[Var(\mathbf{z})=\mathbf{A}Var(\mathbf{y})\mathbf{A}^T\]

A linear transformation multiplies the random vector by a constant matrix: \(\mathbf{z}=\mathbf{A}\mathbf{y}\). Each \(z_r\) is one linear combination of \(\mathbf{y}\).

\(\mathbf{A}\) is a \(k\times n\) constant matrix. Row \(r\) holds the weights of \(z_r\).

\(\mathbf{z}\) and \(E[\mathbf{z}]\) are \(k\times1\). \(Var(\mathbf{z})\) is \((k\times n)(n\times n)(n\times k)=k\times k\).

With \(k=1\) and \(\mathbf{A}=\mathbf{a}^T\), the formula becomes p.29.

How Added

Calculate \(E[\mathbf{A}\mathbf{y}]\) and \(Var(\mathbf{A}\mathbf{y})\)

Write the weights of \(z_r\) as row \(r\) of \(\mathbf{A}\). Write 0 for a missing \(y_j\).
Mean: calculate \(\mathbf{A}\boldsymbol\mu\) (\(k\times1\)).
Variance: calculate \(\mathbf{V}\mathbf{A}^T\) (\(n\times k\)).
Multiply \(\mathbf{A}\) by the result of step 3 (\(k\times k\)).

Self-check: \(Var(\mathbf{z})\) is symmetric with diagonal \(\ge0\). Diagonal entry \(r\) equals \(Var(z_r)\) from 04.5.

Example · brainhead, \(z_1=y_1\) and \(z_2=y_1-y_2\) Added How steps 1–4

\(k=2\), \(n=2\).

Step 1: \[\mathbf{A}=\left[\begin{array}{cc}1&0\\1&-1\end{array}\right],\qquad \mathbf{z}=\mathbf{A}\mathbf{y}=\left[\begin{array}{c}y_1\\y_1-y_2\end{array}\right]\]

Step 2: \[E[\mathbf{z}]=\mathbf{A}\boldsymbol\mu=\left[\begin{array}{c}1(3637.864)+0(1284.263)\\1(3637.864)-1(1284.263)\end{array}\right]=\left[\begin{array}{c}3637.864\\2353.601\end{array}\right]\]

Step 3: \(\mathbf{A}^T=\left[\begin{array}{cc}1&1\\0&-1\end{array}\right]\). \[\mathbf{V}\mathbf{A}^T=\left[\begin{array}{cc}130413.76(1)+34014.61(0)&130413.76(1)+34014.61(-1)\\34014.61(1)+14084.06(0)&34014.61(1)+14084.06(-1)\end{array}\right]=\left[\begin{array}{cc}130413.76&96399.15\\34014.61&19930.55\end{array}\right]\]

Step 4: Row 1 of \(\mathbf{A}\) is \((1,0)\): copy row 1. Row 2 is \((1,-1)\): row 1 minus row 2. \[\mathbf{A}\mathbf{V}\mathbf{A}^T=\left[\begin{array}{cc}130413.76&96399.15\\130413.76-34014.61&96399.15-19930.55\end{array}\right]=\left[\begin{array}{cc}130413.76&96399.15\\96399.15&76468.60\end{array}\right]\]

Self-check: symmetric (96399.15 twice). \(Var(z_1)=130413.76=var(y_1)\).

Self-check: 04.5 with \(\mathbf{b}=(1,-1)^T\) gives \(\mathbf{b}^T\mathbf{V}\mathbf{b}=(1)(96399.15)+(-1)(19930.55)=76468.60=Var(z_2)\). ▲

Why (1) · \(E[\mathbf{A}\mathbf{y}]=\mathbf{A}E[\mathbf{y}]\) Slides p.32–33

The slides use a square \(n\times n\) \(\mathbf{A}\). The steps are the same for \(k\) rows.

\[E[\mathbf{A}\mathbf{y}]=E\left[\left[\begin{array}{cccc}a_{11}&a_{12}&\dots&a_{1n}\\a_{21}&a_{22}&\dots&a_{2n}\\ \vdots&&&\\a_{n1}&a_{n2}&\dots&a_{nn}\end{array}\right]\left[\begin{array}{c}y_1\\y_2\\ \vdots\\y_n\end{array}\right]\right]\]

\[=E\left[\begin{array}{c}\sum_ia_{1i}y_i\\ \sum_ia_{2i}y_i\\ \vdots\\ \sum_ia_{ni}y_i\end{array}\right]\]

Matrix times vector, row by row.

\[=\left[\begin{array}{c}\sum_ia_{1i}E[y_i]\\ \sum_ia_{2i}E[y_i]\\ \vdots\\ \sum_ia_{ni}E[y_i]\end{array}\right]\]

Entry by entry (p.22); linearity of expectation.

\[=\mathbf{A}E[\mathbf{y}]\]

∎

Why (2) · \(Var(\mathbf{A}\mathbf{y})=\mathbf{A}Var(\mathbf{y})\mathbf{A}^T\) Slides p.34–35

The proof uses \((\mathbf{A}\mathbf{B})^T=\mathbf{B}^T\mathbf{A}^T\) (Lecture 5 · p.11).

Check with \(\mathbf{A}\) from the example and \(\mathbf{B}=\mathbf{y}\): \((\mathbf{A}\mathbf{y})^T=(y_1,\ y_1-y_2)\).

\(\mathbf{y}^T\mathbf{A}^T=(y_1,\ y_2)\left[\begin{array}{cc}1&1\\0&-1\end{array}\right]=(y_1,\ y_1-y_2)\). Equal.

\[Var(\mathbf{A}\mathbf{y})=E[(\mathbf{A}\mathbf{y}-E[\mathbf{A}\mathbf{y}])(\mathbf{A}\mathbf{y}-E[\mathbf{A}\mathbf{y}])^T]\]

p.23 with \(\mathbf{A}\mathbf{y}\) for \(\mathbf{y}\).

\[=E[\mathbf{A}(\mathbf{y}-E[\mathbf{y}])(\mathbf{A}(\mathbf{y}-E[\mathbf{y}]))^T]\]

Why (1); take out \(\mathbf{A}\).

\[=E[\mathbf{A}(\mathbf{y}-E[\mathbf{y}])(\mathbf{y}-E[\mathbf{y}])^T\mathbf{A}^T]\]

\((\mathbf{A}\mathbf{B})^T=\mathbf{B}^T\mathbf{A}^T\), \(\mathbf{B}=\mathbf{y}-E[\mathbf{y}]\).

\[=\mathbf{A}E[(\mathbf{y}-E[\mathbf{y}])(\mathbf{y}-E[\mathbf{y}])^T]\mathbf{A}^T\]

Constant matrices move out of \(E[\cdot]\).

\[=\mathbf{A}Var(\mathbf{y})\mathbf{A}^T\]

p.23. ∎

Later units write the estimator \(\hat{\boldsymbol\beta}\) as a constant matrix times \(\mathbf{y}\) and use these two formulas.

04.7Practice Added

In all three questions, \(\mathbf{y}=(y_1,y_2)^T\), \(\boldsymbol\mu=E[\mathbf{y}]=(3637.864,\ 1284.263)^T\), \(\mathbf{V}=var(\mathbf{y})=\left[\begin{array}{cc}130413.76&34014.61\\34014.61&14084.06\end{array}\right]\).

Q1. Let \(z=y_1+y_2\). Write \(z=\mathbf{a}^T\mathbf{y}\). Calculate \(E[z]\) and \(Var(z)\).

Answer

Step 1: \(\mathbf{a}=(1,\ 1)^T\).

Step 2: \(E[z]=\mathbf{a}^T\boldsymbol\mu=3637.864+1284.263=4922.127\).

Step 3: \(Var(z)=\mathbf{a}^T\mathbf{V}\mathbf{a}=\mathbf{V}_{11}+2\mathbf{V}_{12}+\mathbf{V}_{22}=130413.76+2(34014.61)+14084.06\).

Step 4: \(130413.76+68029.22+14084.06=212527.04\).

Self-check: \(\ge0\).

Q2. Let \(z_1=y_1+y_2\) and \(z_2=y_2\). Write \(\mathbf{z}=\mathbf{A}\mathbf{y}\). Calculate \(E[\mathbf{z}]\), \(Var(\mathbf{z})\) and \(cov(z_1,z_2)\).

Answer

Step 1: \(\mathbf{A}=\left[\begin{array}{cc}1&1\\0&1\end{array}\right]\), \(\mathbf{A}^T=\left[\begin{array}{cc}1&0\\1&1\end{array}\right]\).

Step 2: \(E[\mathbf{z}]=\mathbf{A}\boldsymbol\mu=(3637.864+1284.263,\ 1284.263)^T=(4922.127,\ 1284.263)^T\).

Step 3: \(\mathbf{V}\mathbf{A}^T=\left[\begin{array}{cc}130413.76+34014.61&34014.61\\34014.61+14084.06&14084.06\end{array}\right]=\left[\begin{array}{cc}164428.37&34014.61\\48098.67&14084.06\end{array}\right]\).

Step 4: Row 1 is the sum of the two rows of step 3. Row 2 is row 2 of step 3. \[Var(\mathbf{z})=\left[\begin{array}{cc}164428.37+48098.67&34014.61+14084.06\\48098.67&14084.06\end{array}\right]=\left[\begin{array}{cc}212527.04&48098.67\\48098.67&14084.06\end{array}\right]\]

Step 5: \(cov(z_1,z_2)=[Var(\mathbf{z})]_{12}=48098.67\).

Self-check: Symmetric. \(Var(z_1)=212527.04\) (Q1). \(Var(z_2)=14084.06=var(y_2)\).

Q3. Show that \(Var(\mathbf{a}^T\mathbf{y})=\mathbf{a}^TVar(\mathbf{y})\mathbf{a}\) (p.29) is a special case of \(Var(\mathbf{A}\mathbf{y})=\mathbf{A}Var(\mathbf{y})\mathbf{A}^T\) (p.31).

Answer

\[\text{Let } k=1,\ \mathbf{A}=\mathbf{a}^T\ (1\times n)\]

One linear combination: one row.

\[Var(\mathbf{a}^T\mathbf{y})=\mathbf{a}^TVar(\mathbf{y})(\mathbf{a}^T)^T\]

Substitute into p.31.

\[=\mathbf{a}^TVar(\mathbf{y})\mathbf{a}\]

\((\mathbf{a}^T)^T=\mathbf{a}\).

The result is \(1\times1\), as on p.29. ∎

Unit 05Multivariate normal distribution: definition and properties

All examples use \(\mathbf{A}=\left[\begin{array}{cc}2&0\\1&1\end{array}\right]\) and \(\boldsymbol\mu=\left[\begin{array}{c}1\\2\end{array}\right]\).

05.1Definition: construction from a standard normal vector Slides p.36

What Slides p.36

Multivariate Normal Distribution · Lecture 5 · p.36 Let \(\mathbf{z}=(z_1,\dots,z_n)^T\) be a random vector of i.i.d standard normal random variables, i.e. \(z_i\overset{iid}{\sim}N(0,1)\).
Then \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\) has multivariate normal distribution, i.e.: \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) where

\(E[\mathbf{y}]=\boldsymbol\mu\) and
\(Var(\mathbf{y})=\boldsymbol\Sigma=\mathbf{A}\mathbf{A}^T\).

The standard normal distribution \(N(0,1)\) has mean 0 and variance 1.

\(\mathbf{A}\) is a constant matrix. \(\boldsymbol\mu\) is a constant \(n\times1\) vector.

The multivariate normal distribution (MVN) is the distribution of \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\). In \(MVN(\boldsymbol\mu,\boldsymbol\Sigma)\), \(\boldsymbol\mu\) is the mean vector and \(\boldsymbol\Sigma\) is the covariance matrix.

How Added

Find the distribution of \(\mathbf{y}\) from \(\mathbf{A}\) and \(\boldsymbol\mu\)

Write the mean: \(E[\mathbf{y}]=\boldsymbol\mu\).
Find \(\mathbf{A}^T\).
Find \(\boldsymbol\Sigma=\mathbf{A}\mathbf{A}^T\). Entry \((i,j)\) is row \(i\) times row \(j\) of \(\mathbf{A}\), term by term, added.
Write \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\).

Self-check: \(\boldsymbol\Sigma\) is \(n\times n\) and symmetric, with diagonal \(\ge 0\). \(\boldsymbol\mu\) has length \(n\).

Example Added

\(n=2\):

\[\mathbf{A}=\left[\begin{array}{cc}2&0\\1&1\end{array}\right],\qquad \boldsymbol\mu=\left[\begin{array}{c}1\\2\end{array}\right]\]

Step 1: \(E[\mathbf{y}]=\boldsymbol\mu=(1,2)^T\).

Step 2: \(\mathbf{A}^T=\left[\begin{array}{cc}2&1\\0&1\end{array}\right]\).

Step 3: \(\Sigma_{11}=2\cdot2+0\cdot0=4\). \(\Sigma_{12}=2\cdot1+0\cdot1=2\). \(\Sigma_{21}=1\cdot2+1\cdot0=2\). \(\Sigma_{22}=1\cdot1+1\cdot1=2\).

\(\boldsymbol\Sigma=\left[\begin{array}{cc}4&2\\2&2\end{array}\right]\).

Step 4: \(\mathbf{y}\sim MVN\left(\left[\begin{array}{c}1\\2\end{array}\right],\left[\begin{array}{cc}4&2\\2&2\end{array}\right]\right)\).

Self-check: \(\Sigma_{12}=\Sigma_{21}\). Diagonal 4 and 2 are positive.

A random draw is one value of the random vector from one run of the random process.

Figure 05.1 · Left: 400 random draws of \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\); orange: \(\boldsymbol\mu=(1,2)^T\). The cloud slopes up because \(\Sigma_{12}=2>0\). It is wider than tall because \(\Sigma_{11}=4>\Sigma_{22}=2\). Equal axis scales. Section 05.6 uses the right panel.

Why Added

Two rules from Section 04.6, with a constant vector \(\mathbf{b}\) added:

Rule E: \(E[\mathbf{A}\mathbf{z}+\mathbf{b}]=\mathbf{A}\,E[\mathbf{z}]+\mathbf{b}\).

Rule V: \(Var(\mathbf{A}\mathbf{z}+\mathbf{b})=\mathbf{A}\,Var(\mathbf{z})\,\mathbf{A}^T\). A constant shift does not change the covariance.

\(E[z_i]=0\): \(E[\mathbf{z}]=\mathbf{0}\).

\(Var(z_i)=1\), and \(cov(z_i,z_j)=0\) for \(i\ne j\) (independent): \(Var(\mathbf{z})=\mathbf{I}\).

Mean:

\[E[\mathbf{y}]=E[\mathbf{A}\mathbf{z}+\boldsymbol\mu]\]

\[=\mathbf{A}\,E[\mathbf{z}]+\boldsymbol\mu\]

Rule E with \(\mathbf{b}=\boldsymbol\mu\).

\[=\mathbf{A}\,\mathbf{0}+\boldsymbol\mu\]

\(E[\mathbf{z}]=\mathbf{0}\).

\[=\boldsymbol\mu\]

∎

Covariance matrix:

\[Var(\mathbf{y})=Var(\mathbf{A}\mathbf{z}+\boldsymbol\mu)\]

\[=\mathbf{A}\,Var(\mathbf{z})\,\mathbf{A}^T\]

Rule V.

\[=\mathbf{A}\,\mathbf{I}\,\mathbf{A}^T\]

\(Var(\mathbf{z})=\mathbf{I}\).

\[=\mathbf{A}\mathbf{A}^T=\boldsymbol\Sigma\]

∎

05.2Density function Slides p.36

What Slides p.36

MVN density · Lecture 5 · p.36 \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\), if: \[f(\mathbf{y})=\frac{1}{(2\pi)^{\frac n2}|\boldsymbol\Sigma|^{\frac12}}\exp\left\{-\frac12(\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\right\}\]

The joint density \(f(\mathbf{y})\) is the density of all \(n\) components together. \(f(\mathbf{y})\) times the volume of a small region near \(\mathbf{y}\) is about the probability of that region.

The determinant \(|\boldsymbol\Sigma|\) is one number from a square matrix. For \(2\times2\), it is \(ad-bc\) (02.4).

\(\exp\{x\}=e^x\), with \(e\approx2.71828\).

The quadratic form \((\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\) is one number: the distance from \(\mathbf{y}\) to \(\boldsymbol\mu\), scaled by \(\boldsymbol\Sigma\).

\(\boldsymbol\mu\) and \(\boldsymbol\Sigma\) give the full distribution.

How Added

Calculate \(f(\mathbf{y})\) at a point \(\mathbf{y}\)

Calculate \(\mathbf{y}-\boldsymbol\mu\).
Calculate \(|\boldsymbol\Sigma|\).
Calculate \(\boldsymbol\Sigma^{-1}\).
Calculate \(\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\).
Calculate \(q=(\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\): step 4 times step 1, term by term, added.
Put the values into \(f=\dfrac{1}{(2\pi)^{n/2}|\boldsymbol\Sigma|^{1/2}}e^{-q/2}\).

Self-check: \(|\boldsymbol\Sigma|>0\). \(\boldsymbol\Sigma\boldsymbol\Sigma^{-1}=\mathbf{I}\). \(q\ge0\). At \(\mathbf{y}=\boldsymbol\mu\), \(q=0\) and \(f\) is largest.

Example Added

\(\boldsymbol\mu\), \(\boldsymbol\Sigma\) from 05.1; point \(\mathbf{y}=(2,3)^T\); \(n=2\).

Step 1: \(\mathbf{y}-\boldsymbol\mu=(2-1,\ 3-2)^T=(1,1)^T\).

Step 2: \(|\boldsymbol\Sigma|=4\cdot2-2\cdot2=8-4=4\).

Step 3: \(\boldsymbol\Sigma^{-1}=\frac14\left[\begin{array}{cc}2&-2\\-2&4\end{array}\right]=\left[\begin{array}{cc}0.5&-0.5\\-0.5&1\end{array}\right]\).

Step 4: \(\boldsymbol\Sigma^{-1}(1,1)^T=(0.5-0.5,\ -0.5+1)^T=(0,\ 0.5)^T\).

Step 5: \(q=1\cdot0+1\cdot0.5=0.5\).

Step 6: \((2\pi)^{n/2}=(2\pi)^1=2\pi\). \(|\boldsymbol\Sigma|^{1/2}=4^{1/2}=2\). Denominator: \(2\pi\cdot2=4\pi=12.566371\).

Step 7: \(e^{-q/2}=e^{-0.25}=0.778801\).

Step 8: \(f=\dfrac{0.778801}{12.566371}=0.061975\).

Self-check: \(|\boldsymbol\Sigma|=4>0\), \(q=0.5\ge0\). Entry \((1,1)\) of \(\boldsymbol\Sigma\boldsymbol\Sigma^{-1}\): \(4\cdot0.5+2\cdot(-0.5)=1\).

Why Added

For \(n=1\), the MVN density is the normal density. Let \(\boldsymbol\Sigma=\sigma^2\), \(\mathbf{y}=y\), \(\boldsymbol\mu=\mu\).

\[f(y)=\frac{1}{(2\pi)^{\frac12}|\sigma^2|^{\frac12}}\exp\left\{-\frac12(y-\mu)^T(\sigma^2)^{-1}(y-\mu)\right\}\]

Put in \(n=1\).

\[=\frac{1}{\sqrt{2\pi}\,\sigma}\exp\left\{-\frac12(y-\mu)^T(\sigma^2)^{-1}(y-\mu)\right\}\]

\(1\times1\) determinant: \(|\sigma^2|^{1/2}=\sigma\).

\[=\frac{1}{\sqrt{2\pi}\,\sigma}\exp\left\{-\frac12(y-\mu)\frac{1}{\sigma^2}(y-\mu)\right\}\]

A number equals its transpose.

\[=\frac{1}{\sqrt{2\pi}\,\sigma}\exp\left\{-\frac{(y-\mu)^2}{2\sigma^2}\right\}\]

Density of \(N(\mu,\sigma^2)\). ∎

\(\sigma^2\) becomes \(\boldsymbol\Sigma\); division by \(\sigma^2\) becomes multiplication by \(\boldsymbol\Sigma^{-1}\).

05.3Property 1: linearity Slides p.37

What Slides p.37

Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.37 Linearity: If \(\mathbf{u}=\mathbf{C}\mathbf{y}+\mathbf{d}\), then: \[\mathbf{u}\sim MVN(\mathbf{C}\boldsymbol\mu+\mathbf{d},\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\]

\(\mathbf{C}\) is a constant \(m\times n\) matrix. \(\mathbf{d}\) and \(\mathbf{u}\) are \(m\times1\).

\(\mathbf{u}\) is normal. Rules E and V give its parameters.

How Added

Find the distribution of \(\mathbf{u}=\mathbf{C}\mathbf{y}+\mathbf{d}\)

Write \(\mathbf{C}\) (\(m\times n\)) and \(\mathbf{d}\) (\(m\times1\)).
Calculate the mean \(\mathbf{C}\boldsymbol\mu+\mathbf{d}\).
Calculate \(\mathbf{C}\boldsymbol\Sigma\).
Multiply step 3 on the right by \(\mathbf{C}^T\).
Write \(\mathbf{u}\sim MVN(\mathbf{C}\boldsymbol\mu+\mathbf{d},\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\). For \(m=1\), write \(N(\cdot,\cdot)\).

Self-check: The mean has length \(m\). The covariance matrix is \(m\times m\), symmetric, diagonal \(\ge0\).

Example Added

\(\mathbf{y}\) from 05.1. Let \(\mathbf{C}=(1,\ -1)\) and \(\mathbf{d}=3\): \(u=y_1-y_2+3\).

Step 1: \(\mathbf{C}=(1,-1)\), \(m=1\), \(\mathbf{d}=3\).

Step 2: \(\mathbf{C}\boldsymbol\mu+\mathbf{d}=1\cdot1+(-1)\cdot2+3=2\).

Step 3: \(\mathbf{C}\boldsymbol\Sigma=(1\cdot4-1\cdot2,\ 1\cdot2-1\cdot2)=(2,\ 0)\).

Step 4: \(\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=2\cdot1+0\cdot(-1)=2\).

Step 5: \(u\sim N(2,\ 2)\).

Self-check: variance 2 is positive.

Why Added

\[\mathbf{u}=\mathbf{C}\mathbf{y}+\mathbf{d}\]

\[=\mathbf{C}(\mathbf{A}\mathbf{z}+\boldsymbol\mu)+\mathbf{d}\]

\(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\) (p.36).

\[=(\mathbf{C}\mathbf{A})\mathbf{z}+(\mathbf{C}\boldsymbol\mu+\mathbf{d})\]

Form of p.36: MVN, mean \(\mathbf{C}\boldsymbol\mu+\mathbf{d}\).

\[Var(\mathbf{u})=(\mathbf{C}\mathbf{A})(\mathbf{C}\mathbf{A})^T\]

p.36: matrix times its transpose.

\[=\mathbf{C}\mathbf{A}\mathbf{A}^T\mathbf{C}^T\]

\((\mathbf{C}\mathbf{A})^T=\mathbf{A}^T\mathbf{C}^T\).

\[=\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T\]

\(\mathbf{A}\mathbf{A}^T=\boldsymbol\Sigma\). ∎

05.4Property 2: marginal distribution Slides p.38

What Slides p.38

Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.38 Marginal Distribution: If \(\tilde{\mathbf{y}}=(y_1,\dots,y_m)^T\subset\mathbf{y}\) is a vector subset of \(\mathbf{y}\), then \(\tilde{\mathbf{y}}\) is MVN-distributed.

A marginal distribution is the distribution of some components, with all other components ignored.

A vector subset \(\tilde{\mathbf{y}}\subset\mathbf{y}\) holds \(m\le n\) components of \(\mathbf{y}\). The slides take the first \(m\).

Linearity gives the parameters (Why): the matching entries of \(\boldsymbol\mu\) and the matching rows and columns of \(\boldsymbol\Sigma\).

How Added

Find the distribution of \(\tilde{\mathbf{y}}\)

List the indices to keep, for example \(\{1,\dots,m\}\).
Mean: take these entries of \(\boldsymbol\mu\).
Covariance: take the entries of \(\boldsymbol\Sigma\) where these rows and columns cross.
Write \(\tilde{\mathbf{y}}\sim MVN(\tilde{\boldsymbol\mu},\tilde{\boldsymbol\Sigma})\). For one component, \(y_i\sim N(\mu_i,\Sigma_{ii})\).

Self-check: \(\tilde{\boldsymbol\Sigma}\) is \(m\times m\). Its diagonal matches \(\boldsymbol\Sigma\).

Example Added

\(\mathbf{y}\) from 05.1; keep only \(y_1\) (\(m=1\)).

Step 1: Index set \(\{1\}\).

Step 2: \(\mu_1=1\).

Step 3: \(\Sigma_{11}=4\).

Step 4: \(y_1\sim N(1,4)\). Standard deviation \(\sqrt4=2\).

Density of \(y_1\) at 2 (p.36, \(n=1\), \(\boldsymbol\Sigma=4\)):

Step 5: \(y_1-\mu_1=2-1=1\). \(q=1\cdot\frac14\cdot1=0.25\).

Step 6: \((2\pi)^{1/2}\cdot4^{1/2}=2.506628\cdot2=5.013257\).

Step 7: \(e^{-q/2}=e^{-0.125}=0.882497\).

Step 8: \(f(2)=\dfrac{0.882497}{5.013257}=0.176033\).

Figure 05.2 · Blue: histogram of many random draws of \(y_1\), density scale (frequency divided by bin width; total area 1). Orange: the \(N(1,4)\) density. The shapes agree.

Why Added

Let \(\mathbf{C}=[\,\mathbf{I}_m\ \ \mathbf{0}\,]\) (\(m\times n\): identity on the left, zeros on the right) and \(\mathbf{d}=\mathbf{0}\).

\[\mathbf{C}\mathbf{y}=(y_1,\dots,y_m)^T=\tilde{\mathbf{y}}\]

Row \(i\) of \(\mathbf{C}\) has 1 only in column \(i\).

\[\tilde{\mathbf{y}}\sim MVN(\mathbf{C}\boldsymbol\mu,\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\]

Linearity (p.37).

\[\mathbf{C}\boldsymbol\mu=(\mu_1,\dots,\mu_m)^T\]

\[\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=\text{the top-left }m\times m\text{ block of }\boldsymbol\Sigma\]

\(\mathbf{C}\) keeps rows; \(\mathbf{C}^T\) keeps columns. ∎

In the example, \(\mathbf{C}=(1,0)\): \(\mathbf{C}\boldsymbol\mu=1\), \(\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=\Sigma_{11}=4\).

05.5Property 3: conditional distribution Slides p.39

What Slides p.39

Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.39 Conditional Distribution: If \(\mathbf{y}=(\mathbf{y}_1^T,\mathbf{y}_2^T)^T\), then \(\mathbf{y}_1|\mathbf{y}_2\) is MVN-distributed.

A partition \(\mathbf{y}=(\mathbf{y}_1^T,\mathbf{y}_2^T)^T\) cuts \(\mathbf{y}\) into a top sub-vector \(\mathbf{y}_1\) and a bottom sub-vector \(\mathbf{y}_2\).

The conditional distribution \(\mathbf{y}_1|\mathbf{y}_2\) is the distribution of \(\mathbf{y}_1\) when \(\mathbf{y}_2\) has a known, fixed value. "\(|\)" reads "given".

The slides give no formulas for the conditional mean or covariance.

How Added

Identify the type of a conditional distribution

Confirm that \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\).
State the components of \(\mathbf{y}_1\) and of \(\mathbf{y}_2\).
Conclude that \(\mathbf{y}_1|\mathbf{y}_2\) is MVN (univariate normal if \(\mathbf{y}_1\) has one component).

Self-check: \(\mathbf{y}_1\) and \(\mathbf{y}_2\) together hold all components, with no overlap.

Example Added

Step 1: \(\mathbf{y}\sim MVN\left(\left[\begin{array}{c}1\\2\end{array}\right],\left[\begin{array}{cc}4&2\\2&2\end{array}\right]\right)\) (05.1).

Step 2: \(\mathbf{y}_1=y_1\), \(\mathbf{y}_2=y_2\).

Step 3: By p.39, \(y_1\) given \(y_2=3\) is univariate normal.

Self-check: \(\{y_1\}\) and \(\{y_2\}\) hold both components, with no overlap.

Figure 05.3 · Purple: histogram of \(y_1\) in the random draws with \(y_2\) within 0.05 of 3. Orange: a normal density with the same mean and standard deviation. The histogram is bell-shaped, as p.39 says.

Why Added

No proof on Slides p.39

Slide p.39 gives no proof and no conditional mean or covariance. This unit uses only "\(\mathbf{y}_1|\mathbf{y}_2\) is MVN".

05.6Property 4: independence Slides p.40

What Slides p.40

Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.40 Independence: If \(\Sigma_{ij}=0\), then \(\mathbf{y}_i\) and \(\mathbf{y}_j\) are independent.

\(\Sigma_{ij}=cov(y_i,y_j)\).

Two variables are independent when the value of one does not change the distribution of the other.

With densities, independence means \(f(y_i,y_j)=f_i(y_i)f_j(y_j)\).

This property is true only when \(\mathbf{y}\) is MVN.

How Added

Use \(\boldsymbol\Sigma\) to find independent components

Confirm that the vector is MVN.
For \(\mathbf{C}\mathbf{y}+\mathbf{d}\), find its covariance matrix by linearity.
Read the off-diagonal entry \(\Sigma_{ij}\).
If \(\Sigma_{ij}=0\), components \(i\) and \(j\) are independent.

Self-check: \(\Sigma_{ij}=\Sigma_{ji}=0\).

Example Added

In 05.1, \(\Sigma_{12}=2\ne0\). Make a linear transformation with zero covariance: \(\mathbf{w}=\mathbf{C}_2\mathbf{y}\), \(\mathbf{C}_2=\left[\begin{array}{cc}1&-1\\0&1\end{array}\right]\), \(\mathbf{w}=(y_1-y_2,\ y_2)^T\).

Step 1: By linearity, \(\mathbf{w}\sim MVN(\mathbf{C}_2\boldsymbol\mu,\ \mathbf{C}_2\boldsymbol\Sigma\mathbf{C}_2^T)\).

Step 2: \(\mathbf{C}_2\boldsymbol\mu=(1-2,\ 2)^T=(-1,\ 2)^T\).

Step 3: \(\mathbf{C}_2\boldsymbol\Sigma=\left[\begin{array}{cc}4-2&2-2\\2&2\end{array}\right]=\left[\begin{array}{cc}2&0\\2&2\end{array}\right]\).

Step 4: \(\mathbf{C}_2^T=\left[\begin{array}{cc}1&0\\-1&1\end{array}\right]\).

Step 5: \(\mathbf{C}_2\boldsymbol\Sigma\mathbf{C}_2^T=\left[\begin{array}{cc}2-0&0+0\\2-2&0+2\end{array}\right]=\left[\begin{array}{cc}2&0\\0&2\end{array}\right]\).

Step 6: The off-diagonal entries are 0.

Step 7: \(w_1=y_1-y_2\) and \(w_2=y_2\) are independent.

Figure 05.1, right panel: the same 400 draws as \(\mathbf{w}\). The cloud does not slope. Orange: the mean \((-1,2)^T\).

Why Added

Proof for \(n=2\), with \(\Sigma_{12}=\Sigma_{21}=0\): \(\boldsymbol\Sigma=\left[\begin{array}{cc}\Sigma_{11}&0\\0&\Sigma_{22}\end{array}\right]\).

\[|\boldsymbol\Sigma|=\Sigma_{11}\Sigma_{22}-0\cdot0=\Sigma_{11}\Sigma_{22}\]

\[\boldsymbol\Sigma^{-1}=\frac{1}{\Sigma_{11}\Sigma_{22}}\left[\begin{array}{cc}\Sigma_{22}&0\\0&\Sigma_{11}\end{array}\right]\]

\(2\times2\) inverse formula (02.4).

\[=\left[\begin{array}{cc}1/\Sigma_{11}&0\\0&1/\Sigma_{22}\end{array}\right]\]

\[(\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)=\frac{(y_1-\mu_1)^2}{\Sigma_{11}}+\frac{(y_2-\mu_2)^2}{\Sigma_{22}}\]

\[f(\mathbf{y})=\frac{1}{2\pi(\Sigma_{11}\Sigma_{22})^{1/2}}\exp\left\{-\frac{(y_1-\mu_1)^2}{2\Sigma_{11}}-\frac{(y_2-\mu_2)^2}{2\Sigma_{22}}\right\}\]

p.36 density, \((2\pi)^{n/2}=2\pi\).

\[=\frac{1}{\sqrt{2\pi}\sqrt{\Sigma_{11}}}\exp\left\{-\frac{(y_1-\mu_1)^2}{2\Sigma_{11}}\right\}\cdot\frac{1}{\sqrt{2\pi}\sqrt{\Sigma_{22}}}\exp\left\{-\frac{(y_2-\mu_2)^2}{2\Sigma_{22}}\right\}\]

\(e^{a+b}=e^ae^b\).

\[=f_1(y_1)\,f_2(y_2)\]

Marginal densities (p.38). ∎

05.7Practice Added

Q1. Let \(\mathbf{y}=(y_1,y_2)^T\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) with \(\boldsymbol\mu=\left[\begin{array}{c}1\\2\end{array}\right]\) and \(\boldsymbol\Sigma=\left[\begin{array}{cc}4&2\\2&2\end{array}\right]\). Find the distribution of \(u=2y_1+y_2-1\).

Answer

Step 1: \(u=\mathbf{C}\mathbf{y}+\mathbf{d}\), \(\mathbf{C}=(2,\ 1)\), \(\mathbf{d}=-1\).

Step 2: \(\mathbf{C}\boldsymbol\mu+\mathbf{d}=2\cdot1+1\cdot2-1=3\).

Step 3: \(\mathbf{C}\boldsymbol\Sigma=(2\cdot4+1\cdot2,\ 2\cdot2+1\cdot2)=(10,\ 6)\).

Step 4: \(\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=10\cdot2+6\cdot1=26\).

Step 5: By linearity (p.37), \(u\sim N(3,\ 26)\).

Q2. Let \(\mathbf{y}=(y_1,y_2,y_3)^T\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) with \(\boldsymbol\mu=\left[\begin{array}{c}0\\1\\-1\end{array}\right]\) and \(\boldsymbol\Sigma=\left[\begin{array}{ccc}3&1&0\\1&2&0\\0&0&5\end{array}\right]\). (a) Give the distribution of \(\tilde{\mathbf{y}}=(y_1,y_3)^T\). (b) Which pairs of components are independent by the Independence property?

Answer

(a) Step 1: Index set \(\{1,3\}\).

(a) Step 2: \(\tilde{\boldsymbol\mu}=(\mu_1,\mu_3)^T=(0,\ -1)^T\).

(a) Step 3: Rows and columns 1 and 3: \(\tilde{\boldsymbol\Sigma}=\left[\begin{array}{cc}\Sigma_{11}&\Sigma_{13}\\\Sigma_{31}&\Sigma_{33}\end{array}\right]=\left[\begin{array}{cc}3&0\\0&5\end{array}\right]\).

(a) Step 4: By p.38, \(\tilde{\mathbf{y}}\sim MVN\left(\left[\begin{array}{c}0\\-1\end{array}\right],\left[\begin{array}{cc}3&0\\0&5\end{array}\right]\right)\).

(b) \(\Sigma_{13}=\Sigma_{31}=0\): \(y_1\) and \(y_3\) are independent (p.40).

(b) \(\Sigma_{23}=\Sigma_{32}=0\): \(y_2\) and \(y_3\) are independent (p.40).

(b) \(\Sigma_{12}=1\): the property does not apply to \((y_1,y_2)\).

Q3. Let \(z_1,z_2\overset{iid}{\sim}N(0,1)\), \(\mathbf{A}=\left[\begin{array}{cc}1&0\\1&1\end{array}\right]\), \(\boldsymbol\mu=\left[\begin{array}{c}0\\0\end{array}\right]\), \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\). Find \(\boldsymbol\Sigma\) and \(f(\mathbf{y})\) at \(\mathbf{y}=(1,1)^T\).

Answer

Step 1: \(\boldsymbol\Sigma=\mathbf{A}\mathbf{A}^T\). \(\Sigma_{11}=1\cdot1+0\cdot0=1\). \(\Sigma_{12}=\Sigma_{21}=1\cdot1+0\cdot1=1\). \(\Sigma_{22}=1\cdot1+1\cdot1=2\).

\(\boldsymbol\Sigma=\left[\begin{array}{cc}1&1\\1&2\end{array}\right]\).

Step 2: \(\mathbf{y}-\boldsymbol\mu=(1,1)^T\).

Step 3: \(|\boldsymbol\Sigma|=1\cdot2-1\cdot1=1\).

Step 4: \(\boldsymbol\Sigma^{-1}=\frac11\left[\begin{array}{cc}2&-1\\-1&1\end{array}\right]=\left[\begin{array}{cc}2&-1\\-1&1\end{array}\right]\).

Step 5: \(\boldsymbol\Sigma^{-1}(1,1)^T=(2-1,\ -1+1)^T=(1,\ 0)^T\).

Step 6: \(q=1\cdot1+1\cdot0=1\).

Step 7: Denominator \((2\pi)^1\cdot1^{1/2}=2\pi=6.283185\).

Step 8: \(e^{-q/2}=e^{-0.5}=0.606531\).

Step 9: \(f=\dfrac{0.606531}{6.283185}=0.096532\).

Unit 06Exercises on the MVN, MLR as an MVN model, and practice

Two results from unit 05: the definition (p.36), \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\sim MVN(\boldsymbol\mu,\mathbf{A}\mathbf{A}^T)\) for \(z_i\overset{iid}{\sim}N(0,1)\); and linearity (p.37), \(\mathbf{C}\mathbf{y}+\mathbf{d}\sim MVN(\mathbf{C}\boldsymbol\mu+\mathbf{d},\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\).

06.1Exercise: \(X_2\) has a normal distribution Slides p.41–42

What (1) Slides p.41

Exercise · Lecture 5 · p.41 Let r.v.'s \(X_1\sim N(0,1)\) and \(B\sim Bernoulli(0.5)\), independent of \(X_1\). Now consider r.v. \(X_2=\left\{\begin{array}{ll}X_1 & \text{if } B=0\\ -X_1 & \text{if } B=1\end{array}\right.\)
\(X_2\) is clearly Normally distributed (we've just swapped half the signs to be opposite, and since it's symmetric, the distribution is unchanged.)

"r.v." means random variable.

\(B\sim Bernoulli(0.5)\): \(B\) is 0 or 1, with \(P(B=0)=P(B=1)=0.5\) (one fair coin toss).

Symmetric: the \(N(0,1)\) density has the same shape on each side of 0. \(X_1\) and \(-X_1\) have the same distribution.

\(X_2\) is \(X_1\) with a random sign. The sign change does not change a symmetric distribution.

What (2) · the visual check Slides p.42

A simulation makes many random draws with a computer and shows their pattern.

A density curve is a curve whose area over an interval is the probability of that interval.

Slides p.42 shows a density-scale histogram of the draws of \(X_2\), with the \(N(0,1)\) density curve.

The bars follow the curve. The highest bars are near 0, at height about 0.4.

The \(N(0,1)\) density at 0 is \(1/\sqrt{2\pi}\approx0.3989\).

How Added

A cumulative distribution function (CDF) gives \(P(X\le t)\) for each \(t\). Equal CDFs at all \(t\) mean equal distributions. \(\Phi(t)=P(X_1\le t)\) is the CDF of \(N(0,1)\).

Find \(P(X_2\le t)\) for one value of \(t\)

Divide the event into the cases \(B=0\) and \(B=1\).
In each case, replace \(X_2\) by its rule.
Use independence to write each case as a product of two probabilities.
Change \(P(-X_1\le t)\) to \(P(X_1\ge -t)\), then to \(\Phi(t)\) by symmetry.
Add the two cases.

Self-check: \(P(B=0)+P(B=1)=1\). The result equals \(\Phi(t)\).

Example · \(t=1\) Added

Table value: \(\Phi(1)=0.8413\).

Step 1: \(P(X_2\le1)=P(X_2\le1,\ B=0)+P(X_2\le1,\ B=1)\).

Step 2: \(=P(X_1\le1,\ B=0)+P(-X_1\le1,\ B=1)\).

Step 3: \(=P(X_1\le1)\times0.5+P(-X_1\le1)\times0.5\).

Step 4: \(P(-X_1\le1)=P(X_1\ge-1)=1-\Phi(-1)=1-0.1587=0.8413=\Phi(1)\) (Figure 06.1).

Step 5: \(P(X_2\le1)=0.8413\times0.5+0.8413\times0.5=0.42065+0.42065=0.8413\).

Self-check: \(0.5+0.5=1\); the result equals \(\Phi(1)\).

Figure 06.1 · The \(N(0,1)\) density. Blue: area left of \(-1\), 0.1587. Orange: area right of \(1\), also 0.1587: \(P(X_1\ge-1)=P(X_1\le1)\).

Why Added

\[P(X_2\le t)=P(X_2\le t,\ B=0)+P(X_2\le t,\ B=1)\]

\(B\) is 0 or 1.

\[=P(X_1\le t,\ B=0)+P(-X_1\le t,\ B=1)\]

The rule for \(X_2\).

\[=P(X_1\le t)P(B=0)+P(-X_1\le t)P(B=1)\]

\(B\) and \(X_1\) are independent.

\[=\tfrac12\,\Phi(t)+\tfrac12\,P(-X_1\le t)\]

\[=\tfrac12\,\Phi(t)+\tfrac12\,P(X_1\le t)\]

Symmetry.

\[=\tfrac12\,\Phi(t)+\tfrac12\,\Phi(t)=\Phi(t)\]

\(X_2\sim N(0,1)\). ∎

06.2Exercise continued: \(Y=X_1+X_2\) is not normal Slides p.43–44

What Slides p.43–44

Exercise continued · Lecture 5 · p.43–44 Now let \(Y=X_1+X_2\). Is \(Y\) Normal? No!
How do we know?
• \(P(Y=0)=P(B=1)=1/2\)
What's going on here?
• While \(X_1\) and \(X_2\) are each normally distributed, \(\mathbf{x}=(X_1,X_2)^T\) is not multivariate normal.
• I.e. cannot find \(\mathbf{A}_{2\times k}\) s.t. \(\mathbf{x}=\mathbf{A}\mathbf{z}\) for \(\mathbf{z}=(Z_1,\dots,Z_k)^T\), \(Z_i\overset{iid}{\sim}N(0,1)\)

If \(B=1\), \(X_2=-X_1\) and \(Y=X_1-X_1=0\). This case has probability 1/2.

A normal variable has probability 0 at each single value if its variance is positive, and 1 if its variance is 0 (a constant).

\(P(Y=0)=1/2\) is neither 0 nor 1. \(Y\) is not normal.

\(Y=(1,1)\,\mathbf{x}\) is a linear combination of \(\mathbf{x}\). If \(\mathbf{x}\) were MVN, linearity (p.37) would make \(Y\) normal. \(\mathbf{x}\) is not MVN.

"cannot find \(\mathbf{A}_{2\times k}\)": no constant \(2\times k\) matrix \(\mathbf{A}\) gives \(\mathbf{x}=\mathbf{A}\mathbf{z}\) (p.36).

How Added

Use linearity to show that a vector \(\mathbf{x}\) is not MVN

Select a constant row vector \(\mathbf{C}\). Let \(u=\mathbf{C}\mathbf{x}\), for example \(\mathbf{C}=(1,1)\), \(u=Y\).
Assume that \(\mathbf{x}\) is MVN. Then \(u\) is normal (p.37, \(\mathbf{d}=0\)).
Calculate one probability of \(u\), for example \(P(Y=0)\) over the cases \(B=0\) and \(B=1\).
If the result is neither 0 nor 1, the assumption in step 2 is false.

Self-check: \(\mathbf{C}\) is constant: no \(B\) or \(X_1\) in it. \(P(B=0)+P(B=1)=1\).

Example · Six hand-made values Added

Data: \(X_1=(1.2,\ -0.5,\ 0.8,\ -1.5,\ 2.0,\ 0.3)\) and \(B=(0,\ 1,\ 1,\ 0,\ 1,\ 0)\).

Step 1: \(\mathbf{C}=(1,1)\): \(Y=X_1+X_2\).

Step 2: Assume \(\mathbf{x}\) is MVN. Then \(Y\) is normal.

Step 3a, \(B=0\) (rows 1, 4, 6): \(X_2=X_1\), \(Y=2X_1\): \(2\times1.2=2.4\), \(2\times(-1.5)=-3.0\), \(2\times0.3=0.6\).

Step 3b, \(B=1\) (rows 2, 3, 5): \(X_2=-X_1\): \(0.5\), \(-0.8\), \(-2.0\). \(Y\): \(-0.5+0.5=0\), \(0.8-0.8=0\), \(2.0-2.0=0\).

Step 3c: \(Y=(2.4,\ 0,\ 0,\ -3.0,\ 0,\ 0.6)\). Each row with \(B=1\) gives \(Y=0\) exactly.

Step 4: \(P(Y=0)=P(B=1)=1/2\), neither 0 nor 1. \(\mathbf{x}\) is not MVN.

Figure 06.2 · Distribution of \(Y=X_1+X_2\). Purple bar: probability 1/2 at the single value 0 (\(B=1\)). Blue curve: the spread-out part \(Y=2X_1\) (\(B=0\)). A normal distribution has no probability at a single value.

Why Added

First \(P(Y=0)\), then no matrix \(\mathbf{A}\) exists.

\[P(Y=0)=P(Y=0,\ B=0)+P(Y=0,\ B=1)\]

\(B\) is 0 or 1.

\[=P(X_1+X_1=0,\ B=0)+P(X_1-X_1=0,\ B=1)\]

The rule for \(X_2\).

\[=P(2X_1=0,\ B=0)+P(0=0,\ B=1)\]

\[=P(X_1=0)P(B=0)+P(B=1)\]

Independence; \(0=0\) is always true.

\[=0\times\tfrac12+\tfrac12=\tfrac12\]

\(P(X_1=0)=0\). ∎

\[\text{Assume } \mathbf{x}=\mathbf{A}\mathbf{z},\quad \mathbf{A}_{2\times k},\ Z_i\overset{iid}{\sim}N(0,1)\]

Then \(\mathbf{x}\sim MVN(\mathbf{0},\mathbf{A}\mathbf{A}^T)\) (p.36).

\[Y=(1,1)\,\mathbf{x}\]

\[Y\sim MVN\big((1,1)\mathbf{0},\ (1,1)\mathbf{A}\mathbf{A}^T(1,1)^T\big)\]

Linearity (p.37), \(\mathbf{d}=0\).

\[Y\sim N\big(0,\ (1,1)\mathbf{A}\mathbf{A}^T(1,1)^T\big)\]

One-component MVN is normal.

\[P(Y=0)\in\{0,\ 1\}\]

0 if variance positive; 1 if 0.

\[P(Y=0)=\tfrac12\notin\{0,\ 1\}\]

Contradiction: no \(\mathbf{A}_{2\times k}\) exists. ∎

MVN is a property of the full vector, not of each component. The regression model assumes that the full error vector is MVN.

06.3Multiple linear regression as an MVN model Slides p.45

What Slides p.45

Multiple Linear Regression · Lecture 5 · p.45 \[y_i=\beta_0+\beta_1x_{i1}+\dots+\beta_px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] We can write this: \[\left[\begin{array}{c}y_1\\y_2\\\vdots\\y_n\end{array}\right]=\left[\begin{array}{ccccc}1&x_{11}&x_{12}&\dots&x_{1p}\\1&x_{21}&x_{22}&\dots&x_{2p}\\\vdots&\vdots&\vdots&\vdots&\vdots\\1&x_{n1}&x_{n2}&\dots&x_{np}\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\vdots\\\epsilon_n\end{array}\right]\] Or more simply: \[\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon,\quad \boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\quad\iff\quad \mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\]

Page 45 repeats the matrix form of page 8 (01.6) and adds the MVN statement.

\(\sigma^2\mathbf{I}\) has \(\sigma^2\) on the diagonal and 0 elsewhere: each error has variance \(\sigma^2\); each pair has covariance 0.

\(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\) gives \(\boldsymbol\epsilon=\sigma\mathbf{I}\mathbf{z}\): MVN by p.36. \(\mathbf{y}\) is \(\boldsymbol\epsilon\) plus the constant \(\mathbf{X}\boldsymbol\beta\). By linearity, \(\mathbf{y}\) is MVN with mean \(\mathbf{X}\boldsymbol\beta\) and the same covariance.

How Added

Write the distribution of \(\mathbf{y}\) from the distribution of \(\boldsymbol\epsilon\)

Write \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\).
Write \(\mathbf{y}=\mathbf{C}\boldsymbol\epsilon+\mathbf{d}\), with \(\mathbf{C}=\mathbf{I}\), \(\mathbf{d}=\mathbf{X}\boldsymbol\beta\).
Mean: \(\mathbf{C}\mathbf{0}+\mathbf{d}=\mathbf{X}\boldsymbol\beta\).
Covariance: \(\mathbf{C}(\sigma^2\mathbf{I})\mathbf{C}^T=\mathbf{I}\sigma^2\mathbf{I}\mathbf{I}=\sigma^2\mathbf{I}\).
Write \(\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\).

Self-check: \(\mathbf{d}\) is constant. The mean is \(n\times1\). The covariance is \(n\times n\), \(\sigma^2\) on the diagonal, 0 elsewhere.

Example · A small data set Added

\(n=3\), \(p=2\), \((x_{i1},x_{i2})=(2,1),\ (3,0),\ (5,4)\).

Step 1: \(\left[\begin{array}{c}y_1\\y_2\\y_3\end{array}\right]=\left[\begin{array}{ccc}1&2&1\\1&3&0\\1&5&4\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\beta_2\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\), \(\boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\).

Step 2: \(\mathbf{C}=\mathbf{I}\) (\(3\times3\)), \(\mathbf{d}=\mathbf{X}\boldsymbol\beta\).

Step 3: \(\mathbf{X}\boldsymbol\beta=\left[\begin{array}{c}\beta_0+2\beta_1+1\beta_2\\\beta_0+3\beta_1+0\beta_2\\\beta_0+5\beta_1+4\beta_2\end{array}\right]=\left[\begin{array}{c}\beta_0+2\beta_1+\beta_2\\\beta_0+3\beta_1\\\beta_0+5\beta_1+4\beta_2\end{array}\right]\).

Step 4: \(\sigma^2\mathbf{I}=\left[\begin{array}{ccc}\sigma^2&0&0\\0&\sigma^2&0\\0&0&\sigma^2\end{array}\right]\).

Step 5: \(\left[\begin{array}{c}y_1\\y_2\\y_3\end{array}\right]\sim MVN\left(\left[\begin{array}{c}\beta_0+2\beta_1+\beta_2\\\beta_0+3\beta_1\\\beta_0+5\beta_1+4\beta_2\end{array}\right],\ \left[\begin{array}{ccc}\sigma^2&0&0\\0&\sigma^2&0\\0&0&\sigma^2\end{array}\right]\right)\).

Self-check: mean \(3\times1\), covariance \(3\times3\).

Self-check: the marginal property (p.38) keeps only \(y_1\):

\(y_1\sim N(\beta_0+2\beta_1+\beta_2,\ \sigma^2)\), the model for one observation.

Why Added

Lines 1–3 give "⟹"; lines 4–6 give "⟸". Both use linearity (p.37).

\[\mathbf{y}=\mathbf{I}\boldsymbol\epsilon+\mathbf{X}\boldsymbol\beta,\quad \boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\]

\[\mathbf{y}\sim MVN\big(\mathbf{I}\mathbf{0}+\mathbf{X}\boldsymbol\beta,\ \mathbf{I}(\sigma^2\mathbf{I})\mathbf{I}^T\big)\]

Linearity, \(\mathbf{C}=\mathbf{I}\), \(\mathbf{d}=\mathbf{X}\boldsymbol\beta\).

\[\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\ \sigma^2\mathbf{I})\]

\(\mathbf{I}\mathbf{0}=\mathbf{0}\), \(\mathbf{I}^T=\mathbf{I}\), \(\mathbf{I}\mathbf{I}=\mathbf{I}\). ⟹

\[\boldsymbol\epsilon=\mathbf{I}\mathbf{y}-\mathbf{X}\boldsymbol\beta,\quad \mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\]

\[\boldsymbol\epsilon\sim MVN\big(\mathbf{I}\mathbf{X}\boldsymbol\beta-\mathbf{X}\boldsymbol\beta,\ \mathbf{I}(\sigma^2\mathbf{I})\mathbf{I}^T\big)\]

Linearity, \(\mathbf{C}=\mathbf{I}\), \(\mathbf{d}=-\mathbf{X}\boldsymbol\beta\).

\[\boldsymbol\epsilon\sim MVN(\mathbf{0},\ \sigma^2\mathbf{I})\]

⟸ ∎

Later lectures use \(\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\) to find the distribution of \(\hat{\boldsymbol\beta}\).

06.4Practice Q1: mean vectors and covariance matrices Slides p.46–47

Slides p.46 is the title page "Practice". Slides p.47–48 give the two questions.

What Slides p.47

Practice Q1 · Lecture 5 · p.47 Let \(\mathbf{y}=(y_1,y_2,y_3)^T\) such that \(E[\mathbf{y}]=(1,2,2)^T\) and \(Var(\mathbf{y})=\left[\begin{array}{ccc}4&0&1\\0&3&-1\\1&-1&4\end{array}\right]\) and let \(\mathbf{a}=(1,2,3)^T\) and \(\mathbf{A}=\left[\begin{array}{ccc}1&-1&0\\0&1&-1\\1&0&-1\\1&1&1\end{array}\right]\). Compute \(E[\mathbf{a}^T\mathbf{y}]\), \(E[\mathbf{A}\mathbf{y}]\), \(Var(\mathbf{a}^T\mathbf{y})\), \(Var(\mathbf{A}\mathbf{y})\)

\(\mathbf{a}^T\mathbf{y}=y_1+2y_2+3y_3\) is one number. \(\mathbf{A}\mathbf{y}\) is \(4\times1\).

Formulas: \(E[\mathbf{a}^T\mathbf{y}+b]=\mathbf{a}^TE[\mathbf{y}]+b\), \(Var(\mathbf{a}^T\mathbf{y})=\mathbf{a}^TVar(\mathbf{y})\mathbf{a}\) (p.28–29); \(E[\mathbf{A}\mathbf{y}]=\mathbf{A}E[\mathbf{y}]\), \(Var(\mathbf{A}\mathbf{y})=\mathbf{A}Var(\mathbf{y})\mathbf{A}^T\) (p.31).

How Added

Let \(\mathbf{V}=Var(\mathbf{y})\).

Practice Q1

\(E[\mathbf{a}^T\mathbf{y}]\): calculate \(\mathbf{a}^TE[\mathbf{y}]\) (\(b=0\)).
\(E[\mathbf{A}\mathbf{y}]\): do step 1 with each row of \(\mathbf{A}\).
\(Var(\mathbf{a}^T\mathbf{y})\): calculate \(\mathbf{V}\mathbf{a}\) (\(3\times1\)).
Calculate \(\mathbf{a}^T(\mathbf{V}\mathbf{a})\).
\(Var(\mathbf{A}\mathbf{y})\): calculate \(\mathbf{A}\mathbf{V}\) (\(4\times3\)).
Row \(i\) of \(\mathbf{A}\mathbf{V}\): add the rows of \(\mathbf{V}\), with row \(i\) of \(\mathbf{A}\) as weights.
Calculate \((\mathbf{A}\mathbf{V})\mathbf{A}^T\) (\(4\times4\)).
Entry \((i,j)\): row \(i\) of \(\mathbf{A}\mathbf{V}\) times row \(j\) of \(\mathbf{A}\), term by term, added.

Self-check: \(Var(\mathbf{a}^T\mathbf{y})\ge0\) (p.25). \(Var(\mathbf{A}\mathbf{y})\) is symmetric (p.24) with diagonal \(\ge0\).

Example · the full calculation Added

Answer: Practice Q1

Step 1: \(E[\mathbf{a}^T\mathbf{y}]=\mathbf{a}^TE[\mathbf{y}]=1\times1+2\times2+3\times2=1+4+6=11\).

Step 2, row 1: \(1\times1+(-1)\times2+0\times2=1-2+0=-1\).

Step 2, row 2: \(0\times1+1\times2+(-1)\times2=0+2-2=0\).

Step 2, row 3: \(1\times1+0\times2+(-1)\times2=1+0-2=-1\).

Step 2, row 4: \(1\times1+1\times2+1\times2=1+2+2=5\). \(E[\mathbf{A}\mathbf{y}]=(-1,\ 0,\ -1,\ 5)^T\).

Step 3: \(\mathbf{V}\mathbf{a}=\left[\begin{array}{c}4\times1+0\times2+1\times3\\0\times1+3\times2+(-1)\times3\\1\times1+(-1)\times2+4\times3\end{array}\right]=\left[\begin{array}{c}4+0+3\\0+6-3\\1-2+12\end{array}\right]=\left[\begin{array}{c}7\\3\\11\end{array}\right]\).

Step 4: \(Var(\mathbf{a}^T\mathbf{y})=\mathbf{a}^T\mathbf{V}\mathbf{a}=1\times7+2\times3+3\times11=7+6+33=46\).

Step 5: Rows of \(\mathbf{V}\): \(\mathbf{v}_1=(4,0,1)\), \(\mathbf{v}_2=(0,3,-1)\), \(\mathbf{v}_3=(1,-1,4)\).

Step 6, row \((1,-1,0)\): \(\mathbf{v}_1-\mathbf{v}_2=(4-0,\ 0-3,\ 1+1)=(4,-3,2)\).

Step 6, row \((0,1,-1)\): \(\mathbf{v}_2-\mathbf{v}_3=(0-1,\ 3+1,\ -1-4)=(-1,4,-5)\).

Step 6, row \((1,0,-1)\): \(\mathbf{v}_1-\mathbf{v}_3=(4-1,\ 0+1,\ 1-4)=(3,1,-3)\).

Step 6, row \((1,1,1)\): \(\mathbf{v}_1+\mathbf{v}_2+\mathbf{v}_3=(4+0+1,\ 0+3-1,\ 1-1+4)=(5,2,4)\).

Steps 7–8, row \((4,-3,2)\) with the rows of \(\mathbf{A}\): \(4+3+0=7\); \(0-3-2=-5\); \(4+0-2=2\); \(4-3+2=3\).

Steps 7–8, row \((-1,4,-5)\): \(-1-4+0=-5\); \(0+4+5=9\); \(-1+0+5=4\); \(-1+4-5=-2\).

Steps 7–8, row \((3,1,-3)\): \(3-1+0=2\); \(0+1+3=4\); \(3+0+3=6\); \(3+1-3=1\).

Steps 7–8, row \((5,2,4)\): \(5-2+0=3\); \(0+2-4=-2\); \(5+0-4=1\); \(5+2+4=11\).

\[Var(\mathbf{A}\mathbf{y})=\left[\begin{array}{cccc}7&-5&2&3\\-5&9&4&-2\\2&4&6&1\\3&-2&1&11\end{array}\right]\]

Self-check: \(46\ge0\). Symmetric: entries \((1,2)\) and \((2,1)\) are both \(-5\).

Self-check: diagonal 7, 9, 6, 11 are all \(\ge0\).

Why Slides p.34–35

Section 04.6, Why (2), derives \(Var(\mathbf{A}\mathbf{y})\). \(Var(\mathbf{a}^T\mathbf{y})\) is the case \(\mathbf{A}=\mathbf{a}^T\) (04.7, Q3).

06.5Practice Q2: the cross-covariance matrix Slides p.48

What Slides p.48

Practice Q2 · Lecture 5 · p.48 Suppose \(\mathbf{x}=(x_1,\dots,x_n)^T\) is a random vector with \(E[\mathbf{x}]=\boldsymbol\mu\) and \(var(\mathbf{x})=\mathbf{V}\). Let \(\mathbf{y}=\mathbf{A}_{p\times n}\mathbf{x}\) and \(\mathbf{w}=\mathbf{B}_{q\times n}\mathbf{x}\). Let \(\mathbf{C}\) denote the "cross-covariance" matrix, where \(\mathbf{C}_{ij}=cov(y_i,w_j)\). (You can think of this as the off-diagonal block of the variance matrix of \((\mathbf{y}^T,\mathbf{w}^T)^T\), where the diagonal blocks would be \(var(\mathbf{y})\) and \(var(\mathbf{w})\).) Write \(\mathbf{C}\) in terms of \(\boldsymbol\mu,\mathbf{V},\mathbf{A},\mathbf{B}\)

The cross-covariance matrix \(\mathbf{C}\) (\(p\times q\)) has entry \((i,j)\) equal to \(cov(y_i,w_j)\).

Stack \(\mathbf{y}\) above \(\mathbf{w}\). The covariance matrix of this \((p+q)\times1\) vector has four blocks: \(var(\mathbf{y})\) top-left, \(var(\mathbf{w})\) bottom-right, \(\mathbf{C}\) top-right.

Formula from p.26, with \(\mathbf{x}\) as the random vector: \(cov(\mathbf{a}^T\mathbf{x}+c,\ \mathbf{b}^T\mathbf{x}+d)=\mathbf{a}^T\mathbf{V}\mathbf{b}\).

How Added

Practice Q2

Let \(\mathbf{a}_i^T\) be row \(i\) of \(\mathbf{A}\) and \(\mathbf{b}_j^T\) row \(j\) of \(\mathbf{B}\).
Write \(y_i=\mathbf{a}_i^T\mathbf{x}\) and \(w_j=\mathbf{b}_j^T\mathbf{x}\).
Use p.26 with \(c=d=0\): \(\mathbf{C}_{ij}=\mathbf{a}_i^T\mathbf{V}\mathbf{b}_j\).
\(\mathbf{b}_j\) is column \(j\) of \(\mathbf{B}^T\).
Write \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{B}^T\).

Self-check: \((p\times n)(n\times n)(n\times q)=p\times q\). No \(\boldsymbol\mu\). If \(\mathbf{B}=\mathbf{A}\), the answer is \(\mathbf{A}\mathbf{V}\mathbf{A}^T\).

Example · The numbers of Q1 Added

Use the random vector and \(\mathbf{V}\) of Q1 as \(\mathbf{x}\) (\(n=3\)), the \(4\times3\) \(\mathbf{A}\) of Q1 (\(p=4\)), and \(\mathbf{B}=\mathbf{a}^T=(1,2,3)\) (\(q=1\)).

Answer: Practice Q2 and a numerical example

General answer: \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{B}^T\). No \(\boldsymbol\mu\).

Step 1: \(\mathbf{B}^T=\mathbf{a}=(1,2,3)^T\): \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{a}\) (\(4\times1\)).

Step 2: Q1 step 3: \(\mathbf{V}\mathbf{a}=(7,3,11)^T\).

Step 3, row \((1,-1,0)\): \(7-3+0=4\).

Step 3, row \((0,1,-1)\): \(0+3-11=-8\).

Step 3, row \((1,0,-1)\): \(7+0-11=-4\).

Step 3, row \((1,1,1)\): \(7+3+11=21\). \(\mathbf{C}=(4,\ -8,\ -4,\ 21)^T\).

Step 4: Stack \(\mathbf{A}\) above \(\mathbf{a}^T\): a \(5\times3\) matrix \(\mathbf{S}\). \(\mathbf{S}\mathbf{V}\mathbf{S}^T\) is the covariance matrix of \((\mathbf{y}^T,w)^T\).

\[\mathbf{S}\mathbf{V}\mathbf{S}^T=\left[\begin{array}{ccccc}7&-5&2&3&4\\-5&9&4&-2&-8\\2&4&6&1&-4\\3&-2&1&11&21\\4&-8&-4&21&46\end{array}\right]\]

Top-left \(4\times4\) block: \(Var(\mathbf{A}\mathbf{y})\) (Q1). Bottom-right 46: \(Var(\mathbf{a}^T\mathbf{y})\) (Q1).

Column 5, first four entries: \((4,-8,-4,21)=\mathbf{C}\).

Self-check: size \(4\times1=p\times q\).

Why · \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{B}^T\) Added

First by entries, then by matrices.

\[\mathbf{C}_{ij}=cov(y_i,w_j)\]

\[=cov(\mathbf{a}_i^T\mathbf{x},\ \mathbf{b}_j^T\mathbf{x})\]

Rows of \(\mathbf{A}\) and \(\mathbf{B}\).

\[=\mathbf{a}_i^T\mathbf{V}\mathbf{b}_j\]

p.26, \(c=d=0\).

\[=\sum_{k=1}^n\sum_{l=1}^n A_{ik}V_{kl}B_{jl}\]

\[=\sum_{k=1}^n\sum_{l=1}^n A_{ik}V_{kl}(B^T)_{lj}\]

\(B_{jl}=(B^T)_{lj}\).

\[=[\mathbf{A}\mathbf{V}\mathbf{B}^T]_{ij}\]

Matrix product, all \(i,j\). ∎

\[\mathbf{C}=E[(\mathbf{y}-E[\mathbf{y}])(\mathbf{w}-E[\mathbf{w}])^T]\]

Entry \((i,j)\) is \(cov(y_i,w_j)\).

\[=E[(\mathbf{A}\mathbf{x}-\mathbf{A}\boldsymbol\mu)(\mathbf{B}\mathbf{x}-\mathbf{B}\boldsymbol\mu)^T]\]

\(E[\mathbf{A}\mathbf{x}]=\mathbf{A}\boldsymbol\mu\) (p.31).

\[=E[\mathbf{A}(\mathbf{x}-\boldsymbol\mu)(\mathbf{x}-\boldsymbol\mu)^T\mathbf{B}^T]\]

\[=\mathbf{A}E[(\mathbf{x}-\boldsymbol\mu)(\mathbf{x}-\boldsymbol\mu)^T]\mathbf{B}^T\]

\[=\mathbf{A}\mathbf{V}\mathbf{B}^T\]

Line 3 removes \(\boldsymbol\mu\). ∎

06.6Practice Added

P1. Use the data of Section 06.3: \((x_{i1},x_{i2})=(2,1),\ (3,0),\ (5,4)\). Assume \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\). Find the distribution of \(y_1-y_2\).

Answer

Step 1: By 06.3, \(\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\), \(\mathbf{X}\boldsymbol\beta=(\beta_0+2\beta_1+\beta_2,\ \beta_0+3\beta_1,\ \beta_0+5\beta_1+4\beta_2)^T\).

Step 2: \(y_1-y_2=\mathbf{C}\mathbf{y}\), \(\mathbf{C}=(1,-1,0)\), \(\mathbf{d}=0\).

Step 3 (mean): \(\mathbf{C}\mathbf{X}\boldsymbol\beta=(\beta_0+2\beta_1+\beta_2)-(\beta_0+3\beta_1)=(2-3)\beta_1+\beta_2=-\beta_1+\beta_2\).

Step 4 (variance): \(\mathbf{C}(\sigma^2\mathbf{I})\mathbf{C}^T=\sigma^2(1^2+(-1)^2+0^2)=\sigma^2(1+1+0)=2\sigma^2\).

Result: \(y_1-y_2\sim N(-\beta_1+\beta_2,\ 2\sigma^2)\) (p.37). \(\beta_0\) cancels.

P2. Use \(\mathbf{y}\), \(E[\mathbf{y}]\), \(Var(\mathbf{y})\) from Practice Q1. Let \(u=2y_1-y_2+1\). Calculate \(E[u]\), \(Var(u)\), \(cov(u,y_3)\).

Answer

Step 1: \(u=\mathbf{b}^T\mathbf{y}+1\), \(\mathbf{b}=(2,-1,0)^T\).

Step 2: \(E[u]=\mathbf{b}^T\boldsymbol\mu+1=2\times1+(-1)\times2+0\times2+1=2-2+0+1=1\).

Step 3: \(\mathbf{V}\mathbf{b}=(4\times2+0\times(-1)+1\times0,\ 0\times2+3\times(-1)+(-1)\times0,\ 1\times2+(-1)\times(-1)+4\times0)^T=(8,-3,3)^T\).

Step 4: \(Var(u)=\mathbf{b}^T\mathbf{V}\mathbf{b}=2\times8+(-1)\times(-3)+0\times3=16+3+0=19\). The constant 1 does not change the variance.

Step 5: \(y_3=\mathbf{e}_3^T\mathbf{y}\), \(\mathbf{e}_3=(0,0,1)^T\). \(cov(u,y_3)=\mathbf{b}^T\mathbf{V}\mathbf{e}_3\) (p.26).

Step 6: \(\mathbf{V}\) is symmetric: \(\mathbf{b}^T\mathbf{V}\mathbf{e}_3\) is component 3 of \(\mathbf{V}\mathbf{b}\). \(cov(u,y_3)=3\).