STAT 331 · Lecture 5 · Random Vectors and Multivariate Normal # Lecture 5: Random Vectors and Multivariate Normal This lecture writes regression in matrix form, reviews matrix rules, and defines random vectors and the multivariate normal distribution (Lecture 5 · p.1–48). Contents 01 · Recap and multiple linear regression in matrix form 02 · Matrix review: transpose, inverse, trace 03 · Matrix calculus 04 · Random vectors: mean vector, covariance matrix, and linear transformations 05 · Multivariate normal distribution: definition and properties 06 · Exercises on the MVN, MLR as an MVN model, and practice ## 01 · Recap and multiple linear regression in matrix form ### 01.1 Title and section pages Slides p.1–2 Page 1: "Lecture 5: Random Vectors & Multivariate normal". Page 2: section title "Recap" (pages 3–5). ### 01.2 Recap: simple linear regression model Slides p.3 **What** Slides p.3 Recap: Simple Linear Regression · Lecture 5 · p.3 \[y_i=\beta_0+\beta_1x_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] Or: \[y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\] Simple linear regression is a linear model with one explanatory variable. The response \(y_i\) is the quantity to explain in observation \(i\), \(i=1,\dots,n\). The sample size \(n\) is the number of observations. The explanatory variable (covariate) \(x_i\) explains \(y_i\). The model treats \(x_i\) as a known, fixed number. The intercept \(\beta_0\) and the slope \(\beta_1\) are parameters: fixed, unknown numbers of the population. The error \(\epsilon_i\) is the random distance of \(y_i\) from the line \(\beta_0+\beta_1x_i\). A random variable is a number whose value comes from a random result. The expectation (mean) \(E[\cdot]\) is the theoretical average of a random variable. The variance \(Var(\cdot)\) is the expected squared distance from the mean. It measures spread. \(N(\mu,\sigma^2)\) is the bell-shaped normal distribution with mean \(\mu\) and variance \(\sigma^2\). The standard deviation \(\sigma\) is the square root of the variance. \(\overset{iid}{\sim}\) means "independent and identically distributed": the errors do not affect each other and have one distribution. \(\overset{indep}{\sim}\) means "independent". The \(y_i\) have different means, because the mean changes with \(x_i\). The first form is "line plus error". The second form gives the distribution of each \(y_i\). **How** Added Write a data set as a simple linear regression model - Find the response \(y\). - Find the explanatory variable \(x\). - Count the observations \(n\). - Write \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\) and \(i=1,\dots,n\). **Self-check:** The unknown parameters are \(\beta_0\), \(\beta_1\) and \(\sigma^2\). The data give \(x_i\), \(y_i\) and \(n\). **Example · brainhead data** Added The course file brainhead.csv gives the head size and brain weight of 236 persons. All examples in this unit use it. The first 3 rows: Row 1: head size 4512 cm³, brain weight 1530 g. Row 2: head size 3738 cm³, brain weight 1297 g. Row 3: head size 4261 cm³, brain weight 1335 g. 1 Response How step 1 \[y_i=\text{brain weight of person } i\text{ (g)}\] 2 Explanatory variable How step 2 \[x_i=\text{head size of person } i\text{ (cm³)}\] 3 Sample size How step 3 \[n=236\] 4 Model How step 4 \[y_i=\beta_0+\beta_1x_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\qquad i=1,\dots,236\] Lecture 4 · p.7 gives \(\hat\beta_0=335.43231\), \(\hat\beta_1=0.26082\) and \(\hat\sigma=72.35\). A hat \(\hat{\ }\) marks an estimate: a number calculated from the data. The examples use more digits, calculated from brainhead.csv (Added, not on the slides): \(\hat\beta_0=335.4323\), \(\hat\beta_1=0.2608207\), \(\hat\sigma=72.35066\), \(\bar x=3637.864\), \(\bar y=1284.263\), \(\sum_{i=1}^n(x_i-\bar x)^2=30647233.66\). [figure] Figure 01.1 · Brainhead data (\(n=236\)) and the fitted line \(\hat y=335.43+0.2608x\) (green). Orange: the first 3 observations. Each orange dashed line is a residual (vertical distance to the line). **Why · the two forms are the same** Added 1 \[y_i=\beta_0+\beta_1x_i+\epsilon_i\] 2 \[E[y_i]=\beta_0+\beta_1x_i+E[\epsilon_i]\] \(\beta_0+\beta_1x_i\) is a constant. 3 \[E[y_i]=\beta_0+\beta_1x_i\] \(E[\epsilon_i]=0\). 4 \[Var(y_i)=Var(\epsilon_i)=\sigma^2\] A constant does not change the variance. 5 \[y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\] Normal plus constant is normal; independent \(\epsilon_i\). ∎ ### 01.3 Recap: estimating the mean response Slides p.4 **What** Slides p.4 Estimating the Mean Response · Lecture 5 · p.4 \[\hat\mu_0=\hat\beta_0+\hat\beta_1x_0=\bar y+\hat\beta_1(x_0-\bar x)\] \[\frac{\hat\mu_0-\mu_0}{\hat\sigma\sqrt{\dfrac1n+\dfrac{(x_0-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}}\sim t_{n-2}\] 100(1-α)% CI is: \[\hat\mu_0\pm t_{1-\alpha/2,\,n-2}\,\hat\sigma\sqrt{\frac1n+\frac{(x_0-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}\] The mean response \(\mu_0=\beta_0+\beta_1x_0\) is the population mean of \(y\) at \(x=x_0\). It is an unknown parameter. \(\hat\mu_0\) is the estimate of \(\mu_0\). The sample means \(\bar x=\frac1n\sum_{i=1}^nx_i\) and \(\bar y\) are the averages of the \(x_i\) and the \(y_i\). \(\hat\sigma=\sqrt{\sum_ie_i^2/(n-2)}\) estimates \(\sigma\). The residual \(e_i=y_i-\hat y_i\) is the observed value minus the fitted value. \(t_{n-2}\) is the t distribution with \(n-2\) degrees of freedom. It is bell-shaped like \(N(0,1)\), with thicker tails. \(t_{1-\alpha/2,\,n-2}\) is the \(1-\alpha/2\) quantile of \(t_{n-2}\): the point with area \(1-\alpha/2\) to its left. A confidence interval (CI) is calculated from the data. In repeated samples, \(100(1-\alpha)\%\) of these intervals contain the true \(\mu_0\). **How** Added Calculate the \(100(1-\alpha)\%\) CI for \(\mu_0\) - Calculate \(\hat\mu_0=\hat\beta_0+\hat\beta_1x_0\). - Calculate \(\dfrac1n+\dfrac{(x_0-\bar x)^2}{\sum_i(x_i-\bar x)^2}\). - Take the square root of step 2. Multiply it by \(\hat\sigma\). - Find the quantile \(t_{1-\alpha/2,\,n-2}\). - Multiply step 4 by step 3. This is the half-width. - Write \(\hat\mu_0\pm\) the half-width. **Self-check:** The midpoint equals \(\hat\mu_0\). The interval is wider when \(x_0\) is farther from \(\bar x\). **Example · brainhead data, \(x_0=4000\) cm³** Added Find the 95% CI (\(\alpha=0.05\)) for the mean brain weight at head size 4000 cm³. 1 Point estimate How step 1 \[\hat\mu_0=335.4323+0.2608207\times4000\] \[\hat\mu_0=335.4323+1043.2828=1378.7151\] 2 Term in the square root How step 2 \[x_0-\bar x=4000-3637.864=362.136\] \[(x_0-\bar x)^2=362.136^2=131142.48\] \[\frac{131142.48}{30647233.66}=0.0042791\] \[\frac1{236}=0.0042373\] \[0.0042373+0.0042791=0.0085164\] 3 Standard error How step 3 \[\sqrt{0.0085164}=0.0922843\] \[72.35066\times0.0922843=6.67683\] 4 Quantile How step 4 \[t_{1-0.05/2,\,236-2}=t_{0.975,\,234}=1.970154\] 5 Half-width How step 5 \[1.970154\times6.67683=13.15438\] 6 Interval How step 6 \[1378.7151\pm13.1544=(1378.7151-13.1544,\ \ 1378.7151+13.1544)=(1365.5607,\ \ 1391.8695)\] **Self-check:** \((1365.5607+1391.8695)/2=1378.7151=\hat\mu_0\). Result: 95% CI for the mean brain weight at 4000 cm³ is (1365.56, 1391.87) g. ▲ **Why · the second form is correct** Added 1 \[\hat\mu_0=\hat\beta_0+\hat\beta_1x_0\] 2 \[\hat\mu_0=\bar y-\hat\beta_1\bar x+\hat\beta_1x_0\] Put in \(\hat\beta_0=\bar y-\hat\beta_1\bar x\). 3 \[\hat\mu_0=\bar y+\hat\beta_1(x_0-\bar x)\] ∎ ### 01.4 Recap: predicting a new response Slides p.5 **What** Slides p.5 Predicting a New Response · Lecture 5 · p.5 \[\hat y_{new}=\hat\beta_0+\hat\beta_1x_{new}\] \[\frac{\hat y_{new}-y_{new}}{\hat\sigma\sqrt{1+\dfrac1n+\dfrac{(x_{new}-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}}\sim t_{n-2}\] 100(1-α)% prediction interval is: \[\hat y_{new}\pm t_{1-\alpha/2,\,n-2}\,\hat\sigma\sqrt{1+\frac1n+\frac{(x_{new}-\bar x)^2}{\sum_{i=1}^n(x_i-\bar x)^2}}\] The new response \(y_{new}=\beta_0+\beta_1x_{new}+\epsilon_{new}\) is the response of a new, unobserved person. It is a random variable, not a parameter. \(\hat y_{new}\) is the prediction of \(y_{new}\). It uses the same equation as \(\hat\mu_0\). A prediction interval (PI) contains \(y_{new}\) in \(100(1-\alpha)\%\) of repeated samples. **How** Added Calculate the \(100(1-\alpha)\%\) PI for \(y_{new}\) - Calculate \(\hat y_{new}=\hat\beta_0+\hat\beta_1x_{new}\). - Calculate \(1+\dfrac1n+\dfrac{(x_{new}-\bar x)^2}{\sum_i(x_i-\bar x)^2}\). - Take the square root of step 2. Multiply it by \(\hat\sigma\). - Find the quantile \(t_{1-\alpha/2,\,n-2}\). - Multiply step 4 by step 3. This is the half-width. - Write \(\hat y_{new}\pm\) the half-width. **Self-check:** At one point, the PI and the CI have the same midpoint. The PI is always wider. **Example · brainhead data, \(x_{new}=4000\) cm³** Added Find the 95% PI for the brain weight of one new person with head size 4000 cm³. 1 Prediction How step 1 \[\hat y_{new}=335.4323+0.2608207\times4000=1378.7151\] 2 Term in the square root How step 2 \[1+0.0085164=1.0085164\] 0.0085164 from Section 01.3, step 2. 3 Standard error How step 3 \[\sqrt{1.0085164}=1.0042492\] \[72.35066\times1.0042492=72.65809\] 4 Quantile How step 4 \[t_{0.975,\,234}=1.970154\] 5 Half-width How step 5 \[1.970154\times72.65809=143.14763\] 6 Interval How step 6 \[1378.7151\pm143.1476=(1378.7151-143.1476,\ \ 1378.7151+143.1476)=(1235.5675,\ \ 1521.8627)\] **Self-check:** Midpoint 1378.7151, same as the CI. Half-width is \(143.1476/13.1544=10.88\) times the CI half-width. Result: 95% PI for one new person at 4000 cm³ is (1235.57, 1521.86) g. ▲ **Why · the square root has an extra 1** Added The prediction error \(\hat y_{new}-y_{new}\) has two independent sources: the estimated line and the new error \(\epsilon_{new}\). 1 \[Var(\hat y_{new}-y_{new})=Var(\hat y_{new})+Var(y_{new})\] \(y_{new}\) is independent of the data. 2 \[=\sigma^2\left(\frac1n+\frac{(x_{new}-\bar x)^2}{\sum_i(x_i-\bar x)^2}\right)+\sigma^2\] \(Var(\hat y_{new})\) as on page 4; \(Var(y_{new})=\sigma^2\). 3 \[=\sigma^2\left(1+\frac1n+\frac{(x_{new}-\bar x)^2}{\sum_i(x_i-\bar x)^2}\right)\] Put \(\hat\sigma\) for \(\sigma\): ratio is \(t_{n-2}\). ∎ ### 01.5 From simple to multiple linear regression Slides p.6–7 Slides p.6 Page 6 repeats the title page. New material starts. **What** Slides p.7 Linear Regression · Lecture 5 · p.7 Simple Linear Regression: \[y_i=\beta_0+\beta_1x_i+\epsilon_i,\quad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\iff y_i\,|\,x_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\] Multiple Linear Regression: \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i,\quad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] \[\iff y_i\,|\,x_i\overset{indep}{\sim}N(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip},\ \sigma^2)\] Multiple linear regression is a linear model with \(p\) explanatory variables. With \(p=1\), it is simple linear regression. \(x_{ij}\) is the value of variable \(j\) in observation \(i\). Index \(i\) counts rows (observations); \(j=1,\dots,p\) counts columns (variables). \(\beta_j\) is the coefficient of variable \(j\). With the intercept \(\beta_0\), the model has \(p+1\) coefficients. \(y_i\,|\,x_i\) is the conditional distribution of \(y_i\) when \(x_i=(x_{i1},\dots,x_{ip})\) is fixed. \(\iff\) means "if and only if": the two sides give the same model. **How** Added Write a data set as a multiple linear regression model - Find the response \(y\). Count the observations \(n\). - List the explanatory variables. Count them: \(p\). - Number the variables \(1,\dots,p\). - Call the value of variable \(j\) in row \(i\) \(x_{ij}\). - Write \(y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\), with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\) and \(i=1,\dots,n\). **Self-check:** The right side has \(p+1\) coefficients. For \(j\ge1\), \(\beta_j\) multiplies a variable with second index \(j\). **Example · brainhead data** Added The only explanatory variable is head size: \(p=1\). 1 Response and sample size How step 1 \[y_i=\text{brain weight},\qquad n=236\] 2 Variables How steps 2–3 \[\text{variable 1}=\text{head size},\qquad p=1\] 3 Values How step 4 \[x_{11}=4512,\qquad x_{21}=3738,\qquad x_{31}=4261\] 4 Model How step 5 \[y_i=\beta_0+\beta_1x_{i1}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2),\qquad i=1,\dots,236\] This is the model of Section 01.2, with \(x_i\) renamed \(x_{i1}\). ▲ **Why · the two forms are the same** Added 1 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\] 2 \[E[y_i\,|\,x_i]=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+E[\epsilon_i]\] For fixed \(x_i\), the linear part is constant. 3 \[E[y_i\,|\,x_i]=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\] \(E[\epsilon_i]=0\). 4 \[Var(y_i\,|\,x_i)=Var(\epsilon_i)=\sigma^2\] 5 \[y_i\,|\,x_i\overset{indep}{\sim}N(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip},\ \sigma^2)\] Normal plus constant is normal; independent \(\epsilon_i\). ∎ ### 01.6 Matrix form of multiple linear regression Slides p.8 **What** Slides p.8 Multiple Linear Regression · Lecture 5 · p.8 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] We can write this: \[\underbrace{\left[\begin{array}{c}y_1\\y_2\\\vdots\\y_n\end{array}\right]}_{\mathbf{y}}=\underbrace{\left[\begin{array}{ccccc}1&x_{11}&x_{12}&\dots&x_{1p}\\1&x_{21}&x_{22}&\dots&x_{2p}\\\vdots&\vdots&\vdots&\vdots&\vdots\\1&x_{n1}&x_{n2}&\dots&x_{np}\end{array}\right]}_{\mathbf{X}}\underbrace{\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]}_{\boldsymbol\beta}+\underbrace{\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\vdots\\\epsilon_n\end{array}\right]}_{\boldsymbol\epsilon}\] A matrix is a set of numbers in rows and columns. An \(r\times c\) matrix has \(r\) rows and \(c\) columns (its dimension). A column vector is an \(r\times1\) matrix. The slides write vectors in bold lowercase (\(\mathbf{y},\boldsymbol\beta,\boldsymbol\epsilon\)) and matrices in bold uppercase (\(\mathbf{X}\)). \(\mathbf{y}\) (\(n\times1\)) is the response vector, with components \(y_i\). \(\mathbf{X}\) (\(n\times(p+1)\)) is the design matrix. Row \(i\) is observation \(i\). Column 1 is all 1s. Column \(j+1\) is variable \(j\). \(\boldsymbol\beta\) (\((p+1)\times1\)) is the coefficient vector \(\beta_0,\beta_1,\dots,\beta_p\). \(\boldsymbol\epsilon\) (\(n\times1\)) is the error vector, with components \(\epsilon_i\). Matrix multiplication: component \(i\) of \(\mathbf{X}\boldsymbol\beta\) is the sum of the term-by-term products of row \(i\) of \(\mathbf{X}\) and \(\boldsymbol\beta\). Columns of \(\mathbf{X}\) must equal rows of \(\boldsymbol\beta\). An \(n\times(p+1)\) matrix times a \((p+1)\times1\) vector gives an \(n\times1\) vector. Matrix addition adds entries in the same position. The two matrices must have the same dimension. [figure] Figure 01.2 · Block dimensions in \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\). The \(p+1\) columns of \(\mathbf{X}\) match the \(p+1\) rows of \(\boldsymbol\beta\). The product is \(n\times1\), like \(\boldsymbol\epsilon\) and \(\mathbf{y}\). Dark green: column of 1s. **How** Added Write multiple linear regression as \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\) - Write \(\mathbf{y}\): put \(y_1,\dots,y_n\) in one column, in observation order. - Write \(\mathbf{X}\): row \(i\) is \([\,1,\ x_{i1},\ \dots,\ x_{ip}\,]\), in the row order of \(\mathbf{y}\). - Write \(\boldsymbol\beta\): put \(\beta_0,\beta_1,\dots,\beta_p\) in one column. - Write \(\boldsymbol\epsilon\): put \(\epsilon_1,\dots,\epsilon_n\) in one column. **Self-check:** ① \(\mathbf{X}\) is \(n\times(p+1)\), and column 1 is all 1s. ② Row \(i\) of \(\mathbf{X}\) times \(\boldsymbol\beta\) gives \(\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}\). **Example · first 3 rows of the brainhead data** Added Use only the first 3 observations: \(n=3\), \(p=1\). 1 Response vector How step 1 \[\mathbf{y}=\left[\begin{array}{c}1530\\1297\\1335\end{array}\right]\qquad(3\times1)\] 2 Design matrix How step 2 \[\mathbf{X}=\left[\begin{array}{cc}1&4512\\1&3738\\1&4261\end{array}\right]\qquad(3\times2=n\times(p+1))\] 3 Coefficient vector How step 3 \[\boldsymbol\beta=\left[\begin{array}{c}\beta_0\\\beta_1\end{array}\right]\qquad(2\times1)\] 4 Error vector How step 4 \[\boldsymbol\epsilon=\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\qquad(3\times1)\] 5 Matrix equation \[\left[\begin{array}{c}1530\\1297\\1335\end{array}\right]=\left[\begin{array}{cc}1&4512\\1&3738\\1&4261\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\] Multiply each row (self-check ②): Row 1: \(1530=1\cdot\beta_0+4512\cdot\beta_1+\epsilon_1\), that is, \(y_1=\beta_0+\beta_1x_{11}+\epsilon_1\). Row 2: \(1297=1\cdot\beta_0+3738\cdot\beta_1+\epsilon_2\), that is, \(y_2=\beta_0+\beta_1x_{21}+\epsilon_2\). Row 3: \(1335=1\cdot\beta_0+4261\cdot\beta_1+\epsilon_3\), that is, \(y_3=\beta_0+\beta_1x_{31}+\epsilon_3\). The 1 in column 1 puts the intercept \(\beta_0\) in each row. \(\boldsymbol\epsilon\) is unknown, because \(\boldsymbol\beta\) is unknown. As a check, use \(\hat\beta_0=335.4323\) and \(\hat\beta_1=0.2608207\). \(\mathbf{X}\hat{\boldsymbol\beta}\) gives the fitted values; \(\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}\) gives the residuals. 1 Row 1 \[1\times335.4323+4512\times0.2608207=335.4323+1176.8230=1512.2553\] \[e_1=1530-1512.2553=17.7447\] 2 Row 2 \[1\times335.4323+3738\times0.2608207=335.4323+974.9478=1310.3801\] \[e_2=1297-1310.3801=-13.3801\] 3 Row 3 \[1\times335.4323+4261\times0.2608207=335.4323+1111.3570=1446.7893\] \[e_3=1335-1446.7893=-111.7893\] These residuals are the orange dashed lines in Figure 01.1. A positive residual is a point above the line. For all 236 rows: \(\mathbf{y}\) is \(236\times1\), \(\mathbf{X}\) is \(236\times2\), \(\boldsymbol\beta\) is \(2\times1\), \(\boldsymbol\epsilon\) is \(236\times1\). ▲ **Why · row \(i\) of the matrix equation is equation \(i\)** Added 1 \[[\mathbf{y}]_i=[\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon]_i\] \([\cdot]_i\): component \(i\). Equal vectors have equal components. 2 \[y_i=[\mathbf{X}\boldsymbol\beta]_i+\epsilon_i\] Matrix addition, entry by entry. 3 \[y_i=1\cdot\beta_0+x_{i1}\beta_1+x_{i2}\beta_2+\cdots+x_{ip}\beta_p+\epsilon_i\] Row \(i\) of \(\mathbf{X}\) is \([1,x_{i1},\dots,x_{ip}]\). 4 \[y_i=\beta_0+\beta_1x_{i1}+\cdots+\beta_px_{ip}+\epsilon_i\] True for each \(i=1,\dots,n\). ∎ ### 01.7 Practice Added **Q1.** Rows 4 to 6 of brainhead.csv have head sizes 3777, 4177, 3585 and brain weights 1282, 1590, 1300. Use only these rows and the model \(y_i=\beta_0+\beta_1x_{i1}+\epsilon_i\). Write \(\mathbf{y}\), \(\mathbf{X}\), \(\boldsymbol\beta\), \(\boldsymbol\epsilon\) and their dimensions. Answer Step 1: \(\mathbf{y}=\left[\begin{array}{c}1282\\1590\\1300\end{array}\right]\), \(3\times1\). Step 2: \(\mathbf{X}=\left[\begin{array}{cc}1&3777\\1&4177\\1&3585\end{array}\right]\), \(3\times2\). Step 3: \(\boldsymbol\beta=\left[\begin{array}{c}\beta_0\\\beta_1\end{array}\right]\), \(2\times1\). Step 4: \(\boldsymbol\epsilon=\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\), \(3\times1\). **Self-check:** Row 2 gives \(1590=\beta_0+4177\beta_1+\epsilon_2\). **Q2.** A study has \(n=50\) observations and the model \(y_i=\beta_0+\beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3}+\epsilon_i\). Give the dimensions of \(\mathbf{y}\), \(\mathbf{X}\), \(\boldsymbol\beta\), \(\boldsymbol\epsilon\). Write row 7 of \(\mathbf{X}\). Answer Step 1: 3 explanatory variables: \(p=3\), \(p+1=4\). Step 2: \(\mathbf{y}\) is \(50\times1\). \(\mathbf{X}\) is \(50\times4\). \(\boldsymbol\beta\) is \(4\times1\). \(\boldsymbol\epsilon\) is \(50\times1\). Step 3: Row 7 is \([\,1,\ x_{71},\ x_{72},\ x_{73}\,]\). **Self-check:** 4 columns of \(\mathbf{X}\), 4 rows of \(\boldsymbol\beta\); the product is \(50\times1\). **Q3.** Use \(\hat\beta_0=335.4323\) and \(\hat\beta_1=0.2608207\). For Q1, calculate component 2 of \(\mathbf{X}\hat{\boldsymbol\beta}\) and of \(\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}\). Answer Step 1: Row 2 of \(\mathbf{X}\) is \([1,\ 4177]\). Step 2: \([\mathbf{X}\hat{\boldsymbol\beta}]_2=1\times335.4323+4177\times0.2608207=335.4323+1089.4481=1424.8804\). Step 3: \([\mathbf{y}-\mathbf{X}\hat{\boldsymbol\beta}]_2=1590-1424.8804=165.1196\). **Self-check:** The residual is positive: 1590 is above 1424.8804. ## 02 · Matrix review: transpose, inverse, trace Slides p.9–19 The 11 pages show one slide. Each page adds one rule. Notation used in this unit · Added Entry \([\mathbf{C}]_{ij}\): the number in row \(i\), column \(j\) of \(\mathbf{C}\). Also \(a_{ij}=[\mathbf{A}]_{ij}\), \(b_{ij}=[\mathbf{B}]_{ij}\). Multiplication in entry form: if \(\mathbf{A}\) is \(n\times m\) and \(\mathbf{B}\) is \(m\times p\), then \(\mathbf{AB}\) is \(n\times p\), with \([\mathbf{AB}]_{ij}=\sum_{k=1}^{m}a_{ik}b_{kj}\). Diagonal: the entries with row number equal to column number, \(a_{11},a_{22},\dots\). Many examples use this design matrix (\(n=3\), \(p=1\), \(x_1=1\), \(x_2=2\), \(x_3=4\)): \[\mathbf{X}=\left[\begin{array}{cc}1 & 1\\ 1 & 2\\ 1 & 4\end{array}\right]\qquad(3\times2)\] ### 02.1 Transpose Slides p.9 **What** Slides p.9 Lecture 5 · p.9 \([\mathbf{C}^T]_{ij}=[\mathbf{C}]_{ji}\) Transpose \(\mathbf{C}^T\): row \(i\) of \(\mathbf{C}\) becomes column \(i\). If \(\mathbf{C}\) is \(n\times m\), \(\mathbf{C}^T\) is \(m\times n\). **How** Added Find \(\mathbf{C}^T\) - **Find the dimension.** If \(\mathbf{C}\) is \(n\times m\), \(\mathbf{C}^T\) is \(m\times n\). - **Write each row as a column.** Row \(i\) of \(\mathbf{C}\) becomes column \(i\) of \(\mathbf{C}^T\). **Self-check:** For one entry, confirm \([\mathbf{C}^T]_{ij}=[\mathbf{C}]_{ji}\). Diagonal entries do not move. **Example · \(\mathbf{X}^T\)** Added The rows \((1,1)\), \((1,2)\), \((1,4)\) of \(\mathbf{X}\) become columns: \[\mathbf{X}^T=\left[\begin{array}{ccc}1 & 1 & 1\\ 1 & 2 & 4\end{array}\right]\qquad(2\times3)\] **Self-check:** \([\mathbf{X}^T]_{23}=4=[\mathbf{X}]_{32}\). [figure] Figure 02.1 · Each row of \(\mathbf{X}\) becomes the column with the same number in \(\mathbf{X}^T\). The orange cells hold the same number, 4, with the indices changed. **Why** Added This rule is the definition of the transpose. It needs no proof. Later transpose rules follow from it, entry by entry. ### 02.2 Symmetric matrix Slides p.10 **What** Slides p.10 Lecture 5 · p.10 \(\mathbf{C}\) is symmetric if \(\mathbf{C}^T=\mathbf{C}\) Square matrix: an \(n\times n\) matrix. Symmetric matrix: a matrix equal to its transpose, that is, \([\mathbf{C}]_{ij}=[\mathbf{C}]_{ji}\) for all \(i,j\). Only a square matrix can be symmetric, because \(\mathbf{C}^T\) must have the dimension of \(\mathbf{C}\). **How** Added Test if \(\mathbf{C}\) is symmetric - **Check the dimension.** \(\mathbf{C}\) must be \(n\times n\). - **Compare pairs.** For each \(i\neq j\), compare \([\mathbf{C}]_{ij}\) with \([\mathbf{C}]_{ji}\). All equal: symmetric. **Self-check:** Write \(\mathbf{C}^T\). It must be identical to \(\mathbf{C}\). **Example · \(\mathbf{X}^T\mathbf{X}\)** Added A \(2\times3\) matrix times a \(3\times2\) matrix gives a \(2\times2\) matrix. 1 Entry (1,1): row 1 of \(\mathbf{X}^T\), \((1,1,1)\), times column 1 of \(\mathbf{X}\), \((1,1,1)\) \[[\mathbf{X}^T\mathbf{X}]_{11}=1\cdot1+1\cdot1+1\cdot1=3\] 2 Entry (1,2): row 1 of \(\mathbf{X}^T\), \((1,1,1)\), times column 2 of \(\mathbf{X}\), \((1,2,4)\) \[[\mathbf{X}^T\mathbf{X}]_{12}=1\cdot1+1\cdot2+1\cdot4=7\] 3 Entry (2,1): row 2 of \(\mathbf{X}^T\), \((1,2,4)\), times column 1 of \(\mathbf{X}\), \((1,1,1)\) \[[\mathbf{X}^T\mathbf{X}]_{21}=1\cdot1+2\cdot1+4\cdot1=7\] 4 Entry (2,2): row 2 of \(\mathbf{X}^T\) times column 2 of \(\mathbf{X}\) \[[\mathbf{X}^T\mathbf{X}]_{22}=1^2+2^2+4^2\] \[[\mathbf{X}^T\mathbf{X}]_{22}=1+4+16\] \[[\mathbf{X}^T\mathbf{X}]_{22}=21\] \[\mathbf{X}^T\mathbf{X}=\left[\begin{array}{cc}3 & 7\\ 7 & 21\end{array}\right]\] **Self-check:** The only off-diagonal pair is \(7\) and \(7\). The matrix is symmetric. **Why · this example is symmetric** Added Steps 2 and 3 calculate the same sum \(\sum_i x_i\), with each product in the other order. Slides p.11 proves the general case. ### 02.3 Transpose of a product Slides p.11 **What** Slides p.11 Lecture 5 · p.11 \((\mathbf{AB})^T=\mathbf{B}^T\mathbf{A}^T\) Transpose each factor and multiply in reverse order. With \(\mathbf{A}\) \(n\times m\) and \(\mathbf{B}\) \(m\times p\), \(\mathbf{B}^T\mathbf{A}^T\) is \(p\times n\), like \((\mathbf{AB})^T\). **How** Added Expand \((\mathbf{AB})^T\) - **Reverse the factors.** Write \(\mathbf{A},\mathbf{B}\) as \(\mathbf{B},\mathbf{A}\). - **Add \(T\) to each factor.** The result is \(\mathbf{B}^T\mathbf{A}^T\). - **Simplify \((\mathbf{C}^T)^T=\mathbf{C}\).** Two transposes return the indices to their start. **Self-check:** The dimension of the new product is the dimension of \(\mathbf{AB}\) in reverse. **Example · a numerical check and \((\mathbf{X}^T\mathbf{X})^T\)** Added Section 02.4 uses these matrices again. \[\mathbf{A}=\left[\begin{array}{cc}3 & 1\\ 2 & 1\end{array}\right],\qquad \mathbf{B}=\left[\begin{array}{cc}1 & 2\\ 0 & 1\end{array}\right]\] 1 \[\mathbf{AB}=\left[\begin{array}{cc}3\cdot1+1\cdot0 & 3\cdot2+1\cdot1\\ 2\cdot1+1\cdot0 & 2\cdot2+1\cdot1\end{array}\right]\] 2 \[\mathbf{AB}=\left[\begin{array}{cc}3 & 7\\ 2 & 5\end{array}\right]\] 3 \[(\mathbf{AB})^T=\left[\begin{array}{cc}3 & 2\\ 7 & 5\end{array}\right]\] Slides p.9 4 \[\mathbf{B}^T\mathbf{A}^T=\left[\begin{array}{cc}1 & 0\\ 2 & 1\end{array}\right]\left[\begin{array}{cc}3 & 2\\ 1 & 1\end{array}\right]\] 5 \[\mathbf{B}^T\mathbf{A}^T=\left[\begin{array}{cc}1\cdot3+0\cdot1 & 1\cdot2+0\cdot1\\ 2\cdot3+1\cdot1 & 2\cdot2+1\cdot1\end{array}\right]\] 6 \[\mathbf{B}^T\mathbf{A}^T=\left[\begin{array}{cc}3 & 2\\ 7 & 5\end{array}\right]\] **Self-check:** Steps 3 and 6 agree. Apply the rule to \(\mathbf{X}^T\mathbf{X}\) for any \(\mathbf{X}\): 1 \[(\mathbf{X}^T\mathbf{X})^T=\mathbf{X}^T(\mathbf{X}^T)^T\] Slides p.11 with \(\mathbf{A}=\mathbf{X}^T\), \(\mathbf{B}=\mathbf{X}\) 2 \[(\mathbf{X}^T\mathbf{X})^T=\mathbf{X}^T\mathbf{X}\] \((\mathbf{X}^T)^T=\mathbf{X}\) By Slides p.10, \(\mathbf{X}^T\mathbf{X}\) is always symmetric. **Why** Added 1 \[[(\mathbf{AB})^T]_{ij}=[\mathbf{AB}]_{ji}\] Slides p.9 2 \[=\sum_k a_{jk}b_{ki}\] multiplication formula 3 \[=\sum_k b_{ki}a_{jk}\] 4 \[=\sum_k [\mathbf{B}^T]_{ik}[\mathbf{A}^T]_{kj}\] Slides p.9: \(b_{ki}=[\mathbf{B}^T]_{ik}\), \(a_{jk}=[\mathbf{A}^T]_{kj}\) 5 \[=[\mathbf{B}^T\mathbf{A}^T]_{ij}\] multiplication formula ∎ ### 02.4 Inverse Slides p.12 **What** Slides p.12 Lecture 5 · p.12 If square matrix (i.e. \(n\times n\)) \(\mathbf{B}\) is nonsingular, then \(\mathbf{BB}^{-1}=\mathbf{B}^{-1}\mathbf{B}=\mathbf{I}\) Identity matrix \(\mathbf{I}\): a square matrix with 1 on the diagonal and 0 elsewhere. \(\mathbf{CI}=\mathbf{IC}=\mathbf{C}\). For \(2\times2\), \(\mathbf{I}=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]\). Inverse \(\mathbf{B}^{-1}\): the square matrix whose product with \(\mathbf{B}\), on either side, is \(\mathbf{I}\). Nonsingular: \(\mathbf{B}^{-1}\) exists. The inverse acts like the reciprocal \(1/b\) of a number \(b\neq0\). Only a square matrix can have an inverse. **How** Added Find and check \(\mathbf{B}^{-1}\) - **Confirm that \(\mathbf{B}\) is square.** - **Get a candidate inverse.** For \(2\times2\), use the optional formula below. - **Check the candidate.** \(\mathbf{BB}^{-1}\) and \(\mathbf{B}^{-1}\mathbf{B}\) must both equal \(\mathbf{I}\). **Self-check:** If one product is not \(\mathbf{I}\), the candidate is not the inverse. Optional — not on the slides The determinant of a \(2\times2\) matrix is \(\det\left[\begin{array}{cc}a & b\\ c & d\end{array}\right]=ad-bc\). The matrix is nonsingular if and only if \(ad-bc\neq0\). Then the inverse is \(\frac{1}{ad-bc}\left[\begin{array}{cc}d & -b\\ -c & a\end{array}\right]\). **Example · \(\mathbf{A}^{-1}\) and \(\mathbf{B}^{-1}\)** Added Use \(\mathbf{A}=\left[\begin{array}{cc}3 & 1\\ 2 & 1\end{array}\right]\) and \(\mathbf{B}=\left[\begin{array}{cc}1 & 2\\ 0 & 1\end{array}\right]\) from 02.3. 1 Determinant of \(\mathbf{A}\) \[3\cdot1-1\cdot2=1\] 2 Candidate \(\mathbf{A}^{-1}\) \[\mathbf{A}^{-1}=\frac{1}{1}\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]=\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\] optional formula 3 Determinant of \(\mathbf{B}\) \[1\cdot1-2\cdot0=1\] 4 Candidate \(\mathbf{B}^{-1}\) \[\mathbf{B}^{-1}=\frac{1}{1}\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]=\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]\] optional formula Check \(\mathbf{AA}^{-1}\): 1 \[\mathbf{AA}^{-1}=\left[\begin{array}{cc}3 & 1\\ 2 & 1\end{array}\right]\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\] 2 \[=\left[\begin{array}{cc}3\cdot1+1\cdot(-2) & 3\cdot(-1)+1\cdot3\\ 2\cdot1+1\cdot(-2) & 2\cdot(-1)+1\cdot3\end{array}\right]\] 3 \[=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\] Check \(\mathbf{A}^{-1}\mathbf{A}\): 1 \[\mathbf{A}^{-1}\mathbf{A}=\left[\begin{array}{cc}1\cdot3+(-1)\cdot2 & 1\cdot1+(-1)\cdot1\\ (-2)\cdot3+3\cdot2 & (-2)\cdot1+3\cdot1\end{array}\right]\] 2 \[=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\] For \(\mathbf{B}\): \(\mathbf{BB}^{-1}=\left[\begin{array}{cc}1\cdot1+2\cdot0 & 1\cdot(-2)+2\cdot1\\ 0\cdot1+1\cdot0 & 0\cdot(-2)+1\cdot1\end{array}\right]=\mathbf{I}\). **Self-check:** All products equal \(\mathbf{I}\): \(\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\), \(\mathbf{B}^{-1}=\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]\). **Why** Added Slides p.12 is the definition. To test a candidate, multiply it with the matrix and compare with \(\mathbf{I}\). Sections 02.5 and 02.6 use this test. ### 02.5 Inverse of a product Slides p.13 **What** Slides p.13 Lecture 5 · p.13 \((\mathbf{AB})^{-1}=\mathbf{B}^{-1}\mathbf{A}^{-1}\), if both square and nonsingular If \(\mathbf{A}\) and \(\mathbf{B}\) are \(n\times n\) and nonsingular, \(\mathbf{AB}\) is nonsingular. Its inverse is the product of the inverses in reverse order. **How** Added Find \((\mathbf{AB})^{-1}\) - **Check the conditions.** \(\mathbf{A}\) and \(\mathbf{B}\) are square and nonsingular. - **Find each inverse.** Find \(\mathbf{A}^{-1}\) and \(\mathbf{B}^{-1}\). - **Multiply in reverse order.** Calculate \(\mathbf{B}^{-1}\mathbf{A}^{-1}\). **Self-check:** The result times \(\mathbf{AB}\) must equal \(\mathbf{I}\). **Example · \((\mathbf{AB})^{-1}\)** Added From 02.3, \(\mathbf{AB}=\left[\begin{array}{cc}3 & 7\\ 2 & 5\end{array}\right]\). \(\mathbf{A}^{-1}\) and \(\mathbf{B}^{-1}\) are from 02.4. 1 \[\mathbf{B}^{-1}\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -2\\ 0 & 1\end{array}\right]\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\] 2 \[=\left[\begin{array}{cc}1\cdot1+(-2)(-2) & 1\cdot(-1)+(-2)\cdot3\\ 0\cdot1+1\cdot(-2) & 0\cdot(-1)+1\cdot3\end{array}\right]\] 3 \[=\left[\begin{array}{cc}5 & -7\\ -2 & 3\end{array}\right]\] Check against \(\mathbf{AB}\): 1 \[(\mathbf{AB})(\mathbf{B}^{-1}\mathbf{A}^{-1})=\left[\begin{array}{cc}3\cdot5+7\cdot(-2) & 3\cdot(-7)+7\cdot3\\ 2\cdot5+5\cdot(-2) & 2\cdot(-7)+5\cdot3\end{array}\right]\] 2 \[=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\] **Self-check:** \(\det(\mathbf{AB})=3\cdot5-7\cdot2=1\). The optional formula gives \(\left[\begin{array}{cc}5 & -7\\ -2 & 3\end{array}\right]\) again. **Why** Added Use the test from 02.4: 1 \[(\mathbf{AB})(\mathbf{B}^{-1}\mathbf{A}^{-1})=\mathbf{A}(\mathbf{BB}^{-1})\mathbf{A}^{-1}\] Regroup the product (associativity). 2 \[=\mathbf{AIA}^{-1}\] Slides p.12 3 \[=\mathbf{AA}^{-1}\] 4 \[=\mathbf{I}\] Slides p.12 The other order: \((\mathbf{B}^{-1}\mathbf{A}^{-1})(\mathbf{AB})=\mathbf{B}^{-1}(\mathbf{A}^{-1}\mathbf{A})\mathbf{B}=\mathbf{B}^{-1}\mathbf{IB}=\mathbf{B}^{-1}\mathbf{B}=\mathbf{I}\). ∎ The reverse order puts \(\mathbf{B}\) next to \(\mathbf{B}^{-1}\). ### 02.6 Inverse of a transpose Slides p.14 **What** Slides p.14 Lecture 5 · p.14 \((\mathbf{A}^T)^{-1}=(\mathbf{A}^{-1})^T\) For a nonsingular square \(\mathbf{A}\), the transpose and the inverse can be done in either order. **How** Added Find \((\mathbf{A}^T)^{-1}\) - **Find the inverse.** Find \(\mathbf{A}^{-1}\). - **Transpose it.** Write the rows of \(\mathbf{A}^{-1}\) as columns. **Self-check:** The result times \(\mathbf{A}^T\) must equal \(\mathbf{I}\). **Example · \((\mathbf{A}^T)^{-1}\)** Added 1 From 02.4 \[\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -1\\ -2 & 3\end{array}\right]\] 2 \[(\mathbf{A}^{-1})^T=\left[\begin{array}{cc}1 & -2\\ -1 & 3\end{array}\right]\] Slides p.9 3 Other order: \(\mathbf{A}^T=\left[\begin{array}{cc}3 & 2\\ 1 & 1\end{array}\right]\), determinant \(3\cdot1-2\cdot1=1\) \[(\mathbf{A}^T)^{-1}=\frac{1}{1}\left[\begin{array}{cc}1 & -2\\ -1 & 3\end{array}\right]\] optional formula 4 \[(\mathbf{A}^T)^{-1}=\left[\begin{array}{cc}1 & -2\\ -1 & 3\end{array}\right]\] **Self-check:** Steps 2 and 4 agree. **Why** Added Use the test from 02.4: 1 \[\mathbf{A}^T(\mathbf{A}^{-1})^T=(\mathbf{A}^{-1}\mathbf{A})^T\] Slides p.11, right to left 2 \[=\mathbf{I}^T\] Slides p.12 3 \[=\mathbf{I}\] \(\mathbf{I}\) is symmetric The other order: \((\mathbf{A}^{-1})^T\mathbf{A}^T=(\mathbf{AA}^{-1})^T=\mathbf{I}^T=\mathbf{I}\). ∎ It follows that the inverse of a symmetric nonsingular matrix is symmetric (Practice Q3). ### 02.7 Definition of the trace Slides p.15 **What** Slides p.15 Lecture 5 · p.15 \(tr(\mathbf{A})=\sum_j^n a_{jj}\) for \(n\times n\) matrix Trace \(tr(\mathbf{A})\): the sum of the diagonal entries of a square matrix. Only a square matrix has a trace. **How** Added Find \(tr(\mathbf{A})\) - **Confirm that \(\mathbf{A}\) is square.** - **List the diagonal.** Write \(a_{11},\dots,a_{nn}\). - **Add the diagonal entries.Self-check:** The trace is one number. Off-diagonal entries are not in the sum. **Example · \(tr(\mathbf{A})\), \(tr(\mathbf{B})\), \(tr(\mathbf{X}^T\mathbf{X})\)** Added 1 \[tr(\mathbf{A})=3+1=4\] 2 \[tr(\mathbf{B})=1+1=2\] 3 \[tr(\mathbf{X}^T\mathbf{X})=3+21=24\] \(\mathbf{X}^T\mathbf{X}\) from 02.2 **Why** Added Slides p.15 is a definition. It needs no proof. ### 02.8 Trace of a sum Slides p.16 **What** Slides p.16 Lecture 5 · p.16 \(tr(\mathbf{A}+\mathbf{B})=tr(\mathbf{A})+tr(\mathbf{B})\) Matrix addition in entry form: \([\mathbf{A}+\mathbf{B}]_{ij}=a_{ij}+b_{ij}\). **How** Added Split \(tr(\mathbf{A}+\mathbf{B})\) - **Confirm the dimensions.** \(\mathbf{A}\) and \(\mathbf{B}\) are both \(n\times n\). - **Add the two traces.** You do not need \(\mathbf{A}+\mathbf{B}\). **Self-check:** The trace of \(\mathbf{A}+\mathbf{B}\) gives the same number. **Example** Added 1 \[\mathbf{A}+\mathbf{B}=\left[\begin{array}{cc}3+1 & 1+2\\ 2+0 & 1+1\end{array}\right]=\left[\begin{array}{cc}4 & 3\\ 2 & 2\end{array}\right]\] 2 \[tr(\mathbf{A}+\mathbf{B})=4+2=6\] 3 \[tr(\mathbf{A})+tr(\mathbf{B})=4+2=6\] traces from 02.7 **Why** Added 1 \[tr(\mathbf{A}+\mathbf{B})=\sum_j^n[\mathbf{A}+\mathbf{B}]_{jj}\] Slides p.15 2 \[=\sum_j^n(a_{jj}+b_{jj})\] matrix addition 3 \[=\sum_j^n a_{jj}+\sum_j^n b_{jj}\] 4 \[=tr(\mathbf{A})+tr(\mathbf{B})\] Slides p.15 ∎ ### 02.9 Trace of a scalar multiple Slides p.17 **What** Slides p.17 Lecture 5 · p.17 \(tr(c\mathbf{A})=c\,tr(\mathbf{A})\) Scalar \(c\): one ordinary number. \(c\mathbf{A}\) multiplies each entry by \(c\): \([c\mathbf{A}]_{ij}=c\,a_{ij}\). **How** Added Simplify \(tr(c\mathbf{A})\) - **Move \(c\) outside.** Write \(c\,tr(\mathbf{A})\). - **Find \(tr(\mathbf{A})\).** Multiply it by \(c\). **Self-check:** The trace of \(c\mathbf{A}\) gives the same number. **Example · \(c=3\)** Added 1 \[3\mathbf{A}=\left[\begin{array}{cc}3\cdot3 & 3\cdot1\\ 3\cdot2 & 3\cdot1\end{array}\right]=\left[\begin{array}{cc}9 & 3\\ 6 & 3\end{array}\right]\] 2 \[tr(3\mathbf{A})=9+3=12\] 3 \[3\,tr(\mathbf{A})=3\times4=12\] **Why** Added 1 \[tr(c\mathbf{A})=\sum_j^n c\,a_{jj}\] Slides p.15 and scalar multiplication 2 \[=c\sum_j^n a_{jj}\] 3 \[=c\,tr(\mathbf{A})\] Slides p.15 ∎ 02.8 and 02.9 together say that the trace is linear. ### 02.10 Trace of a transpose Slides p.18 **What** Slides p.18 Lecture 5 · p.18 \(tr(\mathbf{A}^T)=tr(\mathbf{A})\) The transpose does not move the diagonal entries. **How** Added Simplify \(tr(\mathbf{A}^T)\) - **Remove the transpose.** Write \(tr(\mathbf{A})\). **Self-check:** The diagonal of \(\mathbf{A}^T\) is the diagonal of \(\mathbf{A}\). **Example** Added \(\mathbf{A}^T=\left[\begin{array}{cc}3 & 2\\ 1 & 1\end{array}\right]\). \(tr(\mathbf{A}^T)=3+1=4=tr(\mathbf{A})\). **Why** Added 1 \[tr(\mathbf{A}^T)=\sum_j^n[\mathbf{A}^T]_{jj}\] Slides p.15 2 \[=\sum_j^n[\mathbf{A}]_{jj}\] Slides p.9 with \(i=j\) 3 \[=tr(\mathbf{A})\] Slides p.15 ∎ ### 02.11 Trace of a product in either order Slides p.19 **What** Slides p.19 Lecture 5 · p.19 \(tr(\mathbf{AB})=tr(\mathbf{BA})\) Let \(\mathbf{A}\) be \(n\times m\) and \(\mathbf{B}\) be \(m\times n\). \(\mathbf{AB}\) (\(n\times n\)) and \(\mathbf{BA}\) (\(m\times m\)) have equal traces. \(\mathbf{A}\) and \(\mathbf{B}\) need not be square. **How** Added Use \(tr(\mathbf{AB})=tr(\mathbf{BA})\) to make a calculation shorter - **Check the dimensions.** \(\mathbf{A}\) is \(n\times m\) and \(\mathbf{B}\) is \(m\times n\). - **Select the smaller product.** Compare \(n\) and \(m\). - **Calculate only its diagonal.** Add the diagonal entries. **Self-check:** The trace of the other product is equal. **Example · 1 · square \(\mathbf{A}\) and \(\mathbf{B}\)** Added From 02.3, \(\mathbf{AB}=\left[\begin{array}{cc}3 & 7\\ 2 & 5\end{array}\right]\). 1 \[\mathbf{BA}=\left[\begin{array}{cc}1\cdot3+2\cdot2 & 1\cdot1+2\cdot1\\ 0\cdot3+1\cdot2 & 0\cdot1+1\cdot1\end{array}\right]=\left[\begin{array}{cc}7 & 3\\ 2 & 1\end{array}\right]\] 2 \[tr(\mathbf{AB})=3+5=8\] 3 \[tr(\mathbf{BA})=7+1=8\] \(\mathbf{AB}\neq\mathbf{BA}\), but both traces are 8. **Example · 2 · non-square factors \(\mathbf{X}\) and \(\mathbf{X}^T\)** Added \(\mathbf{XX}^T\) is \(3\times3\); \(\mathbf{X}^T\mathbf{X}\) is \(2\times2\). Entry \((i,j)\) of \(\mathbf{XX}^T\) is row \(i\) of \(\mathbf{X}\) times row \(j\) of \(\mathbf{X}\): \(1+x_ix_j\). 1 Row 1: \(1+1\cdot1\), \(1+1\cdot2\), \(1+1\cdot4\) \[(2,\ 3,\ 5)\] 2 Row 2: \(1+2\cdot1\), \(1+2\cdot2\), \(1+2\cdot4\) \[(3,\ 5,\ 9)\] 3 Row 3: \(1+4\cdot1\), \(1+4\cdot2\), \(1+4\cdot4\) \[(5,\ 9,\ 17)\] 4 The \(3\times3\) side \[tr(\mathbf{XX}^T)=2+5+17=24\] 5 The \(2\times2\) side, from 02.7 \[tr(\mathbf{X}^T\mathbf{X})=3+21=24\] **Self-check:** The sides are equal. The \(2\times2\) side needs only 2 diagonal entries. [figure] Figure 02.2 · Blue: the diagonals. \(\mathbf{XX}^T\) and \(\mathbf{X}^T\mathbf{X}\) differ in dimension and entries. Both diagonal sums are 24. Both matrices are symmetric (02.3). **Why** Added Let \(\mathbf{A}\) be \(n\times m\) and \(\mathbf{B}\) be \(m\times n\). 1 \[tr(\mathbf{AB})=\sum_{i=1}^{n}[\mathbf{AB}]_{ii}\] Slides p.15 2 \[=\sum_{i=1}^{n}\sum_{k=1}^{m}a_{ik}b_{ki}\] multiplication formula 3 \[=\sum_{k=1}^{m}\sum_{i=1}^{n}b_{ki}a_{ik}\] Change the order of finite sums and factors. 4 \[=\sum_{k=1}^{m}[\mathbf{BA}]_{kk}\] multiplication formula 5 \[=tr(\mathbf{BA})\] Slides p.15 ∎ Lecture 7 uses this rule. ### 02.12 Practice Added **Q1.** Let \(\mathbf{X}\) be any \(n\times(p+1)\) matrix. With only the rules on Lecture 5 p.9–19, show that \(\mathbf{X}^T\mathbf{X}\) is symmetric and that \(tr(\mathbf{X}^T\mathbf{X})=tr(\mathbf{XX}^T)\). Give the dimensions of \(\mathbf{X}^T\mathbf{X}\) and \(\mathbf{XX}^T\). Answer Symmetric: \((\mathbf{X}^T\mathbf{X})^T=\mathbf{X}^T(\mathbf{X}^T)^T=\mathbf{X}^T\mathbf{X}\) (Slides p.11, then \((\mathbf{X}^T)^T=\mathbf{X}\)). By Slides p.10, it is symmetric. Trace: with \(\mathbf{A}=\mathbf{X}^T\), \(\mathbf{B}=\mathbf{X}\), Slides p.19 gives \(tr(\mathbf{X}^T\mathbf{X})=tr(\mathbf{XX}^T)\). Dimensions: \(\mathbf{X}^T\mathbf{X}\) is \((p+1)\times(p+1)\). \(\mathbf{XX}^T\) is \(n\times n\). For the \(\mathbf{X}\) of this unit (\(n=3\), \(p=1\)), both traces are 24 (02.11). **Q2.** Let \(\mathbf{A}=\left[\begin{array}{cc}4 & 1\\ 3 & 1\end{array}\right]\). (a) Verify that \(\mathbf{A}^{-1}=\left[\begin{array}{cc}1 & -1\\ -3 & 4\end{array}\right]\). (b) Find \((\mathbf{A}^T)^{-1}\). (c) Compute \(tr(\mathbf{A}^T\mathbf{A})\). Answer (a) \(\mathbf{AA}^{-1}=\left[\begin{array}{cc}4\cdot1+1\cdot(-3) & 4\cdot(-1)+1\cdot4\\ 3\cdot1+1\cdot(-3) & 3\cdot(-1)+1\cdot4\end{array}\right]=\left[\begin{array}{cc}1 & 0\\ 0 & 1\end{array}\right]=\mathbf{I}\). \(\mathbf{A}^{-1}\mathbf{A}=\left[\begin{array}{cc}1\cdot4+(-1)\cdot3 & 1\cdot1+(-1)\cdot1\\ (-3)\cdot4+4\cdot3 & (-3)\cdot1+4\cdot1\end{array}\right]=\mathbf{I}\). (b) By Slides p.14, \((\mathbf{A}^T)^{-1}=(\mathbf{A}^{-1})^T=\left[\begin{array}{cc}1 & -3\\ -1 & 4\end{array}\right]\). (c) \(\mathbf{A}^T=\left[\begin{array}{cc}4 & 3\\ 1 & 1\end{array}\right]\). \(\mathbf{A}^T\mathbf{A}=\left[\begin{array}{cc}4\cdot4+3\cdot3 & 4\cdot1+3\cdot1\\ 1\cdot4+1\cdot3 & 1\cdot1+1\cdot1\end{array}\right]=\left[\begin{array}{cc}25 & 7\\ 7 & 2\end{array}\right]\). \(tr(\mathbf{A}^T\mathbf{A})=25+2=27\). By Slides p.19, \(tr(\mathbf{AA}^T)=27\) also. **Q3.** \(\mathbf{C}\) is \(n\times n\), symmetric and nonsingular. Show that \(\mathbf{C}^{-1}\) is symmetric. Answer 1 \[(\mathbf{C}^{-1})^T=(\mathbf{C}^T)^{-1}\] Slides p.14 2 \[=\mathbf{C}^{-1}\] \(\mathbf{C}^T=\mathbf{C}\) (Slides p.10) By Slides p.10, \(\mathbf{C}^{-1}\) is symmetric. ∎ For example, \((\mathbf{X}^T\mathbf{X})^{-1}\) is symmetric when it exists. Check with \(\mathbf{X}^T\mathbf{X}=\left[\begin{array}{cc}3 & 7\\ 7 & 21\end{array}\right]\): determinant \(3\cdot21-7\cdot7=14\); inverse \(\frac{1}{14}\left[\begin{array}{cc}21 & -7\\ -7 & 3\end{array}\right]\). Both off-diagonal entries are \(-7/14\). ## 03 · Matrix calculus **Symbols** Added \(\mathbf{y}=(y_1,\dots,y_k)^T\) is a column vector with \(k\) components. \(z=f(y_1,\dots,y_k)\) is a function that gives one scalar \(z\). Partial derivative \(\frac{\partial z}{\partial y_l}\): the derivative of \(z\) with respect to \(y_l\), with all other \(y_i\) constant. Matrix-vector product: \((\mathbf{A}\mathbf{y})_l=\sum_{j=1}^k a_{lj}y_j\) (row \(l\) of \(\mathbf{A}\) times \(\mathbf{y}\)). ### 03.1 The derivative with respect to a vector Slides p.20 **What** Slides p.20 Lecture 5 · p.20 Let \(z=f(y_1,\dots,y_k)\) and \(\mathbf{y}=\left[\begin{array}{c}y_1\\ \vdots\\ y_k\end{array}\right]\), then \[\frac{\partial z}{\partial \mathbf{y}}=\left[\begin{array}{c}\frac{\partial z}{\partial y_1}\\ \frac{\partial z}{\partial y_2}\\ \vdots\\ \frac{\partial z}{\partial y_k}\end{array}\right]\] The gradient \(\frac{\partial z}{\partial \mathbf{y}}\) is the column of the \(k\) partial derivatives. Component \(l\) shows how fast \(z\) changes when only \(y_l\) changes. The gradient is \(k\times1\), like \(\mathbf{y}\). **How** Added Find \(\partial z/\partial \mathbf{y}\) from the definition - Write \(z\) as a formula in \(y_1,\dots,y_k\). - For each \(l=1,\dots,k\), find \(\frac{\partial z}{\partial y_l}\). - Put the \(k\) results in one column, in the order of \(l\). **Self-check:** The result is \(k\times1\). Row \(l\) is the partial derivative with respect to \(y_l\). **Example** Added Let \(k=2\) and \(z=3y_1+y_1y_2\). Find \(\frac{\partial z}{\partial \mathbf{y}}\) at \(\mathbf{y}=(1,2)^T\). 1 Write \(z\) How step 1 \[z=3y_1+y_1y_2\] 2 Partial derivatives How step 2 \[\frac{\partial z}{\partial y_1}=3+y_2,\qquad \frac{\partial z}{\partial y_2}=y_1\] 3 One column How step 3 \[\frac{\partial z}{\partial \mathbf{y}}=\left[\begin{array}{c}3+y_2\\ y_1\end{array}\right]\] \[\frac{\partial z}{\partial \mathbf{y}}\Big|_{\mathbf{y}=(1,2)^T}=\left[\begin{array}{c}3+2\\ 1\end{array}\right]=\left[\begin{array}{c}5\\ 1\end{array}\right]\] **Why** Added Slides p.20 is a definition. It needs no proof. ### 03.2 Linear function: \(\partial(\mathbf{a}^T\mathbf{y})/\partial\mathbf{y}=\mathbf{a}\) Slides p.20 **What** Slides p.20 Lecture 5 · p.20 If \(z=\mathbf{a}^T\mathbf{y}\), where \(\mathbf{a}=(a_1,\dots,a_k)^T\) is a vector, then \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}\] \(\mathbf{a}\) is constant. \(z=\mathbf{a}^T\mathbf{y}=a_1y_1+\dots+a_ky_k\) is a scalar and a linear function: each \(y_i\) has power 1. The gradient is \(\mathbf{a}\) at all \(\mathbf{y}\). **How** Added Use the rule for \(\partial(\mathbf{a}^T\mathbf{y})/\partial\mathbf{y}\) - Write \(z\) as a constant vector, transposed, times \(\mathbf{y}\). - Read \(\mathbf{a}\): \(a_l\) is the coefficient of \(y_l\). - Write \(\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}\). **Self-check:** The result is \(k\times1\) and contains no \(y_i\). For one \(l\), \(\partial z/\partial y_l=a_l\). **Example · a linear function** Added Let \(k=3\) and \(\mathbf{a}=(1,2,3)^T\). 1 Write \(z\) How step 1 \[z=\mathbf{a}^T\mathbf{y}=\left[\begin{array}{ccc}1&2&3\end{array}\right]\left[\begin{array}{c}y_1\\ y_2\\ y_3\end{array}\right]\] \[z=1\cdot y_1+2\cdot y_2+3\cdot y_3=y_1+2y_2+3y_3\] 2 Use the rule How steps 2–3 \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}=\left[\begin{array}{c}1\\ 2\\ 3\end{array}\right]\] 3 Check with the definition 03.1 How \[\frac{\partial z}{\partial y_1}=1,\qquad \frac{\partial z}{\partial y_2}=2,\qquad \frac{\partial z}{\partial y_3}=3\] **Basis:** 03.1. For \(y_1\), the terms \(2y_2\) and \(3y_3\) are constant. At \(\mathbf{y}=(2,-1,1)^T\), \(z=2-2+3=3\), and the gradient is still \((1,2,3)^T\). ▲ **Figure · gradient of \(z=y_1+2y_2\)** Added [figure] Figure 03.1 · \(z=y_1+2y_2\). Blue lines: level lines (constant \(z\)), 1 apart in \(z\). Thick line: \(z=3\). Orange arrow: the gradient \(\mathbf{a}=(1,2)^T\) at the green point \((1,1)^T\). It is at a right angle to the level lines (equal axis scales). The gradient is the same at all points. **Why · derivation of the rule in 03.2** Added 1 \[z=\mathbf{a}^T\mathbf{y}=\sum_{i=1}^k a_iy_i\] 2 \[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k a_i\frac{\partial y_i}{\partial y_l}\] Each \(a_i\) is a constant. 3 \[\frac{\partial y_i}{\partial y_l}=\begin{cases}1, & i=l\\ 0, & i\neq l\end{cases}\] For \(i\neq l\), \(y_i\) is constant. 4 \[\frac{\partial z}{\partial y_l}=a_l\] Only the term \(i=l\) stays. 5 \[\frac{\partial z}{\partial \mathbf{y}}=\left[\begin{array}{c}a_1\\ \vdots\\ a_k\end{array}\right]=\mathbf{a}\] Stack \(l=1,\dots,k\) (03.1). ∎ ### 03.3 Quadratic form: \(\partial(\mathbf{y}^T\mathbf{A}\mathbf{y})/\partial\mathbf{y}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\) Slides p.21 **What** Slides p.21 Lecture 5 · p.21 If \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\) where \(\mathbf{A}\) is a \(k\times k\) matrix, then \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\] and if \(\mathbf{A}\) is symmetric, then: \(\frac{\partial z}{\partial \mathbf{y}}=2\mathbf{A}\mathbf{y}\) A quadratic form is \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}y_iy_j\), a scalar. Each term is a coefficient times two components. If \(\mathbf{A}\) is symmetric, \(\mathbf{A}^T\mathbf{y}=\mathbf{A}\mathbf{y}\), and the two terms become \(2\mathbf{A}\mathbf{y}\). **How** Added Use the rule for \(\partial(\mathbf{y}^T\mathbf{A}\mathbf{y})/\partial\mathbf{y}\) - Confirm that \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\), with \(\mathbf{A}\) a constant \(k\times k\) matrix. - Compare \(a_{ij}\) with \(a_{ji}\) for each pair. All equal: \(\mathbf{A}\) is symmetric. - If \(\mathbf{A}\) is symmetric, calculate \(2\mathbf{A}\mathbf{y}\). - If not, calculate \(\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\). - Put in the given value of \(\mathbf{y}\), if any. **Self-check:** The result is \(k\times1\). Each component is linear in \(y_1,\dots,y_k\), with no constant term. One expanded partial derivative must agree. **Example · a matrix that is not symmetric** Added Let \(k=2\), \(\mathbf{A}=\left[\begin{array}{cc}1&2\\ 0&3\end{array}\right]\), \(\mathbf{y}=(1,2)^T\). 1 Write \(z\) How step 1 \[z=\sum_{i=1}^2\sum_{j=1}^2 a_{ij}y_iy_j=a_{11}y_1^2+a_{12}y_1y_2+a_{21}y_2y_1+a_{22}y_2^2\] \[z=1\cdot y_1^2+2y_1y_2+0\cdot y_2y_1+3y_2^2=y_1^2+2y_1y_2+3y_2^2\] \[z(1,2)=1+4+12=17\] 2 Examine symmetry How step 2 \[\mathbf{A}^T=\left[\begin{array}{cc}1&0\\ 2&3\end{array}\right]\] \[a_{12}=2,\quad a_{21}=0,\quad a_{12}\neq a_{21}\] Not symmetric: use the general rule. 3 Calculate \(\mathbf{A}\mathbf{y}\) How step 4 \[\mathbf{A}\mathbf{y}=\left[\begin{array}{c}1\cdot1+2\cdot2\\ 0\cdot1+3\cdot2\end{array}\right]=\left[\begin{array}{c}5\\ 6\end{array}\right]\] 4 Calculate \(\mathbf{A}^T\mathbf{y}\) How step 4 \[\mathbf{A}^T\mathbf{y}=\left[\begin{array}{c}1\cdot1+0\cdot2\\ 2\cdot1+3\cdot2\end{array}\right]=\left[\begin{array}{c}1\\ 8\end{array}\right]\] 5 Add How steps 4–5 \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}=\left[\begin{array}{c}5+1\\ 6+8\end{array}\right]=\left[\begin{array}{c}6\\ 14\end{array}\right]\] 6 Check with the definition 03.1 How \[\frac{\partial z}{\partial y_1}=2y_1+2y_2=2\cdot1+2\cdot2=6\] \[\frac{\partial z}{\partial y_2}=2y_1+6y_2=2\cdot1+6\cdot2=14\] **Basis:** \(z=y_1^2+2y_1y_2+3y_2^2\) from step 1. **Example · a symmetric matrix** Added Let \(k=3\), \(\mathbf{A}=\left[\begin{array}{ccc}4&0&1\\ 0&3&-1\\ 1&-1&4\end{array}\right]\), \(\mathbf{y}=(1,2,2)^T\). 1 Write \(z\) How step 1 \[z=\sum_{i=1}^3\sum_{j=1}^3 a_{ij}y_iy_j\] \[z=4y_1^2+0\cdot y_1y_2+1\cdot y_1y_3+0\cdot y_2y_1+3y_2^2-1\cdot y_2y_3+1\cdot y_3y_1-1\cdot y_3y_2+4y_3^2\] \[z=4y_1^2+3y_2^2+4y_3^2+2y_1y_3-2y_2y_3\] \[z(1,2,2)=4+12+16+4-8=28\] 2 Examine symmetry How step 2 \[a_{12}=a_{21}=0,\quad a_{13}=a_{31}=1,\quad a_{23}=a_{32}=-1\] \(\mathbf{A}^T=\mathbf{A}\): use \(2\mathbf{A}\mathbf{y}\). 3 Calculate \(\mathbf{A}\mathbf{y}\) How step 3 \[\mathbf{A}\mathbf{y}=\left[\begin{array}{c}4\cdot1+0\cdot2+1\cdot2\\ 0\cdot1+3\cdot2+(-1)\cdot2\\ 1\cdot1+(-1)\cdot2+4\cdot2\end{array}\right]=\left[\begin{array}{c}6\\ 4\\ 7\end{array}\right]\] 4 Multiply by 2 How steps 3, 5 \[\frac{\partial z}{\partial \mathbf{y}}=2\mathbf{A}\mathbf{y}=\left[\begin{array}{c}12\\ 8\\ 14\end{array}\right]\] 5 Check with the definition 03.1 How \[\frac{\partial z}{\partial y_1}=8y_1+2y_3=8+4=12\] \[\frac{\partial z}{\partial y_2}=6y_2-2y_3=12-4=8\] \[\frac{\partial z}{\partial y_3}=8y_3+2y_1-2y_2=16+2-4=14\] **Basis:** \(z\) from step 1, at \(\mathbf{y}=(1,2,2)^T\). For a symmetric matrix, calculate \(\mathbf{A}\mathbf{y}\) one time only. ▲ **Why · derivation of the rule in 03.3** Added The product rule: \((uv)'=u'v+uv'\). 1 \[z=\mathbf{y}^T\mathbf{A}\mathbf{y}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}y_iy_j\] 2 \[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}\frac{\partial (y_iy_j)}{\partial y_l}\] 3 \[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}\left(\frac{\partial y_i}{\partial y_l}y_j+y_i\frac{\partial y_j}{\partial y_l}\right)\] Product rule. 4 \[\frac{\partial z}{\partial y_l}=\sum_{i=1}^k\sum_{j=1}^k a_{ij}\frac{\partial y_i}{\partial y_l}y_j+\sum_{i=1}^k\sum_{j=1}^k a_{ij}y_i\frac{\partial y_j}{\partial y_l}\] 5 \[\frac{\partial z}{\partial y_l}=\sum_{j=1}^k a_{lj}y_j+\sum_{i=1}^k a_{il}y_i\] 03.2 Why step 3: \(i=l\), then \(j=l\), stays. 6 \[\sum_{j=1}^k a_{lj}y_j=(\mathbf{A}\mathbf{y})_l\] Row \(l\) of \(\mathbf{A}\) times \(\mathbf{y}\). 7 \[\sum_{i=1}^k a_{il}y_i=(\mathbf{A}^T\mathbf{y})_l\] Row \(l\) of \(\mathbf{A}^T\) has entries \(a_{il}\). 8 \[\frac{\partial z}{\partial y_l}=(\mathbf{A}\mathbf{y})_l+(\mathbf{A}^T\mathbf{y})_l\] 9 \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\] Stack \(l=1,\dots,k\) (03.1). 10 \[\frac{\partial z}{\partial \mathbf{y}}=\mathbf{A}\mathbf{y}+\mathbf{A}\mathbf{y}\] Symmetric: \(\mathbf{A}^T=\mathbf{A}\). 11 \[\frac{\partial z}{\partial \mathbf{y}}=2\mathbf{A}\mathbf{y}\] ∎ The two terms come from the derivatives of \(y_i\) and of \(y_j\). ### 03.4 Practice Added **Q1.** Let \(\mathbf{y}=(y_1,y_2,y_3)^T\), \(\mathbf{a}=(2,0,-1)^T\), and \(z=\mathbf{a}^T\mathbf{y}+\mathbf{y}^T\mathbf{y}\). Find \(\frac{\partial z}{\partial \mathbf{y}}\) at \(\mathbf{y}=(1,1,1)^T\). Answer The derivative of a sum is the sum of the derivatives. \(\mathbf{y}^T\mathbf{y}=\mathbf{y}^T\mathbf{I}\mathbf{y}\), and \(\mathbf{I}\) is symmetric. \(\frac{\partial(\mathbf{a}^T\mathbf{y})}{\partial\mathbf{y}}=\mathbf{a}\) (03.2). \(\frac{\partial(\mathbf{y}^T\mathbf{I}\mathbf{y})}{\partial\mathbf{y}}=2\mathbf{I}\mathbf{y}=2\mathbf{y}\) (03.3). \(\frac{\partial z}{\partial \mathbf{y}}=\mathbf{a}+2\mathbf{y}\). At \(\mathbf{y}=(1,1,1)^T\): \((2+2,\ 0+2,\ -1+2)^T=(4,2,1)^T\). Check: \(z=2y_1-y_3+y_1^2+y_2^2+y_3^2\). \(\partial z/\partial y_1=2+2y_1=4\), \(\partial z/\partial y_2=2y_2=2\), \(\partial z/\partial y_3=-1+2y_3=1\). **Q2.** Let \(\mathbf{A}=\left[\begin{array}{cc}2&1\\ 3&1\end{array}\right]\) and \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\), \(\mathbf{y}=(y_1,y_2)^T\). (a) Write \(z\) as a polynomial in \(y_1,y_2\). (b) Compute \(\frac{\partial z}{\partial \mathbf{y}}\) at \(\mathbf{y}=(1,-1)^T\) with the rule on p.21. (c) Check (b) with the partial derivatives of (a). Answer (a) \(z=2y_1^2+1\cdot y_1y_2+3y_2y_1+1\cdot y_2^2=2y_1^2+4y_1y_2+y_2^2\). At \((1,-1)^T\): \(z=2-4+1=-1\). (b) \(a_{12}=1\neq a_{21}=3\): not symmetric. Use \(\mathbf{A}\mathbf{y}+\mathbf{A}^T\mathbf{y}\). \(\mathbf{A}\mathbf{y}=(2\cdot1+1\cdot(-1),\ 3\cdot1+1\cdot(-1))^T=(1,2)^T\). \(\mathbf{A}^T=\left[\begin{array}{cc}2&3\\ 1&1\end{array}\right]\), \(\mathbf{A}^T\mathbf{y}=(2\cdot1+3\cdot(-1),\ 1\cdot1+1\cdot(-1))^T=(-1,0)^T\). \(\frac{\partial z}{\partial \mathbf{y}}=(1+(-1),\ 2+0)^T=(0,2)^T\). (c) \(\frac{\partial z}{\partial y_1}=4y_1+4y_2=4-4=0\). \(\frac{\partial z}{\partial y_2}=4y_1+2y_2=4-2=2\). Same as (b). **Q3.** Let \(z=3y_1^2+4y_1y_2+y_2^2\). Find a symmetric \(2\times2\) matrix \(\mathbf{A}\) with \(z=\mathbf{y}^T\mathbf{A}\mathbf{y}\). Find the gradient with \(2\mathbf{A}\mathbf{y}\). Confirm it with partial derivatives. Answer For a symmetric \(\mathbf{A}\), \(z=a_{11}y_1^2+(a_{12}+a_{21})y_1y_2+a_{22}y_2^2=a_{11}y_1^2+2a_{12}y_1y_2+a_{22}y_2^2\). Match the coefficients. \(a_{11}=3\), \(a_{22}=1\), \(2a_{12}=4\): \(a_{12}=a_{21}=2\). \(\mathbf{A}=\left[\begin{array}{cc}3&2\\ 2&1\end{array}\right]\). \(2\mathbf{A}\mathbf{y}=2\left[\begin{array}{c}3y_1+2y_2\\ 2y_1+y_2\end{array}\right]=\left[\begin{array}{c}6y_1+4y_2\\ 4y_1+2y_2\end{array}\right]\). Direct: \(\frac{\partial z}{\partial y_1}=6y_1+4y_2\), \(\frac{\partial z}{\partial y_2}=4y_1+2y_2\). They agree. ## 04 · Random vectors: mean vector, covariance matrix, and linear transformations Given values · Added All examples use brainhead.csv (236 persons; not printed on the slides). Random vector \(\mathbf{y}=(y_1,y_2)^T\): \(y_1\) is head size (cm³), \(y_2\) is brain weight (g) of one person. In the slide formulas, \(n=2\). The sample values of the 236 persons act as the true values. Given mean vector: \(\boldsymbol\mu=(3637.864,\ 1284.263)^T\). Given covariance matrix: \(\mathbf{V}=\left[\begin{array}{cc}130413.76&34014.61\\34014.61&14084.06\end{array}\right]\). ### 04.1 Random vector and mean vector Slides p.22 **What** Slides p.22 Random Vectors · Lecture 5 · p.22 Let \(\mathbf{y}=(y_1,\dots,y_n)^T\) be a random vector. The mean of \(\mathbf{y}\) is \[E[\mathbf{y}]=\left[\begin{array}{c}E[y_1]\\ \vdots\\ E[y_n]\end{array}\right]\] A random vector \(\mathbf{y}\) is an \(n\times1\) column of random variables \(y_1,\dots,y_n\). The mean vector \(E[\mathbf{y}]\) has entry \(i\) equal to \(E[y_i]\). The slides write \(\boldsymbol\mu=E[\mathbf{y}]\) and \(\mu_i=E[y_i]\). A random matrix is a matrix of random variables. Its expectation is also taken entry by entry. **How** Added Write the mean vector \(E[\mathbf{y}]\) - List the entries \(y_1,\dots,y_n\) in order. - Get \(E[y_i]\) for each entry. - Put \(E[y_1],\dots,E[y_n]\) in one column, in the same order. **Self-check:** \(E[\mathbf{y}]\) has the length of \(\mathbf{y}\). **Example · brainhead** Added How steps 1–3 Step 1: \(\mathbf{y}=(y_1,y_2)^T\), \(n=2\). Step 2: mean head size 3637.864; mean brain weight 1284.263. Step 3: \(E[\mathbf{y}]=\boldsymbol\mu=\left[\begin{array}{c}3637.864\\1284.263\end{array}\right]\). ▲ [figure] Figure 04.1 · The 236 persons (blue) and the mean vector \(\boldsymbol\mu\) (orange). The purple lines extend one standard deviation to each side: \(\sqrt{\mathbf{V}_{11}}=361.13\) (horizontal) and \(\sqrt{\mathbf{V}_{22}}=118.68\) (vertical). The upward trend shows \(cov(y_1,y_2)\gt 0\). **Why** Added Slides p.22 is a definition. It needs no proof. ### 04.2 Covariance matrix Slides p.23 **What** Slides p.23 Covariance Matrix · Lecture 5 · p.23 The variance is \[var(\mathbf{y})=E[(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T]\] \[\mathbf{V}=var(\mathbf{y})=\left[\begin{array}{cccc}var(y_1)&cov(y_1,y_2)&\cdots&cov(y_1,y_n)\\cov(y_2,y_1)&var(y_2)&\cdots&cov(y_2,y_n)\\ \vdots&\vdots&\ddots&\vdots\\cov(y_n,y_1)&cov(y_n,y_2)&\cdots&var(y_n)\end{array}\right]\] The covariance \(cov(y_i,y_j)=E[(y_i-\mu_i)(y_j-\mu_j)]\) is the expected product of the two distances from the means. Positive covariance: the two variables tend to be large together. Negative: one tends to be large when the other is small. \(cov(y_i,y_i)=var(y_i)\). The covariance matrix \(\mathbf{V}=var(\mathbf{y})\) is the \(n\times n\) matrix with \(\mathbf{V}_{ij}=cov(y_i,y_j)\). \((\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T\) is an \(n\times1\) column times a \(1\times n\) row: an \(n\times n\) random matrix. **How** Added Write the covariance matrix \(\mathbf{V}=var(\mathbf{y})\) - Find the length \(n\) of \(\mathbf{y}\). Make an empty \(n\times n\) table. - In diagonal entry \(i\), write \(var(y_i)\). - In entry \((i,j)\), \(i\ne j\), write \(cov(y_i,y_j)\). **Self-check:** \(\mathbf{V}\) is \(n\times n\). Diagonal entries are \(\ge0\). \(\mathbf{V}_{ij}=\mathbf{V}_{ji}\) (04.3). **Example · brainhead** Added How steps 1–3 Step 1: \(n=2\): \(\mathbf{V}\) is \(2\times2\). Step 2: \(var(y_1)=130413.76\), \(var(y_2)=14084.06\). Step 3: \(cov(y_1,y_2)=cov(y_2,y_1)=34014.61\). Positive: a larger head tends to come with a larger brain weight. Result: \[\mathbf{V}=var(\mathbf{y})=\left[\begin{array}{cc}var(y_1)&cov(y_1,y_2)\\cov(y_2,y_1)&var(y_2)\end{array}\right]=\left[\begin{array}{cc}130413.76&34014.61\\34014.61&14084.06\end{array}\right]\] Standard deviations: \(\sqrt{130413.76}=361.13\), \(\sqrt{14084.06}=118.68\). ▲ **Why · the entries are variances and covariances** Added 1 \[\mathbf{y}-\boldsymbol\mu=\left[\begin{array}{c}y_1-\mu_1\\ \vdots\\ y_n-\mu_n\end{array}\right]\] 2 \[\big[(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T\big]_{ij}=(y_i-\mu_i)(y_j-\mu_j)\] Column times row. 3 \[\big[E[(\mathbf{y}-\boldsymbol\mu)(\mathbf{y}-\boldsymbol\mu)^T]\big]_{ij}=E[(y_i-\mu_i)(y_j-\mu_j)]\] Expectation entry by entry (04.1). 4 \[=cov(y_i,y_j)\] Definition of covariance. ∎ ### 04.3 \(\mathbf{V}\) is symmetric Slides p.24 **What** Slides p.24 Covariance Matrix · Lecture 5 · p.24 \(\mathbf{V}\) is symmetric: \[\mathbf{V}_{ij}=\mathbf{V}_{ji}\] Calculate only the entries on and above the diagonal. **How** Added Use the symmetry of \(\mathbf{V}\) - Calculate the diagonal entries and the entries with \(i0\). It is wider than tall because \(\Sigma_{11}=4>\Sigma_{22}=2\). Equal axis scales. Section 05.6 uses the right panel. **Why** Added Two rules from Section 04.6, with a constant vector \(\mathbf{b}\) added: Rule E: \(E[\mathbf{A}\mathbf{z}+\mathbf{b}]=\mathbf{A}\,E[\mathbf{z}]+\mathbf{b}\). Rule V: \(Var(\mathbf{A}\mathbf{z}+\mathbf{b})=\mathbf{A}\,Var(\mathbf{z})\,\mathbf{A}^T\). A constant shift does not change the covariance. \(E[z_i]=0\): \(E[\mathbf{z}]=\mathbf{0}\). \(Var(z_i)=1\), and \(cov(z_i,z_j)=0\) for \(i\ne j\) (independent): \(Var(\mathbf{z})=\mathbf{I}\). Mean: 1 \[E[\mathbf{y}]=E[\mathbf{A}\mathbf{z}+\boldsymbol\mu]\] 2 \[=\mathbf{A}\,E[\mathbf{z}]+\boldsymbol\mu\] Rule E with \(\mathbf{b}=\boldsymbol\mu\). 3 \[=\mathbf{A}\,\mathbf{0}+\boldsymbol\mu\] \(E[\mathbf{z}]=\mathbf{0}\). 4 \[=\boldsymbol\mu\] ∎ Covariance matrix: 1 \[Var(\mathbf{y})=Var(\mathbf{A}\mathbf{z}+\boldsymbol\mu)\] 2 \[=\mathbf{A}\,Var(\mathbf{z})\,\mathbf{A}^T\] Rule V. 3 \[=\mathbf{A}\,\mathbf{I}\,\mathbf{A}^T\] \(Var(\mathbf{z})=\mathbf{I}\). 4 \[=\mathbf{A}\mathbf{A}^T=\boldsymbol\Sigma\] ∎ ### 05.2 Density function Slides p.36 **What** Slides p.36 MVN density · Lecture 5 · p.36 \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\), if: \[f(\mathbf{y})=\frac{1}{(2\pi)^{\frac n2}|\boldsymbol\Sigma|^{\frac12}}\exp\left\{-\frac12(\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\right\}\] The joint density \(f(\mathbf{y})\) is the density of all \(n\) components together. \(f(\mathbf{y})\) times the volume of a small region near \(\mathbf{y}\) is about the probability of that region. The determinant \(|\boldsymbol\Sigma|\) is one number from a square matrix. For \(2\times2\), it is \(ad-bc\) (02.4). \(\exp\{x\}=e^x\), with \(e\approx2.71828\). The quadratic form \((\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\) is one number: the distance from \(\mathbf{y}\) to \(\boldsymbol\mu\), scaled by \(\boldsymbol\Sigma\). \(\boldsymbol\mu\) and \(\boldsymbol\Sigma\) give the full distribution. **How** Added Calculate \(f(\mathbf{y})\) at a point \(\mathbf{y}\) - Calculate \(\mathbf{y}-\boldsymbol\mu\). - Calculate \(|\boldsymbol\Sigma|\). - Calculate \(\boldsymbol\Sigma^{-1}\). - Calculate \(\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\). - Calculate \(q=(\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)\): step 4 times step 1, term by term, added. - Put the values into \(f=\dfrac{1}{(2\pi)^{n/2}|\boldsymbol\Sigma|^{1/2}}e^{-q/2}\). **Self-check:** \(|\boldsymbol\Sigma|>0\). \(\boldsymbol\Sigma\boldsymbol\Sigma^{-1}=\mathbf{I}\). \(q\ge0\). At \(\mathbf{y}=\boldsymbol\mu\), \(q=0\) and \(f\) is largest. **Example** Added \(\boldsymbol\mu\), \(\boldsymbol\Sigma\) from 05.1; point \(\mathbf{y}=(2,3)^T\); \(n=2\). Step 1: \(\mathbf{y}-\boldsymbol\mu=(2-1,\ 3-2)^T=(1,1)^T\). Step 2: \(|\boldsymbol\Sigma|=4\cdot2-2\cdot2=8-4=4\). Step 3: \(\boldsymbol\Sigma^{-1}=\frac14\left[\begin{array}{cc}2&-2\\-2&4\end{array}\right]=\left[\begin{array}{cc}0.5&-0.5\\-0.5&1\end{array}\right]\). Step 4: \(\boldsymbol\Sigma^{-1}(1,1)^T=(0.5-0.5,\ -0.5+1)^T=(0,\ 0.5)^T\). Step 5: \(q=1\cdot0+1\cdot0.5=0.5\). Step 6: \((2\pi)^{n/2}=(2\pi)^1=2\pi\). \(|\boldsymbol\Sigma|^{1/2}=4^{1/2}=2\). Denominator: \(2\pi\cdot2=4\pi=12.566371\). Step 7: \(e^{-q/2}=e^{-0.25}=0.778801\). Step 8: \(f=\dfrac{0.778801}{12.566371}=0.061975\). Self-check: \(|\boldsymbol\Sigma|=4>0\), \(q=0.5\ge0\). Entry \((1,1)\) of \(\boldsymbol\Sigma\boldsymbol\Sigma^{-1}\): \(4\cdot0.5+2\cdot(-0.5)=1\). **Why** Added For \(n=1\), the MVN density is the normal density. Let \(\boldsymbol\Sigma=\sigma^2\), \(\mathbf{y}=y\), \(\boldsymbol\mu=\mu\). 1 \[f(y)=\frac{1}{(2\pi)^{\frac12}|\sigma^2|^{\frac12}}\exp\left\{-\frac12(y-\mu)^T(\sigma^2)^{-1}(y-\mu)\right\}\] Put in \(n=1\). 2 \[=\frac{1}{\sqrt{2\pi}\,\sigma}\exp\left\{-\frac12(y-\mu)^T(\sigma^2)^{-1}(y-\mu)\right\}\] \(1\times1\) determinant: \(|\sigma^2|^{1/2}=\sigma\). 3 \[=\frac{1}{\sqrt{2\pi}\,\sigma}\exp\left\{-\frac12(y-\mu)\frac{1}{\sigma^2}(y-\mu)\right\}\] A number equals its transpose. 4 \[=\frac{1}{\sqrt{2\pi}\,\sigma}\exp\left\{-\frac{(y-\mu)^2}{2\sigma^2}\right\}\] Density of \(N(\mu,\sigma^2)\). ∎ \(\sigma^2\) becomes \(\boldsymbol\Sigma\); division by \(\sigma^2\) becomes multiplication by \(\boldsymbol\Sigma^{-1}\). ### 05.3 Property 1: linearity Slides p.37 **What** Slides p.37 Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.37 Linearity: If \(\mathbf{u}=\mathbf{C}\mathbf{y}+\mathbf{d}\), then: \[\mathbf{u}\sim MVN(\mathbf{C}\boldsymbol\mu+\mathbf{d},\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\] \(\mathbf{C}\) is a constant \(m\times n\) matrix. \(\mathbf{d}\) and \(\mathbf{u}\) are \(m\times1\). \(\mathbf{u}\) is normal. Rules E and V give its parameters. **How** Added Find the distribution of \(\mathbf{u}=\mathbf{C}\mathbf{y}+\mathbf{d}\) - Write \(\mathbf{C}\) (\(m\times n\)) and \(\mathbf{d}\) (\(m\times1\)). - Calculate the mean \(\mathbf{C}\boldsymbol\mu+\mathbf{d}\). - Calculate \(\mathbf{C}\boldsymbol\Sigma\). - Multiply step 3 on the right by \(\mathbf{C}^T\). - Write \(\mathbf{u}\sim MVN(\mathbf{C}\boldsymbol\mu+\mathbf{d},\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\). For \(m=1\), write \(N(\cdot,\cdot)\). **Self-check:** The mean has length \(m\). The covariance matrix is \(m\times m\), symmetric, diagonal \(\ge0\). **Example** Added \(\mathbf{y}\) from 05.1. Let \(\mathbf{C}=(1,\ -1)\) and \(\mathbf{d}=3\): \(u=y_1-y_2+3\). Step 1: \(\mathbf{C}=(1,-1)\), \(m=1\), \(\mathbf{d}=3\). Step 2: \(\mathbf{C}\boldsymbol\mu+\mathbf{d}=1\cdot1+(-1)\cdot2+3=2\). Step 3: \(\mathbf{C}\boldsymbol\Sigma=(1\cdot4-1\cdot2,\ 1\cdot2-1\cdot2)=(2,\ 0)\). Step 4: \(\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=2\cdot1+0\cdot(-1)=2\). Step 5: \(u\sim N(2,\ 2)\). Self-check: variance 2 is positive. **Why** Added 1 \[\mathbf{u}=\mathbf{C}\mathbf{y}+\mathbf{d}\] 2 \[=\mathbf{C}(\mathbf{A}\mathbf{z}+\boldsymbol\mu)+\mathbf{d}\] \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\) (p.36). 3 \[=(\mathbf{C}\mathbf{A})\mathbf{z}+(\mathbf{C}\boldsymbol\mu+\mathbf{d})\] Form of p.36: MVN, mean \(\mathbf{C}\boldsymbol\mu+\mathbf{d}\). 4 \[Var(\mathbf{u})=(\mathbf{C}\mathbf{A})(\mathbf{C}\mathbf{A})^T\] p.36: matrix times its transpose. 5 \[=\mathbf{C}\mathbf{A}\mathbf{A}^T\mathbf{C}^T\] \((\mathbf{C}\mathbf{A})^T=\mathbf{A}^T\mathbf{C}^T\). 6 \[=\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T\] \(\mathbf{A}\mathbf{A}^T=\boldsymbol\Sigma\). ∎ ### 05.4 Property 2: marginal distribution Slides p.38 **What** Slides p.38 Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.38 Marginal Distribution: If \(\tilde{\mathbf{y}}=(y_1,\dots,y_m)^T\subset\mathbf{y}\) is a vector subset of \(\mathbf{y}\), then \(\tilde{\mathbf{y}}\) is MVN-distributed. A marginal distribution is the distribution of some components, with all other components ignored. A vector subset \(\tilde{\mathbf{y}}\subset\mathbf{y}\) holds \(m\le n\) components of \(\mathbf{y}\). The slides take the first \(m\). Linearity gives the parameters (Why): the matching entries of \(\boldsymbol\mu\) and the matching rows and columns of \(\boldsymbol\Sigma\). **How** Added Find the distribution of \(\tilde{\mathbf{y}}\) - List the indices to keep, for example \(\{1,\dots,m\}\). - Mean: take these entries of \(\boldsymbol\mu\). - Covariance: take the entries of \(\boldsymbol\Sigma\) where these rows and columns cross. - Write \(\tilde{\mathbf{y}}\sim MVN(\tilde{\boldsymbol\mu},\tilde{\boldsymbol\Sigma})\). For one component, \(y_i\sim N(\mu_i,\Sigma_{ii})\). **Self-check:** \(\tilde{\boldsymbol\Sigma}\) is \(m\times m\). Its diagonal matches \(\boldsymbol\Sigma\). **Example** Added \(\mathbf{y}\) from 05.1; keep only \(y_1\) (\(m=1\)). Step 1: Index set \(\{1\}\). Step 2: \(\mu_1=1\). Step 3: \(\Sigma_{11}=4\). Step 4: \(y_1\sim N(1,4)\). Standard deviation \(\sqrt4=2\). Density of \(y_1\) at 2 (p.36, \(n=1\), \(\boldsymbol\Sigma=4\)): Step 5: \(y_1-\mu_1=2-1=1\). \(q=1\cdot\frac14\cdot1=0.25\). Step 6: \((2\pi)^{1/2}\cdot4^{1/2}=2.506628\cdot2=5.013257\). Step 7: \(e^{-q/2}=e^{-0.125}=0.882497\). Step 8: \(f(2)=\dfrac{0.882497}{5.013257}=0.176033\). [figure] Figure 05.2 · Blue: histogram of many random draws of \(y_1\), density scale (frequency divided by bin width; total area 1). Orange: the \(N(1,4)\) density. The shapes agree. **Why** Added Let \(\mathbf{C}=[\,\mathbf{I}_m\ \ \mathbf{0}\,]\) (\(m\times n\): identity on the left, zeros on the right) and \(\mathbf{d}=\mathbf{0}\). 1 \[\mathbf{C}\mathbf{y}=(y_1,\dots,y_m)^T=\tilde{\mathbf{y}}\] Row \(i\) of \(\mathbf{C}\) has 1 only in column \(i\). 2 \[\tilde{\mathbf{y}}\sim MVN(\mathbf{C}\boldsymbol\mu,\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\] Linearity (p.37). 3 \[\mathbf{C}\boldsymbol\mu=(\mu_1,\dots,\mu_m)^T\] 4 \[\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=\text{the top-left }m\times m\text{ block of }\boldsymbol\Sigma\] \(\mathbf{C}\) keeps rows; \(\mathbf{C}^T\) keeps columns. ∎ In the example, \(\mathbf{C}=(1,0)\): \(\mathbf{C}\boldsymbol\mu=1\), \(\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=\Sigma_{11}=4\). ### 05.5 Property 3: conditional distribution Slides p.39 **What** Slides p.39 Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.39 Conditional Distribution: If \(\mathbf{y}=(\mathbf{y}_1^T,\mathbf{y}_2^T)^T\), then \(\mathbf{y}_1|\mathbf{y}_2\) is MVN-distributed. A partition \(\mathbf{y}=(\mathbf{y}_1^T,\mathbf{y}_2^T)^T\) cuts \(\mathbf{y}\) into a top sub-vector \(\mathbf{y}_1\) and a bottom sub-vector \(\mathbf{y}_2\). The conditional distribution \(\mathbf{y}_1|\mathbf{y}_2\) is the distribution of \(\mathbf{y}_1\) when \(\mathbf{y}_2\) has a known, fixed value. "\(|\)" reads "given". The slides give no formulas for the conditional mean or covariance. **How** Added Identify the type of a conditional distribution - Confirm that \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\). - State the components of \(\mathbf{y}_1\) and of \(\mathbf{y}_2\). - Conclude that \(\mathbf{y}_1|\mathbf{y}_2\) is MVN (univariate normal if \(\mathbf{y}_1\) has one component). **Self-check:** \(\mathbf{y}_1\) and \(\mathbf{y}_2\) together hold all components, with no overlap. **Example** Added Step 1: \(\mathbf{y}\sim MVN\left(\left[\begin{array}{c}1\\2\end{array}\right],\left[\begin{array}{cc}4&2\\2&2\end{array}\right]\right)\) (05.1). Step 2: \(\mathbf{y}_1=y_1\), \(\mathbf{y}_2=y_2\). Step 3: By p.39, \(y_1\) given \(y_2=3\) is univariate normal. Self-check: \(\{y_1\}\) and \(\{y_2\}\) hold both components, with no overlap. [figure] Figure 05.3 · Purple: histogram of \(y_1\) in the random draws with \(y_2\) within 0.05 of 3. Orange: a normal density with the same mean and standard deviation. The histogram is bell-shaped, as p.39 says. **Why** Added No proof on Slides p.39 Slide p.39 gives no proof and no conditional mean or covariance. This unit uses only "\(\mathbf{y}_1|\mathbf{y}_2\) is MVN". ### 05.6 Property 4: independence Slides p.40 **What** Slides p.40 Properties of \(\mathbf{y}\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) · Lecture 5 · p.40 Independence: If \(\Sigma_{ij}=0\), then \(\mathbf{y}_i\) and \(\mathbf{y}_j\) are independent. \(\Sigma_{ij}=cov(y_i,y_j)\). Two variables are independent when the value of one does not change the distribution of the other. With densities, independence means \(f(y_i,y_j)=f_i(y_i)f_j(y_j)\). This property is true only when \(\mathbf{y}\) is MVN. **How** Added Use \(\boldsymbol\Sigma\) to find independent components - Confirm that the vector is MVN. - For \(\mathbf{C}\mathbf{y}+\mathbf{d}\), find its covariance matrix by linearity. - Read the off-diagonal entry \(\Sigma_{ij}\). - If \(\Sigma_{ij}=0\), components \(i\) and \(j\) are independent. **Self-check:** \(\Sigma_{ij}=\Sigma_{ji}=0\). **Example** Added In 05.1, \(\Sigma_{12}=2\ne0\). Make a linear transformation with zero covariance: \(\mathbf{w}=\mathbf{C}_2\mathbf{y}\), \(\mathbf{C}_2=\left[\begin{array}{cc}1&-1\\0&1\end{array}\right]\), \(\mathbf{w}=(y_1-y_2,\ y_2)^T\). Step 1: By linearity, \(\mathbf{w}\sim MVN(\mathbf{C}_2\boldsymbol\mu,\ \mathbf{C}_2\boldsymbol\Sigma\mathbf{C}_2^T)\). Step 2: \(\mathbf{C}_2\boldsymbol\mu=(1-2,\ 2)^T=(-1,\ 2)^T\). Step 3: \(\mathbf{C}_2\boldsymbol\Sigma=\left[\begin{array}{cc}4-2&2-2\\2&2\end{array}\right]=\left[\begin{array}{cc}2&0\\2&2\end{array}\right]\). Step 4: \(\mathbf{C}_2^T=\left[\begin{array}{cc}1&0\\-1&1\end{array}\right]\). Step 5: \(\mathbf{C}_2\boldsymbol\Sigma\mathbf{C}_2^T=\left[\begin{array}{cc}2-0&0+0\\2-2&0+2\end{array}\right]=\left[\begin{array}{cc}2&0\\0&2\end{array}\right]\). Step 6: The off-diagonal entries are 0. Step 7: \(w_1=y_1-y_2\) and \(w_2=y_2\) are independent. Figure 05.1, right panel: the same 400 draws as \(\mathbf{w}\). The cloud does not slope. Orange: the mean \((-1,2)^T\). **Why** Added Proof for \(n=2\), with \(\Sigma_{12}=\Sigma_{21}=0\): \(\boldsymbol\Sigma=\left[\begin{array}{cc}\Sigma_{11}&0\\0&\Sigma_{22}\end{array}\right]\). 1 \[|\boldsymbol\Sigma|=\Sigma_{11}\Sigma_{22}-0\cdot0=\Sigma_{11}\Sigma_{22}\] 2 \[\boldsymbol\Sigma^{-1}=\frac{1}{\Sigma_{11}\Sigma_{22}}\left[\begin{array}{cc}\Sigma_{22}&0\\0&\Sigma_{11}\end{array}\right]\] \(2\times2\) inverse formula (02.4). 3 \[=\left[\begin{array}{cc}1/\Sigma_{11}&0\\0&1/\Sigma_{22}\end{array}\right]\] 4 \[(\mathbf{y}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{y}-\boldsymbol\mu)=\frac{(y_1-\mu_1)^2}{\Sigma_{11}}+\frac{(y_2-\mu_2)^2}{\Sigma_{22}}\] 5 \[f(\mathbf{y})=\frac{1}{2\pi(\Sigma_{11}\Sigma_{22})^{1/2}}\exp\left\{-\frac{(y_1-\mu_1)^2}{2\Sigma_{11}}-\frac{(y_2-\mu_2)^2}{2\Sigma_{22}}\right\}\] p.36 density, \((2\pi)^{n/2}=2\pi\). 6 \[=\frac{1}{\sqrt{2\pi}\sqrt{\Sigma_{11}}}\exp\left\{-\frac{(y_1-\mu_1)^2}{2\Sigma_{11}}\right\}\cdot\frac{1}{\sqrt{2\pi}\sqrt{\Sigma_{22}}}\exp\left\{-\frac{(y_2-\mu_2)^2}{2\Sigma_{22}}\right\}\] \(e^{a+b}=e^ae^b\). 7 \[=f_1(y_1)\,f_2(y_2)\] Marginal densities (p.38). ∎ ### 05.7 Practice Added **Q1.** Let \(\mathbf{y}=(y_1,y_2)^T\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) with \(\boldsymbol\mu=\left[\begin{array}{c}1\\2\end{array}\right]\) and \(\boldsymbol\Sigma=\left[\begin{array}{cc}4&2\\2&2\end{array}\right]\). Find the distribution of \(u=2y_1+y_2-1\). Answer Step 1: \(u=\mathbf{C}\mathbf{y}+\mathbf{d}\), \(\mathbf{C}=(2,\ 1)\), \(\mathbf{d}=-1\). Step 2: \(\mathbf{C}\boldsymbol\mu+\mathbf{d}=2\cdot1+1\cdot2-1=3\). Step 3: \(\mathbf{C}\boldsymbol\Sigma=(2\cdot4+1\cdot2,\ 2\cdot2+1\cdot2)=(10,\ 6)\). Step 4: \(\mathbf{C}\boldsymbol\Sigma\mathbf{C}^T=10\cdot2+6\cdot1=26\). Step 5: By linearity (p.37), \(u\sim N(3,\ 26)\). **Q2.** Let \(\mathbf{y}=(y_1,y_2,y_3)^T\sim MVN(\boldsymbol\mu,\boldsymbol\Sigma)\) with \(\boldsymbol\mu=\left[\begin{array}{c}0\\1\\-1\end{array}\right]\) and \(\boldsymbol\Sigma=\left[\begin{array}{ccc}3&1&0\\1&2&0\\0&0&5\end{array}\right]\). (a) Give the distribution of \(\tilde{\mathbf{y}}=(y_1,y_3)^T\). (b) Which pairs of components are independent by the Independence property? Answer (a) Step 1: Index set \(\{1,3\}\). (a) Step 2: \(\tilde{\boldsymbol\mu}=(\mu_1,\mu_3)^T=(0,\ -1)^T\). (a) Step 3: Rows and columns 1 and 3: \(\tilde{\boldsymbol\Sigma}=\left[\begin{array}{cc}\Sigma_{11}&\Sigma_{13}\\\Sigma_{31}&\Sigma_{33}\end{array}\right]=\left[\begin{array}{cc}3&0\\0&5\end{array}\right]\). (a) Step 4: By p.38, \(\tilde{\mathbf{y}}\sim MVN\left(\left[\begin{array}{c}0\\-1\end{array}\right],\left[\begin{array}{cc}3&0\\0&5\end{array}\right]\right)\). (b) \(\Sigma_{13}=\Sigma_{31}=0\): \(y_1\) and \(y_3\) are independent (p.40). (b) \(\Sigma_{23}=\Sigma_{32}=0\): \(y_2\) and \(y_3\) are independent (p.40). (b) \(\Sigma_{12}=1\): the property does not apply to \((y_1,y_2)\). **Q3.** Let \(z_1,z_2\overset{iid}{\sim}N(0,1)\), \(\mathbf{A}=\left[\begin{array}{cc}1&0\\1&1\end{array}\right]\), \(\boldsymbol\mu=\left[\begin{array}{c}0\\0\end{array}\right]\), \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\). Find \(\boldsymbol\Sigma\) and \(f(\mathbf{y})\) at \(\mathbf{y}=(1,1)^T\). Answer Step 1: \(\boldsymbol\Sigma=\mathbf{A}\mathbf{A}^T\). \(\Sigma_{11}=1\cdot1+0\cdot0=1\). \(\Sigma_{12}=\Sigma_{21}=1\cdot1+0\cdot1=1\). \(\Sigma_{22}=1\cdot1+1\cdot1=2\). \(\boldsymbol\Sigma=\left[\begin{array}{cc}1&1\\1&2\end{array}\right]\). Step 2: \(\mathbf{y}-\boldsymbol\mu=(1,1)^T\). Step 3: \(|\boldsymbol\Sigma|=1\cdot2-1\cdot1=1\). Step 4: \(\boldsymbol\Sigma^{-1}=\frac11\left[\begin{array}{cc}2&-1\\-1&1\end{array}\right]=\left[\begin{array}{cc}2&-1\\-1&1\end{array}\right]\). Step 5: \(\boldsymbol\Sigma^{-1}(1,1)^T=(2-1,\ -1+1)^T=(1,\ 0)^T\). Step 6: \(q=1\cdot1+1\cdot0=1\). Step 7: Denominator \((2\pi)^1\cdot1^{1/2}=2\pi=6.283185\). Step 8: \(e^{-q/2}=e^{-0.5}=0.606531\). Step 9: \(f=\dfrac{0.606531}{6.283185}=0.096532\). ## 06 · Exercises on the MVN, MLR as an MVN model, and practice Two results from unit 05: the definition (p.36), \(\mathbf{y}=\mathbf{A}\mathbf{z}+\boldsymbol\mu\sim MVN(\boldsymbol\mu,\mathbf{A}\mathbf{A}^T)\) for \(z_i\overset{iid}{\sim}N(0,1)\); and linearity (p.37), \(\mathbf{C}\mathbf{y}+\mathbf{d}\sim MVN(\mathbf{C}\boldsymbol\mu+\mathbf{d},\ \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)\). ### 06.1 Exercise: \(X_2\) has a normal distribution Slides p.41–42 **What (1)** Slides p.41 Exercise · Lecture 5 · p.41 Let r.v.'s \(X_1\sim N(0,1)\) and \(B\sim Bernoulli(0.5)\), independent of \(X_1\). Now consider r.v. \(X_2=\left\{\begin{array}{ll}X_1 & \text{if } B=0\\ -X_1 & \text{if } B=1\end{array}\right.\) \(X_2\) is clearly Normally distributed (we've just swapped half the signs to be opposite, and since it's symmetric, the distribution is unchanged.) "r.v." means random variable. \(B\sim Bernoulli(0.5)\): \(B\) is 0 or 1, with \(P(B=0)=P(B=1)=0.5\) (one fair coin toss). Symmetric: the \(N(0,1)\) density has the same shape on each side of 0. \(X_1\) and \(-X_1\) have the same distribution. \(X_2\) is \(X_1\) with a random sign. The sign change does not change a symmetric distribution. **What (2) · the visual check** Slides p.42 A simulation makes many random draws with a computer and shows their pattern. A density curve is a curve whose area over an interval is the probability of that interval. Slides p.42 shows a density-scale histogram of the draws of \(X_2\), with the \(N(0,1)\) density curve. The bars follow the curve. The highest bars are near 0, at height about 0.4. The \(N(0,1)\) density at 0 is \(1/\sqrt{2\pi}\approx0.3989\). **How** Added A cumulative distribution function (CDF) gives \(P(X\le t)\) for each \(t\). Equal CDFs at all \(t\) mean equal distributions. \(\Phi(t)=P(X_1\le t)\) is the CDF of \(N(0,1)\). Find \(P(X_2\le t)\) for one value of \(t\) - Divide the event into the cases \(B=0\) and \(B=1\). - In each case, replace \(X_2\) by its rule. - Use independence to write each case as a product of two probabilities. - Change \(P(-X_1\le t)\) to \(P(X_1\ge -t)\), then to \(\Phi(t)\) by symmetry. - Add the two cases. **Self-check:** \(P(B=0)+P(B=1)=1\). The result equals \(\Phi(t)\). **Example · \(t=1\)** Added Table value: \(\Phi(1)=0.8413\). Step 1: \(P(X_2\le1)=P(X_2\le1,\ B=0)+P(X_2\le1,\ B=1)\). Step 2: \(=P(X_1\le1,\ B=0)+P(-X_1\le1,\ B=1)\). Step 3: \(=P(X_1\le1)\times0.5+P(-X_1\le1)\times0.5\). Step 4: \(P(-X_1\le1)=P(X_1\ge-1)=1-\Phi(-1)=1-0.1587=0.8413=\Phi(1)\) (Figure 06.1). Step 5: \(P(X_2\le1)=0.8413\times0.5+0.8413\times0.5=0.42065+0.42065=0.8413\). Self-check: \(0.5+0.5=1\); the result equals \(\Phi(1)\). [figure] Figure 06.1 · The \(N(0,1)\) density. Blue: area left of \(-1\), 0.1587. Orange: area right of \(1\), also 0.1587: \(P(X_1\ge-1)=P(X_1\le1)\). **Why** Added 1 \[P(X_2\le t)=P(X_2\le t,\ B=0)+P(X_2\le t,\ B=1)\] \(B\) is 0 or 1. 2 \[=P(X_1\le t,\ B=0)+P(-X_1\le t,\ B=1)\] The rule for \(X_2\). 3 \[=P(X_1\le t)P(B=0)+P(-X_1\le t)P(B=1)\] \(B\) and \(X_1\) are independent. 4 \[=\tfrac12\,\Phi(t)+\tfrac12\,P(-X_1\le t)\] 5 \[=\tfrac12\,\Phi(t)+\tfrac12\,P(X_1\le t)\] Symmetry. 6 \[=\tfrac12\,\Phi(t)+\tfrac12\,\Phi(t)=\Phi(t)\] \(X_2\sim N(0,1)\). ∎ ### 06.2 Exercise continued: \(Y=X_1+X_2\) is not normal Slides p.43–44 **What** Slides p.43–44 Exercise continued · Lecture 5 · p.43–44 Now let \(Y=X_1+X_2\). Is \(Y\) Normal? No! How do we know? • \(P(Y=0)=P(B=1)=1/2\) What's going on here? • While \(X_1\) and \(X_2\) are each normally distributed, \(\mathbf{x}=(X_1,X_2)^T\) is not multivariate normal. • I.e. cannot find \(\mathbf{A}_{2\times k}\) s.t. \(\mathbf{x}=\mathbf{A}\mathbf{z}\) for \(\mathbf{z}=(Z_1,\dots,Z_k)^T\), \(Z_i\overset{iid}{\sim}N(0,1)\) If \(B=1\), \(X_2=-X_1\) and \(Y=X_1-X_1=0\). This case has probability 1/2. A normal variable has probability 0 at each single value if its variance is positive, and 1 if its variance is 0 (a constant). \(P(Y=0)=1/2\) is neither 0 nor 1. \(Y\) is not normal. \(Y=(1,1)\,\mathbf{x}\) is a linear combination of \(\mathbf{x}\). If \(\mathbf{x}\) were MVN, linearity (p.37) would make \(Y\) normal. \(\mathbf{x}\) is not MVN. "cannot find \(\mathbf{A}_{2\times k}\)": no constant \(2\times k\) matrix \(\mathbf{A}\) gives \(\mathbf{x}=\mathbf{A}\mathbf{z}\) (p.36). **How** Added Use linearity to show that a vector \(\mathbf{x}\) is not MVN - Select a constant row vector \(\mathbf{C}\). Let \(u=\mathbf{C}\mathbf{x}\), for example \(\mathbf{C}=(1,1)\), \(u=Y\). - Assume that \(\mathbf{x}\) is MVN. Then \(u\) is normal (p.37, \(\mathbf{d}=0\)). - Calculate one probability of \(u\), for example \(P(Y=0)\) over the cases \(B=0\) and \(B=1\). - If the result is neither 0 nor 1, the assumption in step 2 is false. **Self-check:** \(\mathbf{C}\) is constant: no \(B\) or \(X_1\) in it. \(P(B=0)+P(B=1)=1\). **Example · Six hand-made values** Added Data: \(X_1=(1.2,\ -0.5,\ 0.8,\ -1.5,\ 2.0,\ 0.3)\) and \(B=(0,\ 1,\ 1,\ 0,\ 1,\ 0)\). Step 1: \(\mathbf{C}=(1,1)\): \(Y=X_1+X_2\). Step 2: Assume \(\mathbf{x}\) is MVN. Then \(Y\) is normal. Step 3a, \(B=0\) (rows 1, 4, 6): \(X_2=X_1\), \(Y=2X_1\): \(2\times1.2=2.4\), \(2\times(-1.5)=-3.0\), \(2\times0.3=0.6\). Step 3b, \(B=1\) (rows 2, 3, 5): \(X_2=-X_1\): \(0.5\), \(-0.8\), \(-2.0\). \(Y\): \(-0.5+0.5=0\), \(0.8-0.8=0\), \(2.0-2.0=0\). Step 3c: \(Y=(2.4,\ 0,\ 0,\ -3.0,\ 0,\ 0.6)\). Each row with \(B=1\) gives \(Y=0\) exactly. Step 4: \(P(Y=0)=P(B=1)=1/2\), neither 0 nor 1. \(\mathbf{x}\) is not MVN. [figure] Figure 06.2 · Distribution of \(Y=X_1+X_2\). Purple bar: probability 1/2 at the single value 0 (\(B=1\)). Blue curve: the spread-out part \(Y=2X_1\) (\(B=0\)). A normal distribution has no probability at a single value. **Why** Added First \(P(Y=0)\), then no matrix \(\mathbf{A}\) exists. 1 \[P(Y=0)=P(Y=0,\ B=0)+P(Y=0,\ B=1)\] \(B\) is 0 or 1. 2 \[=P(X_1+X_1=0,\ B=0)+P(X_1-X_1=0,\ B=1)\] The rule for \(X_2\). 3 \[=P(2X_1=0,\ B=0)+P(0=0,\ B=1)\] 4 \[=P(X_1=0)P(B=0)+P(B=1)\] Independence; \(0=0\) is always true. 5 \[=0\times\tfrac12+\tfrac12=\tfrac12\] \(P(X_1=0)=0\). ∎ 1 \[\text{Assume } \mathbf{x}=\mathbf{A}\mathbf{z},\quad \mathbf{A}_{2\times k},\ Z_i\overset{iid}{\sim}N(0,1)\] Then \(\mathbf{x}\sim MVN(\mathbf{0},\mathbf{A}\mathbf{A}^T)\) (p.36). 2 \[Y=(1,1)\,\mathbf{x}\] 3 \[Y\sim MVN\big((1,1)\mathbf{0},\ (1,1)\mathbf{A}\mathbf{A}^T(1,1)^T\big)\] Linearity (p.37), \(\mathbf{d}=0\). 4 \[Y\sim N\big(0,\ (1,1)\mathbf{A}\mathbf{A}^T(1,1)^T\big)\] One-component MVN is normal. 5 \[P(Y=0)\in\{0,\ 1\}\] 0 if variance positive; 1 if 0. 6 \[P(Y=0)=\tfrac12\notin\{0,\ 1\}\] Contradiction: no \(\mathbf{A}_{2\times k}\) exists. ∎ MVN is a property of the full vector, not of each component. The regression model assumes that the full error vector is MVN. ### 06.3 Multiple linear regression as an MVN model Slides p.45 **What** Slides p.45 Multiple Linear Regression · Lecture 5 · p.45 \[y_i=\beta_0+\beta_1x_{i1}+\dots+\beta_px_{ip}+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] We can write this: \[\left[\begin{array}{c}y_1\\y_2\\\vdots\\y_n\end{array}\right]=\left[\begin{array}{ccccc}1&x_{11}&x_{12}&\dots&x_{1p}\\1&x_{21}&x_{22}&\dots&x_{2p}\\\vdots&\vdots&\vdots&\vdots&\vdots\\1&x_{n1}&x_{n2}&\dots&x_{np}\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\vdots\\\beta_p\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\vdots\\\epsilon_n\end{array}\right]\] Or more simply: \[\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon,\quad \boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\quad\iff\quad \mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\] Page 45 repeats the matrix form of page 8 (01.6) and adds the MVN statement. \(\sigma^2\mathbf{I}\) has \(\sigma^2\) on the diagonal and 0 elsewhere: each error has variance \(\sigma^2\); each pair has covariance 0. \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\) gives \(\boldsymbol\epsilon=\sigma\mathbf{I}\mathbf{z}\): MVN by p.36. \(\mathbf{y}\) is \(\boldsymbol\epsilon\) plus the constant \(\mathbf{X}\boldsymbol\beta\). By linearity, \(\mathbf{y}\) is MVN with mean \(\mathbf{X}\boldsymbol\beta\) and the same covariance. **How** Added Write the distribution of \(\mathbf{y}\) from the distribution of \(\boldsymbol\epsilon\) - Write \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\). - Write \(\mathbf{y}=\mathbf{C}\boldsymbol\epsilon+\mathbf{d}\), with \(\mathbf{C}=\mathbf{I}\), \(\mathbf{d}=\mathbf{X}\boldsymbol\beta\). - Mean: \(\mathbf{C}\mathbf{0}+\mathbf{d}=\mathbf{X}\boldsymbol\beta\). - Covariance: \(\mathbf{C}(\sigma^2\mathbf{I})\mathbf{C}^T=\mathbf{I}\sigma^2\mathbf{I}\mathbf{I}=\sigma^2\mathbf{I}\). - Write \(\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\). **Self-check:** \(\mathbf{d}\) is constant. The mean is \(n\times1\). The covariance is \(n\times n\), \(\sigma^2\) on the diagonal, 0 elsewhere. **Example · A small data set** Added \(n=3\), \(p=2\), \((x_{i1},x_{i2})=(2,1),\ (3,0),\ (5,4)\). Step 1: \(\left[\begin{array}{c}y_1\\y_2\\y_3\end{array}\right]=\left[\begin{array}{ccc}1&2&1\\1&3&0\\1&5&4\end{array}\right]\left[\begin{array}{c}\beta_0\\\beta_1\\\beta_2\end{array}\right]+\left[\begin{array}{c}\epsilon_1\\\epsilon_2\\\epsilon_3\end{array}\right]\), \(\boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\). Step 2: \(\mathbf{C}=\mathbf{I}\) (\(3\times3\)), \(\mathbf{d}=\mathbf{X}\boldsymbol\beta\). Step 3: \(\mathbf{X}\boldsymbol\beta=\left[\begin{array}{c}\beta_0+2\beta_1+1\beta_2\\\beta_0+3\beta_1+0\beta_2\\\beta_0+5\beta_1+4\beta_2\end{array}\right]=\left[\begin{array}{c}\beta_0+2\beta_1+\beta_2\\\beta_0+3\beta_1\\\beta_0+5\beta_1+4\beta_2\end{array}\right]\). Step 4: \(\sigma^2\mathbf{I}=\left[\begin{array}{ccc}\sigma^2&0&0\\0&\sigma^2&0\\0&0&\sigma^2\end{array}\right]\). Step 5: \(\left[\begin{array}{c}y_1\\y_2\\y_3\end{array}\right]\sim MVN\left(\left[\begin{array}{c}\beta_0+2\beta_1+\beta_2\\\beta_0+3\beta_1\\\beta_0+5\beta_1+4\beta_2\end{array}\right],\ \left[\begin{array}{ccc}\sigma^2&0&0\\0&\sigma^2&0\\0&0&\sigma^2\end{array}\right]\right)\). **Self-check:** mean \(3\times1\), covariance \(3\times3\). **Self-check:** the marginal property (p.38) keeps only \(y_1\): \(y_1\sim N(\beta_0+2\beta_1+\beta_2,\ \sigma^2)\), the model for one observation. **Why** Added Lines 1–3 give "⟹"; lines 4–6 give "⟸". Both use linearity (p.37). 1 \[\mathbf{y}=\mathbf{I}\boldsymbol\epsilon+\mathbf{X}\boldsymbol\beta,\quad \boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\] 2 \[\mathbf{y}\sim MVN\big(\mathbf{I}\mathbf{0}+\mathbf{X}\boldsymbol\beta,\ \mathbf{I}(\sigma^2\mathbf{I})\mathbf{I}^T\big)\] Linearity, \(\mathbf{C}=\mathbf{I}\), \(\mathbf{d}=\mathbf{X}\boldsymbol\beta\). 3 \[\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\ \sigma^2\mathbf{I})\] \(\mathbf{I}\mathbf{0}=\mathbf{0}\), \(\mathbf{I}^T=\mathbf{I}\), \(\mathbf{I}\mathbf{I}=\mathbf{I}\). ⟹ 4 \[\boldsymbol\epsilon=\mathbf{I}\mathbf{y}-\mathbf{X}\boldsymbol\beta,\quad \mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\] 5 \[\boldsymbol\epsilon\sim MVN\big(\mathbf{I}\mathbf{X}\boldsymbol\beta-\mathbf{X}\boldsymbol\beta,\ \mathbf{I}(\sigma^2\mathbf{I})\mathbf{I}^T\big)\] Linearity, \(\mathbf{C}=\mathbf{I}\), \(\mathbf{d}=-\mathbf{X}\boldsymbol\beta\). 6 \[\boldsymbol\epsilon\sim MVN(\mathbf{0},\ \sigma^2\mathbf{I})\] ⟸ ∎ Later lectures use \(\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\) to find the distribution of \(\hat{\boldsymbol\beta}\). ### 06.4 Practice Q1: mean vectors and covariance matrices Slides p.46–47 Slides p.46 is the title page "Practice". Slides p.47–48 give the two questions. **What** Slides p.47 Practice Q1 · Lecture 5 · p.47 Let \(\mathbf{y}=(y_1,y_2,y_3)^T\) such that \(E[\mathbf{y}]=(1,2,2)^T\) and \(Var(\mathbf{y})=\left[\begin{array}{ccc}4&0&1\\0&3&-1\\1&-1&4\end{array}\right]\) and let \(\mathbf{a}=(1,2,3)^T\) and \(\mathbf{A}=\left[\begin{array}{ccc}1&-1&0\\0&1&-1\\1&0&-1\\1&1&1\end{array}\right]\). Compute \(E[\mathbf{a}^T\mathbf{y}]\), \(E[\mathbf{A}\mathbf{y}]\), \(Var(\mathbf{a}^T\mathbf{y})\), \(Var(\mathbf{A}\mathbf{y})\) \(\mathbf{a}^T\mathbf{y}=y_1+2y_2+3y_3\) is one number. \(\mathbf{A}\mathbf{y}\) is \(4\times1\). Formulas: \(E[\mathbf{a}^T\mathbf{y}+b]=\mathbf{a}^TE[\mathbf{y}]+b\), \(Var(\mathbf{a}^T\mathbf{y})=\mathbf{a}^TVar(\mathbf{y})\mathbf{a}\) (p.28–29); \(E[\mathbf{A}\mathbf{y}]=\mathbf{A}E[\mathbf{y}]\), \(Var(\mathbf{A}\mathbf{y})=\mathbf{A}Var(\mathbf{y})\mathbf{A}^T\) (p.31). **How** Added Let \(\mathbf{V}=Var(\mathbf{y})\). Practice Q1 - \(E[\mathbf{a}^T\mathbf{y}]\): calculate \(\mathbf{a}^TE[\mathbf{y}]\) (\(b=0\)). - \(E[\mathbf{A}\mathbf{y}]\): do step 1 with each row of \(\mathbf{A}\). - \(Var(\mathbf{a}^T\mathbf{y})\): calculate \(\mathbf{V}\mathbf{a}\) (\(3\times1\)). - Calculate \(\mathbf{a}^T(\mathbf{V}\mathbf{a})\). - \(Var(\mathbf{A}\mathbf{y})\): calculate \(\mathbf{A}\mathbf{V}\) (\(4\times3\)). - Row \(i\) of \(\mathbf{A}\mathbf{V}\): add the rows of \(\mathbf{V}\), with row \(i\) of \(\mathbf{A}\) as weights. - Calculate \((\mathbf{A}\mathbf{V})\mathbf{A}^T\) (\(4\times4\)). - Entry \((i,j)\): row \(i\) of \(\mathbf{A}\mathbf{V}\) times row \(j\) of \(\mathbf{A}\), term by term, added. **Self-check:** \(Var(\mathbf{a}^T\mathbf{y})\ge0\) (p.25). \(Var(\mathbf{A}\mathbf{y})\) is symmetric (p.24) with diagonal \(\ge0\). **Example · the full calculation** Added Answer: Practice Q1 Step 1: \(E[\mathbf{a}^T\mathbf{y}]=\mathbf{a}^TE[\mathbf{y}]=1\times1+2\times2+3\times2=1+4+6=11\). Step 2, row 1: \(1\times1+(-1)\times2+0\times2=1-2+0=-1\). Step 2, row 2: \(0\times1+1\times2+(-1)\times2=0+2-2=0\). Step 2, row 3: \(1\times1+0\times2+(-1)\times2=1+0-2=-1\). Step 2, row 4: \(1\times1+1\times2+1\times2=1+2+2=5\). \(E[\mathbf{A}\mathbf{y}]=(-1,\ 0,\ -1,\ 5)^T\). Step 3: \(\mathbf{V}\mathbf{a}=\left[\begin{array}{c}4\times1+0\times2+1\times3\\0\times1+3\times2+(-1)\times3\\1\times1+(-1)\times2+4\times3\end{array}\right]=\left[\begin{array}{c}4+0+3\\0+6-3\\1-2+12\end{array}\right]=\left[\begin{array}{c}7\\3\\11\end{array}\right]\). Step 4: \(Var(\mathbf{a}^T\mathbf{y})=\mathbf{a}^T\mathbf{V}\mathbf{a}=1\times7+2\times3+3\times11=7+6+33=46\). Step 5: Rows of \(\mathbf{V}\): \(\mathbf{v}_1=(4,0,1)\), \(\mathbf{v}_2=(0,3,-1)\), \(\mathbf{v}_3=(1,-1,4)\). Step 6, row \((1,-1,0)\): \(\mathbf{v}_1-\mathbf{v}_2=(4-0,\ 0-3,\ 1+1)=(4,-3,2)\). Step 6, row \((0,1,-1)\): \(\mathbf{v}_2-\mathbf{v}_3=(0-1,\ 3+1,\ -1-4)=(-1,4,-5)\). Step 6, row \((1,0,-1)\): \(\mathbf{v}_1-\mathbf{v}_3=(4-1,\ 0+1,\ 1-4)=(3,1,-3)\). Step 6, row \((1,1,1)\): \(\mathbf{v}_1+\mathbf{v}_2+\mathbf{v}_3=(4+0+1,\ 0+3-1,\ 1-1+4)=(5,2,4)\). Steps 7–8, row \((4,-3,2)\) with the rows of \(\mathbf{A}\): \(4+3+0=7\); \(0-3-2=-5\); \(4+0-2=2\); \(4-3+2=3\). Steps 7–8, row \((-1,4,-5)\): \(-1-4+0=-5\); \(0+4+5=9\); \(-1+0+5=4\); \(-1+4-5=-2\). Steps 7–8, row \((3,1,-3)\): \(3-1+0=2\); \(0+1+3=4\); \(3+0+3=6\); \(3+1-3=1\). Steps 7–8, row \((5,2,4)\): \(5-2+0=3\); \(0+2-4=-2\); \(5+0-4=1\); \(5+2+4=11\). \[Var(\mathbf{A}\mathbf{y})=\left[\begin{array}{cccc}7&-5&2&3\\-5&9&4&-2\\2&4&6&1\\3&-2&1&11\end{array}\right]\] Self-check: \(46\ge0\). Symmetric: entries \((1,2)\) and \((2,1)\) are both \(-5\). Self-check: diagonal 7, 9, 6, 11 are all \(\ge0\). **Why** Slides p.34–35 Section 04.6, Why (2), derives \(Var(\mathbf{A}\mathbf{y})\). \(Var(\mathbf{a}^T\mathbf{y})\) is the case \(\mathbf{A}=\mathbf{a}^T\) (04.7, Q3). ### 06.5 Practice Q2: the cross-covariance matrix Slides p.48 **What** Slides p.48 Practice Q2 · Lecture 5 · p.48 Suppose \(\mathbf{x}=(x_1,\dots,x_n)^T\) is a random vector with \(E[\mathbf{x}]=\boldsymbol\mu\) and \(var(\mathbf{x})=\mathbf{V}\). Let \(\mathbf{y}=\mathbf{A}_{p\times n}\mathbf{x}\) and \(\mathbf{w}=\mathbf{B}_{q\times n}\mathbf{x}\). Let \(\mathbf{C}\) denote the "cross-covariance" matrix, where \(\mathbf{C}_{ij}=cov(y_i,w_j)\). (You can think of this as the off-diagonal block of the variance matrix of \((\mathbf{y}^T,\mathbf{w}^T)^T\), where the diagonal blocks would be \(var(\mathbf{y})\) and \(var(\mathbf{w})\).) Write \(\mathbf{C}\) in terms of \(\boldsymbol\mu,\mathbf{V},\mathbf{A},\mathbf{B}\) The cross-covariance matrix \(\mathbf{C}\) (\(p\times q\)) has entry \((i,j)\) equal to \(cov(y_i,w_j)\). Stack \(\mathbf{y}\) above \(\mathbf{w}\). The covariance matrix of this \((p+q)\times1\) vector has four blocks: \(var(\mathbf{y})\) top-left, \(var(\mathbf{w})\) bottom-right, \(\mathbf{C}\) top-right. Formula from p.26, with \(\mathbf{x}\) as the random vector: \(cov(\mathbf{a}^T\mathbf{x}+c,\ \mathbf{b}^T\mathbf{x}+d)=\mathbf{a}^T\mathbf{V}\mathbf{b}\). **How** Added Practice Q2 - Let \(\mathbf{a}_i^T\) be row \(i\) of \(\mathbf{A}\) and \(\mathbf{b}_j^T\) row \(j\) of \(\mathbf{B}\). - Write \(y_i=\mathbf{a}_i^T\mathbf{x}\) and \(w_j=\mathbf{b}_j^T\mathbf{x}\). - Use p.26 with \(c=d=0\): \(\mathbf{C}_{ij}=\mathbf{a}_i^T\mathbf{V}\mathbf{b}_j\). - \(\mathbf{b}_j\) is column \(j\) of \(\mathbf{B}^T\). - Write \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{B}^T\). **Self-check:** \((p\times n)(n\times n)(n\times q)=p\times q\). No \(\boldsymbol\mu\). If \(\mathbf{B}=\mathbf{A}\), the answer is \(\mathbf{A}\mathbf{V}\mathbf{A}^T\). **Example · The numbers of Q1** Added Use the random vector and \(\mathbf{V}\) of Q1 as \(\mathbf{x}\) (\(n=3\)), the \(4\times3\) \(\mathbf{A}\) of Q1 (\(p=4\)), and \(\mathbf{B}=\mathbf{a}^T=(1,2,3)\) (\(q=1\)). Answer: Practice Q2 and a numerical example General answer: \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{B}^T\). No \(\boldsymbol\mu\). Step 1: \(\mathbf{B}^T=\mathbf{a}=(1,2,3)^T\): \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{a}\) (\(4\times1\)). Step 2: Q1 step 3: \(\mathbf{V}\mathbf{a}=(7,3,11)^T\). Step 3, row \((1,-1,0)\): \(7-3+0=4\). Step 3, row \((0,1,-1)\): \(0+3-11=-8\). Step 3, row \((1,0,-1)\): \(7+0-11=-4\). Step 3, row \((1,1,1)\): \(7+3+11=21\). \(\mathbf{C}=(4,\ -8,\ -4,\ 21)^T\). Step 4: Stack \(\mathbf{A}\) above \(\mathbf{a}^T\): a \(5\times3\) matrix \(\mathbf{S}\). \(\mathbf{S}\mathbf{V}\mathbf{S}^T\) is the covariance matrix of \((\mathbf{y}^T,w)^T\). \[\mathbf{S}\mathbf{V}\mathbf{S}^T=\left[\begin{array}{ccccc}7&-5&2&3&4\\-5&9&4&-2&-8\\2&4&6&1&-4\\3&-2&1&11&21\\4&-8&-4&21&46\end{array}\right]\] Top-left \(4\times4\) block: \(Var(\mathbf{A}\mathbf{y})\) (Q1). Bottom-right 46: \(Var(\mathbf{a}^T\mathbf{y})\) (Q1). Column 5, first four entries: \((4,-8,-4,21)=\mathbf{C}\). Self-check: size \(4\times1=p\times q\). **Why · \(\mathbf{C}=\mathbf{A}\mathbf{V}\mathbf{B}^T\)** Added First by entries, then by matrices. 1 \[\mathbf{C}_{ij}=cov(y_i,w_j)\] 2 \[=cov(\mathbf{a}_i^T\mathbf{x},\ \mathbf{b}_j^T\mathbf{x})\] Rows of \(\mathbf{A}\) and \(\mathbf{B}\). 3 \[=\mathbf{a}_i^T\mathbf{V}\mathbf{b}_j\] p.26, \(c=d=0\). 4 \[=\sum_{k=1}^n\sum_{l=1}^n A_{ik}V_{kl}B_{jl}\] 5 \[=\sum_{k=1}^n\sum_{l=1}^n A_{ik}V_{kl}(B^T)_{lj}\] \(B_{jl}=(B^T)_{lj}\). 6 \[=[\mathbf{A}\mathbf{V}\mathbf{B}^T]_{ij}\] Matrix product, all \(i,j\). ∎ 1 \[\mathbf{C}=E[(\mathbf{y}-E[\mathbf{y}])(\mathbf{w}-E[\mathbf{w}])^T]\] Entry \((i,j)\) is \(cov(y_i,w_j)\). 2 \[=E[(\mathbf{A}\mathbf{x}-\mathbf{A}\boldsymbol\mu)(\mathbf{B}\mathbf{x}-\mathbf{B}\boldsymbol\mu)^T]\] \(E[\mathbf{A}\mathbf{x}]=\mathbf{A}\boldsymbol\mu\) (p.31). 3 \[=E[\mathbf{A}(\mathbf{x}-\boldsymbol\mu)(\mathbf{x}-\boldsymbol\mu)^T\mathbf{B}^T]\] 4 \[=\mathbf{A}E[(\mathbf{x}-\boldsymbol\mu)(\mathbf{x}-\boldsymbol\mu)^T]\mathbf{B}^T\] 5 \[=\mathbf{A}\mathbf{V}\mathbf{B}^T\] Line 3 removes \(\boldsymbol\mu\). ∎ ### 06.6 Practice Added **P1.** Use the data of Section 06.3: \((x_{i1},x_{i2})=(2,1),\ (3,0),\ (5,4)\). Assume \(\mathbf{y}=\mathbf{X}\boldsymbol\beta+\boldsymbol\epsilon\), \(\boldsymbol\epsilon\sim MVN(\mathbf{0},\sigma^2\mathbf{I})\). Find the distribution of \(y_1-y_2\). Answer Step 1: By 06.3, \(\mathbf{y}\sim MVN(\mathbf{X}\boldsymbol\beta,\sigma^2\mathbf{I})\), \(\mathbf{X}\boldsymbol\beta=(\beta_0+2\beta_1+\beta_2,\ \beta_0+3\beta_1,\ \beta_0+5\beta_1+4\beta_2)^T\). Step 2: \(y_1-y_2=\mathbf{C}\mathbf{y}\), \(\mathbf{C}=(1,-1,0)\), \(\mathbf{d}=0\). Step 3 (mean): \(\mathbf{C}\mathbf{X}\boldsymbol\beta=(\beta_0+2\beta_1+\beta_2)-(\beta_0+3\beta_1)=(2-3)\beta_1+\beta_2=-\beta_1+\beta_2\). Step 4 (variance): \(\mathbf{C}(\sigma^2\mathbf{I})\mathbf{C}^T=\sigma^2(1^2+(-1)^2+0^2)=\sigma^2(1+1+0)=2\sigma^2\). Result: \(y_1-y_2\sim N(-\beta_1+\beta_2,\ 2\sigma^2)\) (p.37). \(\beta_0\) cancels. **P2.** Use \(\mathbf{y}\), \(E[\mathbf{y}]\), \(Var(\mathbf{y})\) from Practice Q1. Let \(u=2y_1-y_2+1\). Calculate \(E[u]\), \(Var(u)\), \(cov(u,y_3)\). Answer Step 1: \(u=\mathbf{b}^T\mathbf{y}+1\), \(\mathbf{b}=(2,-1,0)^T\). Step 2: \(E[u]=\mathbf{b}^T\boldsymbol\mu+1=2\times1+(-1)\times2+0\times2+1=2-2+0+1=1\). Step 3: \(\mathbf{V}\mathbf{b}=(4\times2+0\times(-1)+1\times0,\ 0\times2+3\times(-1)+(-1)\times0,\ 1\times2+(-1)\times(-1)+4\times0)^T=(8,-3,3)^T\). Step 4: \(Var(u)=\mathbf{b}^T\mathbf{V}\mathbf{b}=2\times8+(-1)\times(-3)+0\times3=16+3+0=19\). The constant 1 does not change the variance. Step 5: \(y_3=\mathbf{e}_3^T\mathbf{y}\), \(\mathbf{e}_3=(0,0,1)^T\). \(cov(u,y_3)=\mathbf{b}^T\mathbf{V}\mathbf{e}_3\) (p.26). Step 6: \(\mathbf{V}\) is symmetric: \(\mathbf{b}^T\mathbf{V}\mathbf{e}_3\) is component 3 of \(\mathbf{V}\mathbf{b}\). \(cov(u,y_3)=3\).