This lecture estimates \(\beta_0\), \(\beta_1\), and \(\sigma^2\) in the simple linear regression model with least squares and maximum likelihood (Lecture 2 · p.1–38).
Unit 01Recap: mean, variance, covariance, and correlation
The first five people of the brainhead data (Slides p.10).
\(x_i\): head size (cm³). \(y_i\): brain weight (g).
\(x_1,\dots,x_5\): 4512, 3738, 4261, 3777, 4177.
\(y_1,\dots,y_5\): 1530, 1297, 1335, 1282, 1590.
01.1Title, section, housekeeping Slides p.1–3
Page 1: Lecture 2: Simple Linear Regression: Estimation.
Page 2: section "Recap" (pages 3 to 8).
Page 3: review the course outline. Assignment 1 is due on September 25.
Ask questions on Piazza. Office hours: Wednesdays, 8:30–9:30 and 11:30–12:30.
01.2Mean and expectation Slides p.4
WhatSlides p.4
Review: Mean · Lecture 2 · p.4
Mean/Expectation:
\[E[Y]=\int y f(y)\,dy\]
Linearity of expectation: For random variables \(Y_1,\dots,Y_m\) and constants \(a_i,b_i\) for \(i=1,\dots,m\):
\[E\left[\sum_{i=1}^m (a_iY_i+b_i)\right]=\sum_{i=1}^m a_iE[Y_i]+\sum_{i=1}^m b_i\]
For observations \(y_1,\dots,y_n\), the sample mean is:
\[\bar y=\frac1n\sum_i y_i\]
A random variable \(Y\) is a quantity whose value comes from chance. \(y\) is one value of \(Y\).
The probability density function (PDF) \(f(y)\) shows how probable values near \(y\) are. It is not negative, and \(\int f(y)\,dy=1\).
The expectation \(E[Y]\) is the population mean of \(Y\).
A constant, such as \(a_i\) or \(b_i\), is a fixed number.
Linearity: constant factors move outside \(E\), sums split, and \(E\) of a constant is the constant.
The observations \(y_1,\dots,y_n\) are the data. \(n\) is the sample size.
The sample mean \(\bar y\) ("y bar") estimates \(E[Y]\) from the data.
HowAdded
Calculate the sample mean \(\bar y\)
Add the \(n\) observations.
Divide the sum by \(n\).
Self-check: \(\bar y\) is between the smallest and largest observation. It has the unit of \(y_i\).
Figure 1-1 (Added) · Slides p.10 data. Point labels are \(i\). Orange dashed lines: \(\bar x\), \(\bar y\). Blue points have a positive product \((y_i-\bar y)(x_i-\bar x)\). Purple point 3 has a negative product.
33340.75 cm³·g is positive: people with larger heads usually have heavier brains.
WhyAdded
Proof of the second form. The constants are \(\mu_X=E[X]\) and \(\mu_Y=E[Y]\).
1
\(\mathrm{cov}[X,Y]=E[(X-\mu_X)(Y-\mu_Y)]\)
2
\(=E[XY-\mu_YX-\mu_XY+\mu_X\mu_Y]\)
3
\(=E[XY]-\mu_YE[X]-\mu_XE[Y]+\mu_X\mu_Y\)
linearity
4
\(=E[XY]-\mu_Y\mu_X-\mu_X\mu_Y+\mu_X\mu_Y\)
substitute \(E[X]=\mu_X,\ E[Y]=\mu_Y\)
5
\(=E[XY]-\mu_X\mu_Y\)
6
\(=E[XY]-E[X]E[Y]\)
Proof of \(\mathrm{cov}(aY+c,bX+d)=ab\,\mathrm{cov}(X,Y)\). Linearity gives \(E[aY+c]=a\mu_Y+c\) and \(E[bX+d]=b\mu_X+d\).
Recap: Goals and Notation · Lecture 2 · p.8
Possible goals:
• Characterize the relationship between outcome \(y\) and covariate \(x\)
• Predict \(y\) given \(x\)
For the \(i\)th observation:
\(y_i\) is the outcome
\(x_i\) is a covariate
\(i=1,\dots,n\) indexes the observations in the sample
The outcome \(y_i\) is the quantity to explain or predict.
We use the covariate \(x_i\) to explain or predict \(y_i\).
To characterize the relationship is to describe how \(y\) changes with \(x\).
HowAdded
Match a data set to the notation
Call the quantity to explain or predict \(y\).
Call the quantity that explains it \(x\).
Count the rows to get \(n\). Each row is one \(i\).
Self-check: Each \(i\) has one \(y_i\) and one \(x_i\), from the same person.
ExampleAdded
\(y\): brain weight (g). \(x\): head size (cm³).
\(n=5\). Row 1 is \(i=1\): \(y_1=1530\) and \(x_1=4512\).
WhyAdded
Notation only. No derivation.
01.7Practice Added
Q1. Change brain weight to kilograms: \(y^*=y/1000\). Use \(S_{xy}=133363\) and \(r=0.7068\). Find \(S_{xy^*}\) and \(r^*\) for \((y^*,x)\).
Answer
\(y_i^*-\bar y^*=(y_i-\bar y)/1000\). Then \(S_{xy^*}=S_{xy}/1000=133363/1000=133.363\).
The numerator and the denominator both get the factor \(1/1000\) (Slides p.5, p.6). They cancel: \(r^*=r=0.7068\).
Q2. For the five people, \(s_x^2=109180.5\), \(s_y^2=20381.7\), and the sample covariance is 33340.75. Use Slides p.6. Find the sample variance of \(x+y\).
Use \(\mathrm{cov}(aY+c,bX+d)=ab\,\mathrm{cov}(X,Y)\) with \(a=2,\ c=1,\ b=3,\ d=-5\).
\(\mathrm{cov}(2Y+1,3X-5)=2\times3\times3=18\).
Unit 02The simple linear regression model and its coefficients
Plan · Slides p.9–18
Slides p.10: first 6 rows of the data.
Slides p.11: histograms of \(x\) and \(y\).
Slides p.12: scatterplot of \(y\) against \(x\).
Slides p.13: the model: parameters, errors, data.
Slides p.14–16: meaning of \(\beta_0\) and \(\beta_1\).
Slides p.17: the model as one picture (figure Added).
Slides p.18: four assumptions and the estimation question.
02.1Title page Slides p.9
Title: Lecture 2: Simple Linear Regression: Estimation.
Slides p.1–8 are a review (Unit 01). New material starts at Slides p.10.
02.2Data example: the brainhead data Slides p.10
WhatSlides p.10
Data Example · Lecture 2 · p.10
The slide gives two columns of the brainhead data: \(x\) = head size, \(y\) = brain weight. It prints the first 6 rows:
row \(i\)
\(x\)
\(y\)
1
4512
1530
2
3738
1297
3
4261
1335
4
3777
1282
5
4177
1590
6
3585
1300
An observation is one row of the data: one person.
A variable is one column of the data: one type of measurement.
Head size \(x\): head volume, in cubic centimetres (cm³).
Brain weight \(y\): in grams (g).
The brainhead data has \(n=236\) rows (Lecture 1 · p.4).
HowAdded
Read one row of the data
Call the column that explains \(x\).
Call the column to explain \(y\).
Select a row \(i\).
Read \(x_i\) and \(y_i\), with units.
Self-check: The \(x\) and \(y\) columns both have \(n\) rows.
Example · Rows 1 and 2Added
Steps 1–2: \(x\) is head size. \(y\) is brain weight.
Steps 3–4, row 1: \(x_1=4512\) cm³ and \(y_1=1530\) g.
Steps 3–4, row 2: \(x_2=3738\) cm³ and \(y_2=1297\) g.
Why · The names \(x\) and \(y\)Added
Each formula symbol refers to one column. For example, \(x_1=4512\) is the first value in the \(x\) column.
02.3Exploratory data analysis: histograms Slides p.11
WhatSlides p.11
Exploratory Data Analysis · Lecture 2 · p.11
Two histograms: "Head Size (cm^3)" and "Brain Weight (grams)". The vertical axis is "Frequency".
Exploratory data analysis (EDA) uses plots and simple summaries to look at the data before modelling.
A histogram divides the number line into equal intervals (bins). Each bin has one bar.
Frequency: the number of observations in a bin. It is the height of the bar.
HowAdded
Read a histogram
Read the variable and its unit on the horizontal axis.
Read the bin width.
Read the height of each bar.
Find the highest bar.
Compare the two sides of the highest bar.
Self-check: The bar heights add to \(n=236\).
Example · The brainhead dataAdded
The bar heights come from a count of the 236 rows in the data file.
Figure 2-1 (Slides p.11, redrawn from the data) · Left: head size \(x\), bin width 200 cm³. Right: brain weight \(y\), bin width 50 g.
Head size, bar heights from left to right: 1, 6, 19, 41, 48, 45, 36, 24, 8, 7, 1.
Brain weight, bar heights from left to right: 3, 10, 17, 25, 45, 37, 40, 22, 18, 6, 7, 4, 2.
Head size: the highest bar, 3400 to 3600 cm³, has 48 people. Range: 2773 to 4747 cm³ (Lecture 1 · p.23).
Brain weight: the highest bar, 1200 to 1250 g, has 45 people. Range: 1012 to 1635 g (Lecture 1 · p.23).
Shape: both are high in the middle and low at the sides. Both are approximately symmetric.
Why · The sum of the bar heights is \(n\)Added
Each observation is in exactly one bin. The bar heights count each person one time.
1
\[\text{sum for } x=(1+6+19+41)+(48+45+36)+(24+8+7+1)\]
2
\[=67+129+40\]
3
\[=236=n\]
4
\[\text{sum for } y=(3+10+17+25)+(45+37+40)+(22+18+6+7+4+2)\]
5
\[=55+122+59\]
6
\[=236=n\]
02.4Exploratory data analysis: scatterplot Slides p.12
WhatSlides p.12
EDA · Lecture 2 · p.12
Scatterplot: Brain Weight (grams) against Head Size (cm^3).
A scatterplot has one point \((x_i,y_i)\) for each observation. The brainhead data gives 236 points.
"\(y\) against \(x\)": \(y\) is on the vertical axis, \(x\) on the horizontal axis.
HowAdded
Draw and read a scatterplot
Put \(x\) on the horizontal axis and \(y\) on the vertical axis.
Draw observation \(i\) at \((x_i,y_i)\).
Find the direction from left to right: up, down, or none.
Find if the points follow approximately a straight line.
Compare the vertical spread at the left and at the right.
Self-check: Find row 1, \((4512,1530)\), on the plot.
Example · The brainhead dataAdded
Figure 2-2 (Slides p.12, redrawn from the data) · All 236 observations. Orange points 1–6 are the rows of Slides p.10. For example, point 1 is \((4512,1530)\), and point 2 is \((3738,1297)\).
Step 3: the points go up from bottom left to top right. Larger heads usually have heavier brains.
Step 4: the points follow approximately a straight line. There is no clear curve.
Step 5: at one head size, brain weights spread above and below. Point 5 has a smaller head than point 1, but a heavier brain.
Points 1 and 5: \(4512-4177=335\) cm³ for \(x\), and \(1590-1530=60\) g for \(y\).
Why · The next step uses a straight-line modelAdded
The points follow a line, with random spread above and below it. The model writes the line as \(\beta_0+\beta_1x_i\) and the spread as the random error \(\epsilon_i\).
02.5The simple linear regression model Slides p.13
WhatSlides p.13
Simple Linear Regression · Lecture 2 · p.13
\[y_i=\beta_0+\beta_1x_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\]
Or:
\[y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\]
• \(\beta_0,\beta_1,\sigma^2\): fixed, unknown parameters
• \(\epsilon_i\) is unobserved random error term
• \(y_i,x_i\) are the observed data
• Treat \(x_i\) as fixed
Simple linear regression (SLR) describes the mean of \(y\) with a straight line in one covariate \(x\).
A parameter is a fixed, unknown number of the population.
The error term \(\epsilon_i\) is the random distance of \(y_i\) from the line \(\beta_0+\beta_1x_i\). We cannot observe it.
\(\sigma^2\) is the variance of \(\epsilon_i\).
The normal distribution \(N(\mu,\sigma^2)\) is a continuous, bell-shaped distribution with mean \(\mu\) and variance \(\sigma^2\).
\(\overset{iid}{\sim}\): independent and identically distributed. All \(\epsilon_i\) are independent and have the same distribution \(N(0,\sigma^2)\).
\(\overset{indep}{\sim}\): independent only. The means \(\beta_0+\beta_1x_i\) change with \(i\).
"Treat \(x_i\) as fixed": \(x_i\) is a known constant. Only \(\epsilon_i\) and \(y_i\) are random.
Each \(y_i\) contains one \(\epsilon_i\). The \(\epsilon_i\) are independent.
02.6The meaning of \(\beta_0,\beta_1\) Slides p.14–16
Slides p.14–16 are three build steps of one slide (footer 12/27).
WhatSlides p.14–16
Regression Coefficients \((\beta_0,\beta_1)\) · Lecture 2 · p.14–16
\[y_i=\beta_0+\beta_1x_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\]
\[E[y_i|x_i]=\beta_0+\beta_1x_i\]
• \(\beta_0\) is an intercept
• \(E[y_i|x_i=0]=\beta_0+\beta_1(0)=\beta_0\)
• \(\beta_1\) is a slope
• \(E[y_i|x_i=x^*]=\beta_0+\beta_1(x^*)\)
• \(E[y_i|x_i=x^*+1]=\beta_0+\beta_1(x^*+1)\)
\[\Longrightarrow E[y_i|x_i=x^*+1]-E[y_i|x_i=x^*]=\beta_0+\beta_1(x^*+1)-[\beta_0+\beta_1(x^*)]=\beta_1\]
The conditional expectation \(E[y_i|x_i]\) is the mean of \(y_i\) at a given \(x_i\). Read "|" as "given".
The regression coefficients are \(\beta_0\) and \(\beta_1\).
The intercept \(\beta_0\) is the mean of \(y\) at \(x=0\): the height of the line at the vertical axis.
The slope \(\beta_1\) is the change in the mean of \(y\) when \(x\) increases by 1 unit.
\(x^*\) is any one fixed value of \(x\).
Figure 2-3 (Added) · Blue: the mean line. Green: height \(\beta_0\) at \(x=0\). Orange: \(x\) increases by 1. Purple: the mean increases by \(\beta_1\).
HowAdded
Say what \(\beta_0\) and \(\beta_1\) mean
Write the mean line \(E[y_i|x_i]=\beta_0+\beta_1x_i\).
Put \(x_i=0\). Write: "When \(x\) is 0, the expected \(y\) is \(\beta_0\)."
Put \(x^*\) and \(x^*+1\) into the line. Subtract the first result from the second.
Write: "When \(x\) increases by 1 unit, the expected \(y\) changes by \(\beta_1\)."
Replace "\(x\)" and "\(y\)" with the variable names and units.
For an increase of \(d\) units in \(x\), the change is \(d\beta_1\). Added
Self-check: Each sentence says "expected" or "mean": the coefficients describe \(E[y_i|x_i]\), not one \(y_i\). \(\beta_1\) has the unit "unit of \(y\) per unit of \(x\)".
Example · The brainhead dataAdded
We have no estimates yet. The sentences have no number for \(\beta_1\).
02.7A picture of the model Slides p.17 · figure Added
Slides p.17 has only the title "Simple Linear Regression".
Figure 2-4 (Added) · Orange bell curves (rotated by 90°) show the distribution of \(y\) at three values of \(x\). Purple points are possible observed values of \(y\).
Blue line: the mean line \(E[y_i|x_i]=\beta_0+\beta_1x_i\).
Orange bell curve: \(y_i\sim N(\beta_0+\beta_1x_i,\sigma^2)\) at a fixed \(x_i\). Its centre is on the line.
Vertical distance from a purple point to the orange centre: the error \(\epsilon_i=y_i-(\beta_0+\beta_1x_i)\).
All bell curves have the same width: the same variance \(\sigma^2\) at each \(x\).
Figure 2-2 has the same pattern: a line, with spread above and below it.
02.8The four assumptions Slides p.18
WhatSlides p.18
Simple Linear Regression · Lecture 2 · p.18
\[y_i=\beta_0+\beta_1x_i+\epsilon_i,\qquad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\]
Or:
\[y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\]
Assumptions:
1. Linearity
2. Independence
3. Normality
4. Equal variance (homoskedasticity)
How to estimate \(\beta_0\) and \(\beta_1\)?
An assumption is a condition that the model accepts as true. The data do not prove it.
1. Linearity
\(E[y_i|x_i]=\beta_0+\beta_1x_i\) is a straight line in \(x_i\).
2. Independence
The errors \(\epsilon_i\) are independent. Then the \(y_i\) are independent.
3. Normality
Each \(\epsilon_i\) has a normal distribution.
4. Equal variance (homoskedasticity)
Each \(\epsilon_i\) has the same variance \(\sigma^2\) at each \(x_i\). "Homoskedasticity" means "same variance".
HowAdded
Find each assumption in the model formula
\(\beta_0+\beta_1x_i\) is a straight line: assumption 1.
The first "i" in \(\overset{iid}{\sim}\) means "independent": assumption 2.
The name \(N\): assumption 3.
\(\sigma^2\) has no index \(i\): assumption 4.
Self-check: Each assumption matches one symbol in \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\) or in \(\beta_0+\beta_1x_i\).
Example · The brainhead dataAdded
1. Linearity: at head size \(x\) cm³, the mean brain weight is \(\beta_0+\beta_1x\) g.
2. Independence: the error of \(y_1=1530\) does not affect the error of \(y_2=1297\).
3. Normality: at one head size, brain weights are bell-shaped around their mean.
4. Equal variance: at 3738 cm³ and at 4512 cm³, brain weights have the same variance \(\sigma^2\).
Figure 2-2 agrees with assumptions 1 and 4.
Why · The four assumptions together give the modelAdded
\[\mathrm{Var}[\epsilon_i]=\sigma^2\ \text{ for all } i\]
Assumption 4.
3
\[\epsilon_i\sim N(0,\sigma^2)\]
Assumption 3.
4
\[\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\]
Assumption 2.
Lines 1 and 4 together are the first line of Slides p.18.
We know only the 236 pairs \((x_i,y_i)\), not \(\beta_0\) and \(\beta_1\). An estimate is a number from the data that replaces an unknown parameter.
02.9Practice Added
Q1. Use row 3 of the brainhead data on Slides p.10. Write the model for observation 3 in both forms. Which quantities are observed, and which are unknown?
Answer
Row 3: \((x_3,y_3)=(4261,1335)\).
First form: \(1335=\beta_0+\beta_1(4261)+\epsilon_3,\ \epsilon_3\sim N(0,\sigma^2)\).
Second form: \(y_3\sim N(\beta_0+4261\beta_1,\ \sigma^2)\).
Observed: \(x_3=4261\) and \(y_3=1335\). Unknown: \(\beta_0,\beta_1,\sigma^2\) and \(\epsilon_3\).
Q2. Use the SLR model for brain weight (g) on head size (cm³). Show that \(E[y_i|x_i=4261]-E[y_i|x_i=3738]\) depends only on \(\beta_1\). Then say what \(\beta_1\) means for these data.
\(\beta_0\) cancels. Meaning: when head size increases by 1 cm³, the expected brain weight changes by \(\beta_1\) g.
Q3. Name the model assumption that each statement describes.
(a) \(\mathrm{Var}[\epsilon_i]=\sigma^2\) for each \(i\).
(b) \(E[y_i|x_i]=\beta_0+\beta_1x_i\).
(c) The error of person 1 gives no information about the error of person 2.
(d) \(\epsilon_i\) has a bell-shaped \(N\) distribution.
Answer
(a) Equal variance (homoskedasticity): \(\sigma^2\) has no index \(i\).
(b) Linearity.
(c) Independence.
(d) Normality.
Unit 03Least squares estimation
Plan · Slides p.19–26
Slides p.19–20: many lines fit one scatterplot. Select the best line.
Slides p.21: make the sum of squares \(S(\beta_0,\beta_1)\) a minimum. Set both partial derivatives to 0.
Slides p.22–23: get \(\hat\beta_0=\bar y-\hat\beta_1\bar x\).
Slides p.24–25: get a formula for \(\hat\beta_1\).
Slides p.26: write \(\hat\beta_1\) as \(S_{xy}/S_{xx}\).
SymbolsAdded
\(\sum_{i=1}^n a_i\) is \(a_1+a_2+\dots+a_n\). A \(\sum\) without limits means \(\sum_{i=1}^n\).
The sample mean gives \(\sum x_i=n\bar x\) and \(\sum y_i=n\bar y\).
An estimator is a formula that gives a value for a parameter from the data. A hat marks it: \(\hat\beta_0\) estimates \(\beta_0\).
DataSlides p.10
The examples use the first 6 brainhead observations: \(n=6\).
\(i\)
1
2
3
4
5
6
\(x_i\) (cm³)
4512
3738
4261
3777
4177
3585
\(y_i\) (g)
1530
1297
1335
1282
1590
1300
\(\sum x_i=4512+3738+4261+3777+4177+3585=24050\) and \(\sum y_i=1530+1297+1335+1282+1590+1300=8334\).
03.1Section page and the line of best fit Slides p.19–20
The line of best fit is the line nearest to the points, by a stated rule.
Three candidate lines Added
Figure 3-1 (Added) · 236 brainhead points. Orange dashed: the mean brain weight, approximately 1284 g (Lecture 1 p.23). Purple dashed: \(y=385.4323+0.2608207x\). Green: \(y=335.4323+0.2608207x\).
All three lines are near the points. We need a rule that gives a number for each line.
03.2The least squares rule Slides p.21
WhatSlides p.21
Least Squares Estimation · Lecture 2 · p.21
Minimize the sum of squares:
\[S(\beta_0,\beta_1)=\sum_{i=1}^n (y_i-\beta_0-\beta_1x_i)^2\]
\[\Longrightarrow\quad \frac{\partial S(\beta_0,\beta_1)}{\partial\beta_0}=0,\qquad \frac{\partial S(\beta_0,\beta_1)}{\partial\beta_1}=0\]
\(\beta_0+\beta_1x_i\) is the height of the line at \(x_i\).
\(y_i-\beta_0-\beta_1x_i\) is the vertical distance from point \(i\) to the line. It is negative below the line.
Squares are never negative. Distances above and below the line do not cancel.
The sum of squares \(S(\beta_0,\beta_1)\) is a function of the line. A different line gives a different \(S\).
Least squares (LS) selects the \((\beta_0,\beta_1)\) with the smallest \(S\). These values, \(\hat\beta_0,\hat\beta_1\), are the least squares estimators.
The partial derivative \(\partial S/\partial\beta_0\) is the rate of change of \(S\) in \(\beta_0\), with \(\beta_1\) constant. \(\partial S/\partial\beta_1\) keeps \(\beta_0\) constant.
HowAdded
Calculate \(S\) for a given line
For each point, calculate the height \(\beta_0+\beta_1x_i\).
Calculate each distance \(y_i-\beta_0-\beta_1x_i\).
Square each distance. Add the \(n\) squares to get \(S\).
The line with the smallest \(S\) is the best by least squares.
Self-check: \(S\ge0\). \(S\) has the unit g².
Example · 6 observations from Slides p.10Added
Line A: \(\beta_0=1389,\ \beta_1=0\), a horizontal line at \(\bar y=8334/6=1389\).
Line B has the smallest \(S\) of the three. Section 03.5 finds the smallest \(S\) of all lines.
Figure 3-2 (Added) · 236 points, the green line \(y=335.4323+0.2608207x\), and each vertical distance (orange). \(S\) is the sum of the squared orange lengths.
Why · The partial derivatives are 0 at the minimumAdded
\(S(\beta_0,\beta_1)\) is a smooth polynomial of degree 2 in \(\beta_0\) and \(\beta_1\).
At the minimum, a small change in \(\beta_0\) alone cannot make \(S\) smaller: \(\partial S/\partial\beta_0=0\).
The same is true for \(\beta_1\): \(\partial S/\partial\beta_1=0\).
This gives two equations in two unknowns.
03.3The derivative for \(\beta_0\): solve for \(\hat\beta_0\) Slides p.22–23
WhatSlides p.22–23
Least Squares Estimation · Lecture 2 · p.23
\[\frac{\partial S(\beta_0,\beta_1)}{\partial\beta_0}=\sum_{i=1}^n 2(y_i-\beta_0-\beta_1x_i)(-1),\quad\text{set to 0 and solve:}\]
\[0=\sum_{i=1}^n(y_i-\hat\beta_0-\hat\beta_1x_i)\]
\[0=\sum_{i=1}^n y_i-n\hat\beta_0-\hat\beta_1\sum_{i=1}^n x_i\]
\[\hat\beta_0=\left(\frac1n\sum_{i=1}^n y_i\right)-\hat\beta_1\left(\frac1n\sum_{i=1}^n x_i\right)=\bar y-\hat\beta_1\bar x\]
Why the hat appears on line 2: before "set to 0", \(\beta_0,\beta_1\) can have any value. "Set to 0" keeps only the values that make \(S\) a minimum. These values are the estimates, so they get a hat.
When you know \(\hat\beta_1\), the two sample means give \(\hat\beta_0\). The same equation, \(\bar y=\hat\beta_0+\hat\beta_1\bar x\), puts \((\bar x,\bar y)\) on the least squares line.
Slides p.22 shows only the first line. Slides p.23 shows the full slide.
\(S(290.787,\,0.2739824)=42105.59\). It is smaller than 91032 (Line A), 42389.2968 (Line B), and 43726.4608 (Line C).
All 236 observations: \(\hat\beta_0=335.4323150\) and \(\hat\beta_1=0.2608207\). Slides p.38
Figure 3-3 (Added) · 236 points and the least squares line \(\hat y=335.432+0.2608x\) (green). The orange point \((\bar x,\bar y)\), approximately \((3638,\ 1284)\), is on the line.
Meaning (Unit 02): 1 cm³ more head size gives approximately 0.261 g more expected brain weight.
Q2. For the same data, calculate \(\hat\beta_1\) with \(\hat\beta_1=\dfrac{\sum y_ix_i-n\bar y\bar x}{\sum x_i^2-n\bar x^2}\). Check that it agrees with Q1.
Answer
Step 1: \(\sum y_ix_i=2+8+15+16+25=66\).
Step 2: \(\sum x_i^2=1+4+9+16+25=55\).
Step 3: numerator \(66-5\times4\times3=66-60=6\).
Step 4: denominator \(55-5\times3^2=55-45=10\).
Step 5: \(\hat\beta_1=6/10=0.6\), as in Q1. The numerator is \(S_{xy}\), and the denominator is \(S_{xx}\).
Q3. Show that \(\sum_{i=1}^n (x_i-\bar x)=0\). Use it to show that \(\sum_{i=1}^n x_i(x_i-\bar x)=\sum_{i=1}^n (x_i-\bar x)^2\).
This identity makes the two denominators on Slides p.26 equal.
Unit 04Maximum likelihood estimation and its equality to least squares
Plan · Slides p.27–29
Slides p.27: write the likelihood \(L\). Take the log to get \(\ell\).
Slides p.28: differentiate \(\ell\) for \(\beta_0\), \(\beta_1\), \(\sigma^2\): (⋆), (⋆⋆), and a third equation.
Slides p.29: (⋆) = 0 and (⋆⋆) = 0 is the same as "minimize the sum of squares": \(\hat\beta^{LS}=\hat\beta^{ML}\).
Start point (results of Units 02–03) · Added
Model: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\). Same as \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\).
Least squares (LS) minimizes \(S(\beta_0,\beta_1)=\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)^2\). Result: \(\hat\beta_1^{LS}=S_{xy}/S_{xx}\) and \(\hat\beta_0^{LS}=\bar y-\hat\beta_1^{LS}\bar x\) (Slides p.26).
Brainhead: \((x_1,y_1)=(4512,1530)\) and \((x_2,y_2)=(3738,1297)\) (Slides p.10). Slides p.38: \(\hat\beta_0=335.4323150\), \(\hat\beta_1=0.2608207\), fitted values 1512.255 and 1310.380 for observations 1 and 2.
Small data set (Added): \((x_i,y_i)\) = (1, 2), (2, 3), (3, 5), (4, 6), \(n=4\).
04.1Likelihood and log-likelihood Slides p.27
WhatSlides p.27
Maximum Likelihood Estimation · Lecture 2 · p.27
Recall: \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\)
We can write the likelihood:
\[L(\beta_0,\beta_1,\sigma^2|y)=\prod_{i=1}^n\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(y_i-[\beta_0+\beta_1x_i])^2}{2\sigma^2}\right)=(2\pi\sigma^2)^{-n/2}\exp\left(-\sum_{i=1}^n\frac{(y_i-[\beta_0+\beta_1x_i])^2}{2\sigma^2}\right)\]
It's more convenient to work with the log-likelihood:
\[\ell(\beta_0,\beta_1,\sigma^2|y)=-\frac n2\log(2\pi)-\frac n2\log(\sigma^2)-\frac{1}{2\sigma^2}\sum_{i=1}^n(y_i-[\beta_0+\beta_1x_i])^2\]
Normal density: the PDF of \(N(\mu,\sigma^2)\), \(f(y)=\dfrac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\dfrac{(y-\mu)^2}{2\sigma^2}\right)\). It is largest at \(y=\mu\).
\(\exp(a)=e^a\), with \(e\approx2.718\). \(\log\) is the natural log (base \(e\)).
\(\prod_{i=1}^n a_i=a_1\times a_2\times\cdots\times a_n\) is a product.
The likelihood \(L(\beta_0,\beta_1,\sigma^2|y)\) is the joint density of the data, as a function of the parameters. "\(|y\)": the data \(y=(y_1,\dots,y_n)\) are fixed.
The log-likelihood is \(\ell=\log L\).
Maximum likelihood estimation (MLE) uses the parameter values that make \(L\) largest: \(\hat\beta_0^{ML},\hat\beta_1^{ML}\).
The \(y_i\) are independent. Their joint density is the product of the single densities. The log is increasing: \(L\) and \(\ell\) have their maximum at the same point. The log changes the product into a sum, which is easier to differentiate.
HowAdded
Write \(\ell\) and calculate its value
Write one density. Replace \(\mu\) with \(\beta_0+\beta_1x_i\) in the normal density.
Multiply to get \(L\). The \(y_i\) are independent.
Take the log to get \(\ell\). It has three terms: \(-\frac n2\log(2\pi)\), \(-\frac n2\log(\sigma^2)\), and \(-\frac{1}{2\sigma^2}\sum(y_i-[\beta_0+\beta_1x_i])^2\).
Find each difference \(y_i-[\beta_0+\beta_1x_i]\).
Square the differences and add them.
Calculate the three terms with \(n\), \(\sigma^2\), and the sum of squares.
Add the three terms.
Self-check: ① The first term uses only \(n\). ② The second uses only \(n\) and \(\sigma^2\). ③ The third term is always \(\le0\). ④ One observation contributes the log of its density.
Example 1 · Density of one brainhead observationAdded
Observation 1, with \(\beta_0=335.4323150\), \(\beta_1=0.2608207\) (Slides p.38) and \(\sigma^2=5000\).
The value 5000 is only for this example. Unit 06 estimates \(\sigma^2\).
(⋆) and (⋆⋆) both start with the positive constant \(1/\sigma^2\).
04.3MLE and LS give the same estimates (\(\hat\beta^{LS}=\hat\beta^{ML}\)) Slides p.29
WhatSlides p.29
Maximum Likelihood Estimation · Lecture 2 · p.29
Setting \((\star)\) and \((\star\star)\) to 0 and solving is equivalent to minimizing the sum of squares!
• That is \(\hat\beta_0^{LS}=\hat\beta_0^{ML}\) and \(\hat\beta_1^{LS}=\hat\beta_1^{ML}\)
• Here on out we will call the LS/MLE estimators \(\hat\beta\).
Equivalent: the two problems have the same solution.
\(\hat\beta\): from here, \(\hat\beta_0\) and \(\hat\beta_1\) are both the LS estimators and the MLE.
In \(\ell\), \(\beta_0\) and \(\beta_1\) occur only in \(-\frac{1}{2\sigma^2}S(\beta_0,\beta_1)\). This factor is negative. A smaller \(S\) gives a larger \(\ell\).
This answers Slides p.26: in the normal model, least squares gives the MLE.
HowAdded
Find the MLE of \(\beta_0\) and \(\beta_1\)
Check the model: \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\).
Calculate \(\bar x\), \(\bar y\), \(S_{xx}\), and \(S_{xy}\).
These are the MLE: \(\hat\beta_0^{ML}=\hat\beta_0^{LS}\) and \(\hat\beta_1^{ML}=\hat\beta_1^{LS}\).
Self-check: ① \(\sum(y_i-[\hat\beta_0+\hat\beta_1x_i])=0\): (⋆) = 0. ② \(\sum(y_i-[\hat\beta_0+\hat\beta_1x_i])x_i=0\): (⋆⋆) = 0. ③ \(\ell\) at \(\hat\beta\) is larger than at another line, for each \(\sigma^2\).
Example 3 · MLE for the small data setAdded
1
Check the model How step 1
Use \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\).
\(-3.775754>-4.375754\): the LS line has the larger \(\ell\). ▲
Figure 4-1 (Added) · Small data set, \(\beta_0\) fixed at \(\hat\beta_0=0.5\). Top (blue): \(S\), minimum \(S=0.2\) at \(\hat\beta_1=1.4\). Bottom (green): \(\ell\) with \(\sigma^2=1\), maximum \(\ell=-3.775754\) at the same \(\hat\beta_1=1.4\). Purple: line A (\(\beta_1=1.2\)), \(S=1.4\), \(\ell=-4.375754\).
The bottom curve is the top curve times \(-\frac12\), plus a constant.
Why · (⋆) = 0 and (⋆⋆) = 0 are the same as "minimize \(S\)"Slides p.29
\(-\frac{1}{2\sigma^2}\neq0\). Same solution: \(\hat\beta_0^{ML}=\bar y-\hat\beta_1\bar x=\hat\beta_0^{LS}\), \(\hat\beta_1^{ML}=S_{xy}/S_{xx}=\hat\beta_1^{LS}\). ∎
04.4Practice Added
Q1. Brainhead observation 2 has \(x_2=3738\) and \(y_2=1297\) (Slides p.10). Its fitted value is 1310.380 (Slides p.38). Use \(\sigma^2=5000\). Calculate (a) the normal density of \(y_2\), and (b) its contribution to \(\ell\).
Q3. True or false: under \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\), the maximum likelihood estimates of \(\beta_0\) and \(\beta_1\) depend on \(\sigma^2\). Use the log-likelihood to justify your answer.
Answer
False.
\(\ell=-\frac n2\log(2\pi)-\frac n2\log(\sigma^2)-\frac{1}{2\sigma^2}S(\beta_0,\beta_1)\). For each fixed \(\sigma^2>0\), the first two terms do not contain \(\beta_0,\beta_1\), and \(-\frac{1}{2\sigma^2}\) is negative. The \(\beta_0,\beta_1\) that make \(\ell\) largest always make \(S\) smallest.
Small data set with \(\sigma^2=2\): \(-\frac42\log(2)=-2\times0.693147=-1.386294\).
At \((0.5,1.4)\): \(\ell=-3.675754-1.386294-\frac{0.2}{4}=-5.112048\).
At \((0.5,1.2)\): \(\ell=-3.675754-1.386294-\frac{1.4}{4}=-5.412048\).
The least squares line \((0.5,1.4)\) again has the larger \(\ell\).
Points 2 and 3 are above the fitted line. Points 1, 4, and 5 are below it.
Example 2 · Observations 4 and 5 of the brainhead dataAdded
1
\(e_4=y_4-\hat y_4=1282-1320.552\)
\(\hat y_4\) from Section 05.2, Example 2
2
\(e_4=-38.552\)
3
\(e_5=y_5-\hat y_5=1590-1424.880\)
\(y_5\): Slides p.10; \(\hat y_5\): Slides p.38
4
\(e_5=165.120\)
Person 4: 38.552 g below the fitted line.
Person 5: 165.120 g above the fitted line.
Figure 5-1 (Added) · Purple: \((x_5,y_5)=(4177,1590)\). Blue line: the fitted line. Open blue point: \(\hat y_5=1424.880\) at \(x_5=4177\). Orange: \(e_5=165.120\), positive.
The difference comes only from \(\hat\beta_0-\beta_0\) and \(\hat\beta_1-\beta_1\). Both are unknown.
05.5A picture of the fitted values and residuals Slides p.32
Slides p.32 has only the title "Fitted Values and Residuals". Figure 5-2 is Added.
Gray points: observations 7 to 236.
Purple points 1–6: the six observations of Slides p.10.
Blue line: \(\hat\beta_0+\hat\beta_1x=335.4323150+0.2608207x\) (Slides p.38).
Open blue points: the fitted values \((x_i,\hat y_i)\).
Orange segments: the residuals \(e_i\), from \(\hat y_i\) to \(y_i\).
Figure 5-2 (Added) · Points 1, 5, and 6 are above the line: \(e_1,e_5,e_6\) are positive. Points 2, 3, and 4 are below: \(e_2,e_3,e_4\) are negative.
The fitted line makes \(\sum_{i=1}^{236}e_i^2\) as small as possible.
05.6Practice Added
Q1. Use \(\hat\beta_0=335.4323150\) and \(\hat\beta_1=0.2608207\) (Slides p.38). Observation 6 has \(x_6=3585\) and \(y_6=1300\) (Slides p.10). Calculate \(\hat y_6\) and \(e_6\). Is point 6 above or below the fitted line?
Answer
1
\(\hat y_6=335.4323150+0.2608207\times3585\)
2
\(\hat y_6=335.4323150+935.0422\)
3
\(\hat y_6=1270.4745\approx1270.475\)
sixth value on Slides p.38
4
\(e_6=1300-1270.475=29.525\)
\(e_6>0\): point 6 is above the fitted line.
Q2. The data are \((x_i,y_i)\): (1, 1), (2, 3), (3, 2). Calculate \(\hat\beta_0\), \(\hat\beta_1\), the three fitted values, and the three residuals.
Answer
Step 1: \(\bar x=6/3=2\) and \(\bar y=6/3=2\).
Step 2: \(x_i-\bar x=(-1,0,1)\) and \(y_i-\bar y=(-1,1,0)\).
Step 3: \(S_{xy}=(-1)(-1)+(0)(1)+(1)(0)=1\) and \(S_{xx}=1+0+1=2\).
Step 4: \(\hat\beta_1=1/2=0.5\) and \(\hat\beta_0=2-0.5\times2=1\).
Point 2 is above the fitted line. Points 1 and 3 are below it.
Q3. Use the second line of Slides p.23, \(0=\sum_{i=1}^n(y_i-\hat\beta_0-\hat\beta_1x_i)\). Show that the residuals add to 0. Check the result with the small data set.
Unit 06Estimating \(\sigma^2\): the MLE and the unbiased estimator
Plan · Slides p.33–38
Slides p.33: section page.
Slides p.34: solve \(\partial\ell/\partial\sigma^2=0\) for \(\hat\sigma^2_{ML}\).
Slides p.35: the course estimator \(\hat\sigma^2=\sum e_i^2/(n-2)\).
Slides p.36: \(\hat\sigma^2\) is unbiased (chi-squared result).
Slides p.37: calculate \(\hat\beta_0\), \(\hat\beta_1\), and \(\hat\sigma\) in order.
Slides p.38: brainhead estimates and fitted values.
Start point (results of Units 02–05) · Added
Model: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\). Same as \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\).
The error variance \(\sigma^2=\mathrm{Var}[\epsilon_i]\) is the spread of the points around \(\beta_0+\beta_1x_i\). \(\sigma\) is the standard deviation of the errors, in the unit of \(y\).
\(\hat\beta_1=S_{xy}/S_{xx}\) and \(\hat\beta_0=\bar y-\hat\beta_1\bar x\). Least squares and maximum likelihood give the same values (Slides p.26–29).
\(\hat y_i=\hat\beta_0+\hat\beta_1x_i\) and \(e_i=y_i-\hat y_i\) (Slides p.31).
\(\sum e_i^2\) is the sum of squared residuals. \(\sum\) without limits means \(\sum_{i=1}^n\).
Residuals at their \(x\) positions Added. An estimate of \(\sigma\) measures their usual distance from \(e=0\).
Figure 6-1 (Added) · Orange: the five residuals \(e_i\). Blue: \(e=0\). Green band: \(\pm\hat\sigma=\pm0.894\) (Section 06.3). Four of the five residuals are in the band.
substitute \(e_i=y_i-(\hat\beta_0+\hat\beta_1x_i)\); Slides p.34, last line ∎
06.3The estimator \(\hat\sigma^2\) that the course uses Slides p.35
WhatSlides p.35
Maximum Likelihood Estimation Continued · Lecture 2 · p.35
However we typically adopt a different estimator:
\[\hat\sigma^2=\frac{\sum_{i=1}^n e_i^2}{n-2}\qquad\text{Why?}\]
Only the denominator changes: \(n-2\), not \(n\).
From here, \(\hat\sigma^2\) (no subscript) always means this estimator. \(\hat\sigma=\sqrt{\hat\sigma^2}\) estimates \(\sigma\). Slides p.36 answers "Why?".
HowAdded
Calculate \(\hat\sigma^2\) and \(\hat\sigma\) from data
Do steps 1–4 of Section 06.2 to get \(\sum e_i^2\).
Calculate \(n-2\).
Divide: \(\hat\sigma^2=\sum e_i^2/(n-2)\).
Take the square root: \(\hat\sigma=\sqrt{\hat\sigma^2}\).
Self-check: \(\hat\sigma^2\) is larger than \(\hat\sigma^2_{ML}\). The ratio \(\hat\sigma^2/\hat\sigma^2_{ML}\) is exactly \(n/(n-2)\).
Example · The same small data setAdded
Section 06.2: \(\sum e_i^2=2.4\), \(n=5\).
Step 2: \(n-2=5-2=3\).
Step 3: \(\hat\sigma^2=2.4/3=0.8\).
Step 4: \(\hat\sigma=\sqrt{0.8}=0.894\).
Check: \(\hat\sigma^2/\hat\sigma^2_{ML}=0.8/0.48=1.667\) and \(n/(n-2)=5/3=1.667\).
06.4The unbiased estimator Slides p.36
The slide gives only the intuition. It uses a chi-squared result without proof.
WhatSlides p.36
Unbiased Estimator · Lecture 2 · p.36
Why do we use this \(\hat\sigma^2\) instead of \(\hat\sigma^2_{ML}\)?
\[E\left[\frac{\sum_{i=1}^n e_i^2}{n-2}\right]=\sigma^2\]
Intuition:
\[\frac{1}{\sigma^2}\sum_{i=1}^n e_i^2\sim\chi^2_{(n-2)}\quad\text{and}\quad E[\chi^2_\nu]=\nu\]
\[\Longrightarrow E\left[\frac{\sum_{i=1}^n e_i^2}{n-2}\right]=\sigma^2E\left[\frac{1}{\sigma^2}\frac{\sum_{i=1}^n e_i^2}{n-2}\right]=\frac{\sigma^2}{(n-2)}E\left[\frac{1}{\sigma^2}\sum_{i=1}^n e_i^2\right]=\sigma^2\]
Often doesn't matter when \(n\ge50\)
An estimator is unbiased when its expectation equals the parameter: \(E[\hat\sigma^2]=\sigma^2\).
In words: over many new data sets, the mean of the \(\hat\sigma^2\) values is \(\sigma^2\).
The chi-squared distribution \(\chi^2_\nu\) has only positive values. Lecture 1 p.26: if \(Z_1,\dots,Z_\nu\overset{iid}{\sim}N(0,1)\), then \(\sum Z_i^2\sim\chi^2_\nu\), and \(E[\chi^2_\nu]=\nu\).
The degrees of freedom \(\nu\) is the parameter of \(\chi^2_\nu\). Here \(\nu=n-2\).
\(\sim\): "has the distribution".
\(\frac{1}{\sigma^2}\sum e_i^2\) has mean \(n-2\). The division by \(n-2\) cancels it. For \(n\ge50\), \(n\) and \(n-2\) are almost equal, and the choice seldom changes a conclusion.
The \(\chi^2_3\) density and its mean Added:
Figure 6-2 (Added) · Blue: the \(\chi^2_3\) density, above 0 only for positive values. Purple dashed: the mean \(E[\chi^2_3]=3\).
The right tail is long, but the mean is exactly the degrees of freedom.
HowAdded
Find if an estimator of \(\sigma^2\) is unbiased, and if the denominator is important
Write the estimator as \(c\cdot\sum e_i^2\). Find \(c\).
Use \(E\left[\frac{1}{\sigma^2}\sum e_i^2\right]=n-2\). This gives \(E\left[\sum e_i^2\right]=(n-2)\sigma^2\).
Calculate \(E[c\sum e_i^2]=c(n-2)\sigma^2\).
If \(c(n-2)=1\), the estimator is unbiased.
If \(n\ge50\), \(n/(n-2)\) is near 1. The difference is usually small.
Self-check: Only \(c=1/(n-2)\) gives \(c(n-2)=1\).
Example · Two sample sizesAdded
The small data set (\(n=5\)) and \(n=50\).
Step 1: for \(\hat\sigma^2\), \(c=1/(n-2)\). For \(\hat\sigma^2_{ML}\), \(c=1/n\).
Steps 2–3 with \(n=5\): \(E[\hat\sigma^2]=\frac13\cdot3\sigma^2=\sigma^2\). \(E[\hat\sigma^2_{ML}]=\frac15\cdot3\sigma^2=0.6\sigma^2\).
Step 4: \(\hat\sigma^2\) is unbiased. With \(n=5\), \(\hat\sigma^2_{ML}\) is 40% too low on average.
Step 5 with \(n=50\): \(E[\hat\sigma^2_{ML}]=\frac{1}{50}\cdot48\sigma^2=0.96\sigma^2\). The ratio \(n/(n-2)=50/48=1.042\): only 4%.
WhySlides p.36
The only input without proof is \(\frac{1}{\sigma^2}\sum e_i^2\sim\chi^2_{(n-2)}\).
linearity of \(E\); \(\frac{1}{n-2}\) is a constant
5
\[=\frac{\sigma^2}{(n-2)}\cdot(n-2)\]
\(\frac{1}{\sigma^2}\sum e_i^2\sim\chi^2_{(n-2)}\) and \(E[\chi^2_\nu]=\nu\)
6
\[=\sigma^2\]
∎
06.5Data example: all three estimates in order Slides p.37
Slides p.37 shows this order as code. Only the statistical steps are here.
WhatSlides p.37
Data Example · Lecture 2 · p.37 (statistical content of the slide)
Prepare \(\bar x\), \(\bar y\), \(S_{xy}=\sum(y_i-\bar y)(x_i-\bar x)\), \(S_{xx}=\sum(x_i-\bar x)^2\), and \(n\) (number of observations).
\(\beta_1\) estimate (MLE): \(\hat\beta_1=S_{xy}/S_{xx}\).
\(\beta_0\) estimate (MLE): \(\hat\beta_0=\bar y-\hat\beta_1\bar x\).
Fitted values: \(\hat y_i=\hat\beta_0+\hat\beta_1x_i\). Residuals: \(e_i=y_i-\hat y_i\).
\(\sigma\) (unbiased estimate): \(\hat\sigma=\sqrt{\sum e_i^2/(n-2)}\).
A slide note gives a second way to get \(\hat\beta_1\). Divide the sample covariance of \(y\) and \(x\) by the sample variance of \(x\). The \(n-1\) cancels, giving \(S_{xy}/S_{xx}\).
\(\hat\sigma\) is the square root of \(\hat\sigma^2\), not of \(\hat\sigma^2_{ML}\).
HowAdded
Get the three point estimates \(\hat\beta_0\), \(\hat\beta_1\), \(\hat\sigma\)
Observation 5, Step 3: \(\hat y_5=335.4323150+1089.4481=1424.8804\). Rounded: 1424.880, the fifth value on Slides p.38.
Observation 5, Step 5: \(e_5=1590-1424.880=165.120\) g, above the fitted line.
For \(\hat\sigma\) of these data, use all 236 residuals (Section 06.5, step 6).
Summary of this unit
\(\hat\sigma^2_{ML}=\sum e_i^2/n\). The course uses the unbiased \(\hat\sigma^2=\sum e_i^2/(n-2)\). Lecture 3 uses \(\hat\sigma\) in confidence intervals and tests.
06.7Practice Added
Q1. Data: \(x=(0,1,2,3)\) and \(y=(1,3,2,6)\). Calculate \(\hat\beta_0\), \(\hat\beta_1\), the residuals, \(\hat\sigma^2_{ML}\), \(\hat\sigma^2\), and \(\hat\sigma\).
\(n/(n-2)=4/2=2\): the unbiased estimate is two times the MLE.
Q2. Use \(\frac{1}{\sigma^2}\sum_{i=1}^n e_i^2\sim\chi^2_{(n-2)}\) to find \(E[\hat\sigma^2_{ML}]\). Is \(\hat\sigma^2_{ML}\) unbiased? Give the factor for \(n=5\) and for \(n=50\).
The factor \(\frac{n-2}{n}\) is less than 1. \(E[\hat\sigma^2_{ML}]\ne\sigma^2\): \(\hat\sigma^2_{ML}\) is not unbiased. On average it is too low. For \(n=5\), the factor is \(3/5=0.6\). For \(n=50\), it is \(48/50=0.96\).
Q3. Use the six fitted values on Slides p.38 and the six values \(y_i\) on Slides p.10. Calculate \(e_1,\dots,e_6\) and their sum. Can you use these six residuals to calculate \(\hat\sigma\) for the brainhead data?