Lecture 3: Simple Linear Regression: Inference

Unit 01Recap: the SLR model, least squares, and the estimate of \(\sigma^2\)

Plan · Slides p.1–8

Step 1: Title pages (Slides p.1–2).

Step 2: Model and assumptions (Slides p.3–4).

Step 3: Meaning of \(\beta_1\) (Slides p.5).

Step 4: \(\hat\beta_0\) and \(\hat\beta_1\) (Slides p.6–7).

Step 5: Residuals and \(\hat\sigma^2\) (Slides p.8).

Data for this unit · Added

Small data set: \(x=(1,2,3,4,5)\), \(y=(2,4,5,4,5)\), \(n=5\).

Brainhead data: \(n=236\) persons, \(y\) = brain weight (g), \(x\) = head size (cm³).

01.1Title page and section page Slides p.1–2

Slides p.1 (title): "Lecture 3: Simple Linear Regression: Inference".

Inference uses sample data to make statements about unknown population parameters, for example with confidence intervals and hypothesis tests.

Slides p.2: section "Recap" (Slides p.3–8).

01.2The SLR model and its assumptions Slides p.3–4

All inference in this lecture needs the four assumptions.

What Slides p.3–4

Simple Linear Regression · Lecture 3 · p.3–4 \[y_i=\beta_0+\beta_1x_i+\epsilon_i,\quad \epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\] Or: \[y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\] Assumptions:
1. Linearity
2. Independence
3. Normality
4. Equal variance (homoskedasticity)

Simple linear regression (SLR): the mean of a response \(y\) is a straight-line function of one explanatory variable \(x\).

The response \(y_i\) is the result value of observation \(i\), \(i=1,\dots,n\). The sample size \(n\) is the number of observations.

The explanatory variable \(x_i\) is the input value of observation \(i\). The model treats it as a known constant.

The intercept \(\beta_0\) and the slope \(\beta_1\) are parameters (unknown model constants) of the population line.

The error \(\epsilon_i\) is the random, unobservable difference between \(y_i\) and the population line \(\beta_0+\beta_1x_i\).

\(N(\mu,\sigma^2)\) is the bell-shaped Normal distribution with mean (center) \(\mu\) and variance (mean squared distance from the mean) \(\sigma^2\).

\(\sigma^2\) is the error variance, the third parameter. \(\sigma\) is the standard deviation.

"iid": independent and identically distributed. All \(\epsilon_i\) have one distribution and do not affect each other.

"indep": independent. The \(y_i\) are not identically distributed: their mean \(\beta_0+\beta_1x_i\) changes with \(x_i\).

1. Linearity: the mean of \(y\) is \(\beta_0+\beta_1x\).

2. Independence: the errors are independent.

3. Normality: each error is Normal.

4. Equal variance (homoskedasticity): the error variance is \(\sigma^2\) at each \(x\).

Both forms give the same model: "line plus error", and the distribution of \(y_i\).

How Added

Write a data set as an SLR model

Find the response \(y\): the variable to explain or predict.
Find the explanatory variable \(x\): the variable that explains \(y\).
Write \(y_i=\beta_0+\beta_1x_i+\epsilon_i\) with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\), \(i=1,\dots,n\).
List the three parameters: \(\beta_0\), \(\beta_1\), \(\sigma^2\).
For the second form, put the mean \(\beta_0+\beta_1x_i\) and the variance \(\sigma^2\) into \(N(\cdot,\cdot)\).

Self-check: Greek letters: only \(\beta_0\), \(\beta_1\), \(\sigma^2\), \(\epsilon_i\). \(n\) equals the number of data pairs.

Example 1 · The small data set Added

Step 1–2: in each pair \((1,2),(2,4),(3,5),(4,4),(5,5)\), \(x\) is first. \(n=5\).

Step 3: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\), \(i=1,\dots,5\).

Observation 3: \(5=\beta_0+\beta_1\cdot3+\epsilon_3\), with \(\beta_0\), \(\beta_1\), and \(\epsilon_3\) unknown.

Step 5: \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\). For observation 3, \(y_3\sim N(\beta_0+3\beta_1,\sigma^2)\).

Example 2 · The brainhead data Added

Step 1–2: for person \(i\), \(y_i\) is brain weight (g) and \(x_i\) is head size (cm³).

Step 3: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\), \(i=1,\dots,236\).

Self-check: Slides p.45 gives \(n-2=234\) degrees of freedom. \(n=234+2=236\).

Why · The two forms are the same Added

\(x_i\), \(\beta_0\), and \(\beta_1\) are constants.

\(E[y_i]=E[\beta_0+\beta_1x_i+\epsilon_i]\)

\(=\beta_0+\beta_1x_i+E[\epsilon_i]\)

linearity of expectation

\(=\beta_0+\beta_1x_i+0\)

\(\epsilon_i\sim N(0,\sigma^2)\)

\(\mathrm{Var}[y_i]=\mathrm{Var}[\beta_0+\beta_1x_i+\epsilon_i]\)

\(=\mathrm{Var}[\epsilon_i]\)

constants do not change variance

\(=\sigma^2\)

A Normal variable plus a constant is Normal: \(y_i\sim N(\beta_0+\beta_1x_i,\sigma^2)\). Independent \(\epsilon_i\) give independent \(y_i\). ∎

01.3The interpretation of \(\beta_1\) Slides p.5

What Slides p.5

Interpretations · Lecture 3 · p.5 \[\beta_1=E[y|x=x^*+1]-E[y|x=x^*]\] \(\beta_1\) is the mean difference comparing a population with \(x\) to a population with \(x\) a unit lower
\(\hat\beta_1\) is an estimate of this mean difference

The conditional expectation \(E[y|x=x^*]\) is the mean of \(y\) over the units with \(x=x^*\). \(x^*\) is any fixed value of \(x\).

A population is the set of all possible units. "A population with \(x\)" is all units with that value of \(x\).

An estimate is a value calculated from sample data. The hat "^" marks it: \(\hat\beta_1\) is the estimate of \(\beta_1\).

How Added

Interpret \(\hat\beta_1\) in one sentence

Find the unit of \(x\) and the unit of \(y\).
Write: "Compare two populations whose \(x\) differs by 1 (unit of \(x\))."
Continue: "The estimated mean of \(y\) differs by \(\hat\beta_1\) (unit of \(y\))."
Use the sign of \(\hat\beta_1\) to say "higher" or "lower" for the larger \(x\).
For a difference of \(k\) units in \(x\), use the mean difference \(k\hat\beta_1\).

Self-check: The sentence says "mean" and gives both units.

Example 1 · The small data set Added

\(\hat\beta_1=0.6\) (Section 01.4). No units.

Step 2–4: compare two populations whose \(x\) differs by 1. The estimated mean of \(y\) is 0.6 higher at the larger \(x\).

Step 5: for \(x=5\) and \(x=1\), \(k=5-1=4\). The mean difference is \(4\times0.6=2.4\).

Example 2 · The brainhead data Added

Step 1: \(x\) is in cm³. \(y\) is in g.

Step 2–4: compare two populations whose head sizes differ by 1 cm³. The estimated mean brain weight is 0.26082 g higher at the larger head size (Slides p.45).

Step 5: for a difference of 100 cm³, the mean difference is \(100\times0.26082=26.082\) g.

This is a difference of means, not a change for one person.

Why Added

Put two values of \(x\) into \(E[y|x]=\beta_0+\beta_1x\) (Section 01.2) and subtract.

\(E[y|x=x^*+1]-E[y|x=x^*]=\big(\beta_0+\beta_1(x^*+1)\big)-\big(\beta_0+\beta_1x^*\big)\)

put in \(E[y|x]=\beta_0+\beta_1x\)

\(=\beta_0+\beta_1x^*+\beta_1-\beta_0-\beta_1x^*\)

\(=\beta_1\)

∎ The result does not contain \(x^*\): it is the same at each \(x\).

01.4Least squares estimation Slides p.6–7

What Slides p.6–7

Least Squares Estimation · Lecture 3 · p.6 Minimize the sum of squares: \[S(\beta_0,\beta_1)=\sum_{i=1}^n(y_i-\beta_0-\beta_1x_i)^2\] \[\Longrightarrow\ \frac{\partial S(\beta_0,\beta_1)}{\partial\beta_0}=0,\qquad \frac{\partial S(\beta_0,\beta_1)}{\partial\beta_1}=0\]

Estimates · Lecture 3 · p.7 So the least squares estimators are: \[\hat\beta_1=\frac{S_{xy}}{S_{xx}},\qquad \hat\beta_0=\bar y-\hat\beta_1\bar x.\] OLS=MLE under the assumption of Normality
OLS well defined even without Normality, but only equal to MLE under Normality

The sum of squares \(S(\beta_0,\beta_1)\) adds the squared vertical differences \(y_i-\beta_0-\beta_1x_i\) of a candidate line.

The partial derivative \(\partial S/\partial\beta_0\) is the derivative of \(S\) in \(\beta_0\), with \(\beta_1\) constant.

Least squares (OLS, ordinary least squares) uses the \(\beta_0\) and \(\beta_1\) that make \(S\) smallest.

An estimator is a formula that changes data into an estimate, for example \(\hat\beta_1=S_{xy}/S_{xx}\).

The sample means are \(\bar x=\frac1n\sum_{i=1}^n x_i\) and \(\bar y=\frac1n\sum_{i=1}^n y_i\).

\(S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2\) is the sum of squared deviations of \(x\). \(S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)\) is the sum of cross products of deviations.

The maximum likelihood estimator (MLE) uses the parameter values that make the density of the data largest.

How Added

Calculate \(\hat\beta_1\) and \(\hat\beta_0\)

Calculate \(\bar x\) and \(\bar y\).
Calculate \(x_i-\bar x\) and \(y_i-\bar y\) for each observation.
Calculate \(S_{xx}=\sum(x_i-\bar x)^2\).
Calculate \(S_{xy}=\sum(x_i-\bar x)(y_i-\bar y)\).
Calculate \(\hat\beta_1=S_{xy}/S_{xx}\).
Calculate \(\hat\beta_0=\bar y-\hat\beta_1\bar x\).

Self-check: \(S_{xx}>0\). \(\hat\beta_1\) has the sign of \(S_{xy}\). The line goes through \((\bar x,\bar y)\): \(\hat\beta_0+\hat\beta_1\bar x=\bar y\).

Example 1 · The small data set Added How steps 1–6

Step 1: sample means.

\(\bar x=\dfrac{1+2+3+4+5}{5}=\dfrac{15}{5}=3\)

\(\bar y=\dfrac{2+4+5+4+5}{5}=\dfrac{20}{5}=4\)

Step 2: deviations.

\(x_i-\bar x\): \(1-3,\ 2-3,\ 3-3,\ 4-3,\ 5-3=-2,\ -1,\ 0,\ 1,\ 2\).

\(y_i-\bar y\): \(2-4,\ 4-4,\ 5-4,\ 4-4,\ 5-4=-2,\ 0,\ 1,\ 0,\ 1\).

Step 3–4: sums.

\(S_{xx}=(-2)^2+(-1)^2+0^2+1^2+2^2\)

\(=4+1+0+1+4=10\)

\(S_{xy}=(-2)(-2)+(-1)(0)+(0)(1)+(1)(0)+(2)(1)\)

\(=4+0+0+0+2=6\)

Step 5–6: slope and intercept.

\(\hat\beta_1=\dfrac{S_{xy}}{S_{xx}}=\dfrac{6}{10}=0.6\)

\(\hat\beta_0=\bar y-\hat\beta_1\bar x=4-0.6\times3\)

\(=4-1.8=2.2\)

Self-check: \(S_{xx}=10>0\). \(S_{xy}=6\) and \(\hat\beta_1=0.6\) are both positive. \(\hat\beta_0+\hat\beta_1\bar x=2.2+0.6\times3=2.2+1.8=4=\bar y\).

Fitted line: \(\hat y=2.2+0.6x\).

Example 2 · The brainhead data Slides p.45

Slides p.45: \(\hat\beta_0=335.43231\) g, \(\hat\beta_1=0.26082\) g per cm³. Fitted line: \(\hat y=335.43231+0.26082x\).

Figure 1-1 · Brainhead data (blue, \(n=236\)) and the least squares line from Slides p.45 (orange). The line goes through \((\bar x,\bar y)\) (green).

Results for later units

Small data set: \(n=5\), \(\bar x=3\), \(\bar y=4\), \(S_{xx}=10\), \(S_{xy}=6\), \(\hat\beta_1=0.6\), \(\hat\beta_0=2.2\).

Brainhead data (Slides p.45): \(n=236\), \(\hat\beta_0=335.43231\), \(\hat\beta_1=0.26082\).

Why Added

Set the two partial derivatives of Slides p.6 to 0. Solve for \(\hat\beta_0\).

\(\dfrac{\partial S}{\partial\beta_0}=\sum_{i=1}^n2(y_i-\beta_0-\beta_1x_i)(-1)\)

chain rule

\(0=\sum_{i=1}^n(y_i-\hat\beta_0-\hat\beta_1x_i)\)

set to 0, divide by \(-2\)

\(0=\sum_{i=1}^ny_i-n\hat\beta_0-\hat\beta_1\sum_{i=1}^nx_i\)

\(n\hat\beta_0=\sum_{i=1}^ny_i-\hat\beta_1\sum_{i=1}^nx_i\)

\(\hat\beta_0=\frac1n\sum_{i=1}^ny_i-\hat\beta_1\frac1n\sum_{i=1}^nx_i\)

\(\hat\beta_0=\bar y-\hat\beta_1\bar x\)

definition of \(\bar x,\bar y\)

Solve for \(\hat\beta_1\).

\(\dfrac{\partial S}{\partial\beta_1}=\sum_{i=1}^n2(y_i-\beta_0-\beta_1x_i)(-x_i)\)

chain rule

\(0=\sum_{i=1}^n(y_i-\hat\beta_0-\hat\beta_1x_i)x_i\)

set to 0, divide by \(-2\)

\(0=\sum_{i=1}^ny_ix_i-\hat\beta_0n\bar x-\hat\beta_1\sum_{i=1}^nx_i^2\)

\(\sum x_i=n\bar x\)

\(0=\sum_{i=1}^ny_ix_i-(\bar y-\hat\beta_1\bar x)n\bar x-\hat\beta_1\sum_{i=1}^nx_i^2\)

put in \(\hat\beta_0=\bar y-\hat\beta_1\bar x\)

\(0=\sum_{i=1}^ny_ix_i-n\bar y\bar x+\hat\beta_1n\bar x^2-\hat\beta_1\sum_{i=1}^nx_i^2\)

\(0=\sum_{i=1}^ny_ix_i-n\bar y\bar x+\hat\beta_1\Big(n\bar x^2-\sum_{i=1}^nx_i^2\Big)\)

\(\hat\beta_1=\dfrac{\sum_{i=1}^ny_ix_i-n\bar y\bar x}{\sum_{i=1}^nx_i^2-n\bar x^2}\)

The denominator is \(S_{xx}\).

\(S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2\)

\(=\sum_{i=1}^nx_i^2-2\bar x\sum_{i=1}^nx_i+n\bar x^2\)

\(=\sum_{i=1}^nx_i^2-2n\bar x^2+n\bar x^2\)

\(\sum x_i=n\bar x\)

\(=\sum_{i=1}^nx_i^2-n\bar x^2\)

The numerator is \(S_{xy}\).

\(S_{xy}=\sum_{i=1}^n(x_i-\bar x)(y_i-\bar y)\)

\(=\sum_{i=1}^nx_iy_i-\bar y\sum_{i=1}^nx_i-\bar x\sum_{i=1}^ny_i+n\bar x\bar y\)

\(=\sum_{i=1}^nx_iy_i-n\bar x\bar y-n\bar x\bar y+n\bar x\bar y\)

\(\sum x_i=n\bar x,\ \sum y_i=n\bar y\)

\(=\sum_{i=1}^nx_iy_i-n\bar x\bar y\)

∎ \(\hat\beta_1=S_{xy}/S_{xx}\).

Small data check: \(\sum x_i^2=55\). \(\sum x_i^2-n\bar x^2=55-5\times3^2=55-45=10=S_{xx}\).

Small data check: \(\sum x_iy_i=66\). \(\sum x_iy_i-n\bar x\bar y=66-5\times3\times4=66-60=6=S_{xy}\).

01.5Fitted values, residuals, and \(\hat\sigma^2\) Slides p.8

What Slides p.8

Fitted Values, Residuals, and σ̂² · Lecture 3 · p.8 Fitted values: \[\hat y_i=\hat\beta_0+\hat\beta_1x_i\] Residuals: \[e_i=y_i-\hat y_i=y_i-(\hat\beta_0+\hat\beta_1x_i)\] • Note: \(e_i\) are different from errors, \(\epsilon_i=y_i-(\beta_0+\beta_1x_i)\), and have different properties
Estimate of \(\sigma^2\) is: \[\hat\sigma^2=\frac{\sum_{i=1}^n(y_i-\hat y_i)^2}{n-2}=\frac{\sum_{i=1}^ne_i^2}{n-2}\] • This is not the MLE—instead it is an unbiased estimate of \(\sigma^2\)

The fitted value \(\hat y_i\) is the height of the fitted line at \(x=x_i\).

The residual \(e_i\) is the observed value minus the fitted value. It replaces \(\epsilon_i\), which needs the unknown population line.

The sum of squared residuals \(\sum_{i=1}^ne_i^2\) is the minimum of \(S\), at the least squares line.

An estimator is unbiased when its expected value equals the parameter: \(E[\hat\sigma^2]=\sigma^2\).

\(\hat\sigma=\sqrt{\hat\sigma^2}\) has the unit of \(y\).

How Added

Calculate \(\hat y_i\), \(e_i\), and \(\hat\sigma^2\)

Calculate each fitted value \(\hat y_i=\hat\beta_0+\hat\beta_1x_i\).
Calculate each residual \(e_i=y_i-\hat y_i\).
Square each residual. Add the squares to get \(\sum e_i^2\).
Divide the sum by \(n-2\) to get \(\hat\sigma^2\).
Take the square root to get \(\hat\sigma\).

Self-check: \(\sum e_i=0\), up to rounding. \(\hat\sigma^2>0\). The denominator is \(n-2\), not \(n\).

Example 1 · The small data set Added How steps 1–5

Step 1: fitted values.

\(\hat\beta_1x_i:\ 0.6(1),\ 0.6(2),\ 0.6(3),\ 0.6(4),\ 0.6(5)=0.6,\ 1.2,\ 1.8,\ 2.4,\ 3.0\)

\(\hat y_i:\ 2.2+0.6,\ 2.2+1.2,\ 2.2+1.8,\ 2.2+2.4,\ 2.2+3.0\)

\(\hat y_i:\ 2.8,\ 3.4,\ 4.0,\ 4.6,\ 5.2\)

Step 2: residuals.

\(e_i:\ 2-2.8,\ 4-3.4,\ 5-4.0,\ 4-4.6,\ 5-5.2\)

\(e_i:\ -0.8,\ 0.6,\ 1.0,\ -0.6,\ -0.2\)

Self-check: \(-0.8+0.6+1.0-0.6-0.2=0\).

Figure 1-2 (Added) · Blue: observations \((x_i,y_i)\). Orange: fitted values \((x_i,\hat y_i)\). Each purple segment has length \(|e_i|\). Points above the line have \(e_i>0\). Points below have \(e_i<0\).

Step 3–5: squared residuals and \(\hat\sigma^2\).

\(e_i^2:\ (-0.8)^2,\ 0.6^2,\ 1.0^2,\ (-0.6)^2,\ (-0.2)^2=0.64,\ 0.36,\ 1.00,\ 0.36,\ 0.04\)

\(\sum_{i=1}^5e_i^2=0.64+0.36+1.00+0.36+0.04=2.4\)

\(\hat\sigma^2=\dfrac{2.4}{5-2}=\dfrac{2.4}{3}=0.8\)

\(\hat\sigma=\sqrt{0.8}=0.8944\)

A typical point is about 0.89 from the line.

Example 2 · The brainhead data Slides p.45

Slides p.45: \(\hat\sigma=72.35\) g, with 234 degrees of freedom (the denominator \(n-2\)).

\(n-2=236-2=234\)

\(\hat\sigma^2=72.35^2=5234.52\ \text{g}^2\)

from the rounded \(\hat\sigma\)

A typical brain weight is about 72.35 g from the line.

Results for later units (continued)

Small data set: \(\sum e_i^2=2.4\), \(\hat\sigma^2=0.8\), \(\hat\sigma=0.8944\), \(n-2=3\).

Brainhead data (Slides p.45): \(\hat\sigma=72.35\), \(n-2=234\).

Why · \(\sum e_i=0\) and the denominator \(n-2\) Added

\(0=\sum_{i=1}^n(y_i-\hat\beta_0-\hat\beta_1x_i)\)

01.4, \(\hat\beta_0\) chain, line 2

\(0=\sum_{i=1}^n(y_i-\hat y_i)\)

definition of \(\hat y_i\)

\(0=\sum_{i=1}^ne_i\)

definition of \(e_i\)

The chi-squared distribution \(\chi^2_\nu\) has non-negative values, \(\nu\) degrees of freedom, and mean \(E[\chi^2_\nu]=\nu\). The course gives \(\frac{1}{\sigma^2}\sum_{i=1}^ne_i^2\sim\chi^2_{(n-2)}\).

\(E[\hat\sigma^2]=E\left[\dfrac{\sum_{i=1}^ne_i^2}{n-2}\right]\)

\(=\sigma^2E\left[\dfrac{1}{\sigma^2}\dfrac{\sum_{i=1}^ne_i^2}{n-2}\right]\)

\(=\dfrac{\sigma^2}{n-2}E\left[\dfrac{1}{\sigma^2}\sum_{i=1}^ne_i^2\right]\)

linearity of expectation

\(=\dfrac{\sigma^2}{n-2}(n-2)\)

mean of \(\chi^2_{(n-2)}\) is \(n-2\)

\(=\sigma^2\)

∎ The "2" counts the estimated line parameters \(\beta_0\) and \(\beta_1\).

01.6Practice Added

Q1. A data set has \(x=(1,2,3,4,5)\) and \(y=(2,4,5,4,5)\). The least squares line is \(\hat y=2.2+0.6x\). Calculate \(\hat y_3\) and \(e_3\). Is point 3 above or below the line?

Answer

\(\hat y_3=2.2+0.6\times3\)

\(=2.2+1.8=4.0\)

\(e_3=y_3-\hat y_3=5-4.0=1.0\)

\(e_3>0\): point 3 is above the line.

Q2. For the brainhead data, \(\hat\beta_1=0.26082\) (Slides p.45). Interpret \(\hat\beta_1\). Then estimate the mean brain weight difference between head sizes 4000 cm³ and 3000 cm³.

Answer

Compare two populations whose head sizes differ by 1 cm³. The estimated mean brain weight is 0.26082 g higher at the larger head size.

\(k=4000-3000=1000\)

\(k\hat\beta_1=1000\times0.26082\)

01.3, How step 5

\(=260.82\) g

Q3. For the small data set, \(\sum_{i=1}^5e_i^2=2.4\). Calculate \(\hat\sigma^2\) and \(\hat\sigma\). Why is the denominator \(n-2\)?

Answer

\(\hat\sigma^2=\dfrac{2.4}{5-2}=\dfrac{2.4}{3}\)

\(=0.8\)

\(\hat\sigma=\sqrt{0.8}=0.8944\)

\(\frac{1}{\sigma^2}\sum e_i^2\sim\chi^2_{(n-2)}\) has mean \(n-2\). Dividing by \(n-2\) gives \(E[\hat\sigma^2]=\sigma^2\) (unbiased).

Unit 02Sampling distribution and unbiasedness of \(\hat\beta_1\)

Plan · Slides p.9–18

Step 1: Title page (Slides p.9).

Step 2: Different samples give different estimates (Slides p.10).

Step 3: Section page (Slides p.11).

Step 4: Write \(\hat\beta_1=\sum w_iy_i\) (Slides p.12).

Step 5: Find the distribution of \(\hat\beta_1\) (Slides p.13).

Step 6: Show \(E[\hat\beta_1]=\beta_1\) (Slides p.14–18).

02.1Title page Slides p.9

Title: Lecture 3: Simple Linear Regression: Inference. New material starts on this page.

02.2Why we need inference Slides p.10

What Slides p.10

Inference for \(\beta\) · Lecture 3 · p.10 (slide 7/25) In two different samples, we will get different estimates of β.
How do we acknowledge this variability?
How do we characterize uncertainty in our estimates?
• Standard errors
• Confidence intervals
• Hypothesis tests

A sample is the \(n\) data pairs \((x_i,y_i)\) that we have. The brainhead sample (\(n=236\)) gives \(\hat\beta_1=0.2608207\) (Lecture 2 · p.38).

Variability: a different sample gives a different estimate.

Uncertainty: the distance from the estimate to the true \(\beta_1\) is unknown.

Standard errors, confidence intervals, and hypothesis tests describe uncertainty. All three need the distribution of \(\hat\beta_1\) over samples.

How Added

Compare the estimates from two samples

Divide the data into two samples.
For each sample, calculate \(\bar x\) and \(\bar y\).
Calculate \(S_{xy}=\sum_i(y_i-\bar y)(x_i-\bar x)\) and \(S_{xx}=\sum_i(x_i-\bar x)^2\).
Calculate \(\hat\beta_1=S_{xy}/S_{xx}\).
Compare the two values.

Self-check: Keep each pair \((x_i,y_i)\) together in one sample.

Example · Two samples of three observations Added

The first six brainhead observations (Lecture 2 · p.10), \(x\) in cm³, \(y\) in g: \((4512,1530)\), \((3738,1297)\), \((4261,1335)\), \((3777,1282)\), \((4177,1590)\), \((3585,1300)\).

Sample A: observations 1–3. Sample B: observations 4–6.

Sample A, step 2:

\[\bar x=\frac{4512+3738+4261}{3}=\frac{12511}{3}=4170.333\]

\[\bar y=\frac{1530+1297+1335}{3}=\frac{4162}{3}=1387.333\]

Sample A, step 3. Deviations: \(x_i-\bar x=341.667,\ -432.333,\ 90.667\); \(y_i-\bar y=142.667,\ -90.333,\ -52.333\).

\[S_{xy}=(142.667)(341.667)+(-90.333)(-432.333)+(-52.333)(90.667)\]

\[=48744.444+39054.111-4744.889\]

\[=83053.667\]

\[S_{xx}=341.667^2+(-432.333)^2+90.667^2\]

\[=116736.111+186912.111+8220.444\]

\[=311868.667\]

Sample A, step 4:

\[\hat\beta_1=\frac{83053.667}{311868.667}=0.2663\]

Sample B, step 2:

\[\bar x=\frac{3777+4177+3585}{3}=\frac{11539}{3}=3846.333\]

\[\bar y=\frac{1282+1590+1300}{3}=\frac{4172}{3}=1390.667\]

Sample B, step 3. Deviations: \(x_i-\bar x=-69.333,\ 330.667,\ -261.333\); \(y_i-\bar y=-108.667,\ 199.333,\ -90.667\).

\[S_{xy}=(-108.667)(-69.333)+(199.333)(330.667)+(-90.667)(-261.333)\]

\[=7534.222+65912.889+23694.222\]

\[=97141.333\]

\[S_{xx}=(-69.333)^2+330.667^2+(-261.333)^2\]

\[=4807.111+109340.444+68295.111\]

\[=182442.667\]

Sample B, step 4:

\[\hat\beta_1=\frac{97141.333}{182442.667}=0.5324\]

Step 5:

Sample A: \(\hat\beta_1=0.2663\). Sample B: \(\hat\beta_1=0.5324\). All 236 observations: \(\hat\beta_1=0.2608207\) (Lecture 2 · p.38).

We do not know which value is nearest to the true \(\beta_1\).

Why Added

\(\hat\beta_1\) is a function of the \(y_i=\beta_0+\beta_1x_i+\epsilon_i\). A new sample has new random \(\epsilon_i\), new \(y_i\), and a new \(\hat\beta_1\).

A random variable is a quantity whose value comes from a random outcome. \(\hat\beta_1\) is a random variable.

02.3Section page: properties of the LS estimator Slides p.11

Section title: "Properties of LS Estimator". The least squares (LS) estimator of the slope is \(\hat\beta_1=S_{xy}/S_{xx}\).

02.4Write \(\hat\beta_1\) as a linear combination of the \(y_i\) Slides p.12

What Slides p.12

Properties of \(\hat\beta_1\) · Lecture 3 · p.12 (slide 8/25) Recall: \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\) \[\hat\beta_1=\frac{\sum_i(y_i-\bar y)(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}=\frac{\sum_i y_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}=\sum_{i=1}^n w_iy_i\] where \(w_i=\frac{(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\) are fixed with respect to \(y\).
Hence \(\hat\beta_1\) is a linear combination of independent Normals!

The denominator is \(S_{xx}\). Its index \(j\) keeps it apart from the numerator index \(i\).

Weight \(w_i\): the constant that multiplies \(y_i\) in \(\hat\beta_1\).

Fixed with respect to \(y\): \(w_i\) uses only the \(x_i\), and the model treats each \(x_i\) as a known constant.

Linear combination: a sum \(a_1y_1+a_2y_2+\dots+a_ny_n\) with constants \(a_i\).

How Added

Calculate the \(w_i\) and use them to get \(\hat\beta_1\)

Calculate \(\bar x\).
Calculate \(S_{xx}=\sum_{j=1}^n(x_j-\bar x)^2\).
For each \(i\), calculate \(w_i=(x_i-\bar x)/S_{xx}\).
Calculate \(\sum_{i=1}^n w_iy_i\).
Compare the result with \(S_{xy}/S_{xx}\).

Self-check: (1) \(\sum w_iy_i=S_{xy}/S_{xx}\). (2) \(\sum w_i=0\). (3) \(\sum w_ix_i=1\).

(4) \(x_i>\bar x\) gives \(w_i>0\). \(x_i<\bar x\) gives \(w_i<0\). Section 02.6 shows the reasons for (2) and (3).

Example · Six observations of the brainhead data Added

Data: the six observations of Section 02.2.

Step 1:

\[\bar x=\frac{4512+3738+4261+3777+4177+3585}{6}\]

\[=\frac{24050}{6}\]

\[=4008.333\]

Step 2. Deviations \(x_i-\bar x\): \(503.667,\ -270.333,\ 252.667,\ -231.333,\ 168.667,\ -423.333\).

\[S_{xx}=503.667^2+(-270.333)^2+252.667^2+(-231.333)^2+168.667^2+(-423.333)^2\]

\[=253680.111+73080.111+63840.444+53515.111+28448.444+179211.111\]

\[=651775.333\]

Step 3: divide each deviation by \(S_{xx}=651775.333\).

\(w_1=503.667/651775.333=7.7276\times10^{-4}\)

\(w_2=-270.333/651775.333=-4.1476\times10^{-4}\)

\(w_3=252.667/651775.333=3.8766\times10^{-4}\)

\(w_4=-231.333/651775.333=-3.5493\times10^{-4}\)

\(w_5=168.667/651775.333=2.5878\times10^{-4}\)

\(w_6=-423.333/651775.333=-6.4951\times10^{-4}\)

Step 4:

\[\sum w_iy_i=(7.7276\times10^{-4})(1530)+(-4.1476\times10^{-4})(1297)+\dots+(-6.4951\times10^{-4})(1300)\]

\[=1.18232-0.53795+0.51752-0.45502+0.41146-0.84436\]

\[=0.2740\]

Rounded terms sum to 0.27397; unrounded, 0.27398.

Step 5: \(\bar y=8334/6=1389\), and \(y_i-\bar y=141,\ -92,\ -54,\ -107,\ 201,\ -89\).

\[S_{xy}=(141)(503.667)+(-92)(-270.333)+\dots+(-89)(-423.333)\]

\[=71017+24870.667-13644+24752.667+33902+37676.667\]

\[=178575\]

\[\frac{S_{xy}}{S_{xx}}=\frac{178575}{651775.333}=0.2740\]

Check (1): \(\sum w_iy_i=0.2740=S_{xy}/S_{xx}\).

Check (2): \(\sum w_i=(503.667-270.333+252.667-231.333+168.667-423.333)/651775.333=0/651775.333=0\).

Check (3): \(\sum w_ix_i=3.48670-1.55039+1.65182-1.34056+1.08093-2.32849=1.00001\). The exact value is 1. The 0.00001 comes from rounding.

Check (4): observations 1, 3, 5 have \(x_i>\bar x=4008.333\) and \(w_i>0\). Observations 2, 4, 6 have \(x_i<\bar x\) and \(w_i<0\).

Figure 2-1 (Added): the 236 brainhead weights \(w_i\) against \(x_i\). Orange dashed line: \(\bar x\). Grey dashed line: \(w=0\). All points lie on one line through \((\bar x,0)\) with slope \(1/S_{xx}\). A point far from \(\bar x\) has a large \(|w_i|\).

Why Slides p.12

Slides p.12 does not prove the first equals sign. The proof needs \(\sum_i(x_i-\bar x)=0\).

Helper result: the deviations add to 0.

\[\sum_{i=1}^n(x_i-\bar x)=\sum_{i=1}^n x_i-\sum_{i=1}^n\bar x\]

\[=\sum_{i=1}^n x_i-n\bar x\]

Adds the constant \(\bar x\) \(n\) times.

\[=n\bar x-n\bar x\]

\(\bar x=\frac1n\sum_{i=1}^n x_i\).

\[=0\]

The numerator:

\[\sum_{i=1}^n(y_i-\bar y)(x_i-\bar x)=\sum_{i=1}^n\left[y_i(x_i-\bar x)-\bar y(x_i-\bar x)\right]\]

\[=\sum_{i=1}^n y_i(x_i-\bar x)-\sum_{i=1}^n\bar y(x_i-\bar x)\]

\[=\sum_{i=1}^n y_i(x_i-\bar x)-\bar y\sum_{i=1}^n(x_i-\bar x)\]

\(\bar y\) has no index \(i\).

\[=\sum_{i=1}^n y_i(x_i-\bar x)-\bar y\cdot0\]

Helper result.

\[=\sum_{i=1}^n y_i(x_i-\bar x)\]

Check on the six observations: the terms \(y_i(x_i-\bar x)\) are \(770610-350622.333+337310-296569.333+268180-550333.333\). Their sum, \(178575\), is \(S_{xy}\).

The split into \(\sum w_iy_i\):

\[\hat\beta_1=\frac{\sum_{i=1}^n y_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=\sum_{i=1}^n\frac{y_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

The denominator has no index \(i\).

\[=\sum_{i=1}^n\frac{(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\,y_i\]

\[=\sum_{i=1}^n w_iy_i\]

Definition of \(w_i\). ∎

02.5The sampling distribution of \(\hat\beta_1\) Slides p.13

What Slides p.13

Properties of \(\hat\beta_1\) · Lecture 3 · p.13 (slide 8/25) Hence \(\hat\beta_1\) is a linear combination of independent Normals! \[\Longrightarrow\ \hat\beta_1\sim N\Big(\sum_{i=1}^n w_i(\beta_0+\beta_1x_i),\ \sigma^2\sum_{i=1}^n w_i^2\Big)\]

Sampling distribution: the distribution of an estimator over all possible samples.

Expectation \(E[\cdot]\): the mean of a random variable.

Rule: a linear combination of independent Normals is Normal. If \(y_i\overset{indep}{\sim}N(\mu_i,\sigma_i^2)\) and the \(a_i\) are constants, then \(\sum a_iy_i\sim N(\sum a_i\mu_i,\ \sum a_i^2\sigma_i^2)\).

How Added

Write the distribution of a linear combination

Write the estimator as \(\sum a_iy_i\), with no \(y\) in any \(a_i\) (for \(\hat\beta_1\), \(a_i=w_i\)).
Make sure that the \(y_i\) are independent and Normal.
Write each mean \(\mu_i\) and variance \(\sigma_i^2\). For SLR, \(\mu_i=\beta_0+\beta_1x_i\) and \(\sigma_i^2=\sigma^2\).
Calculate the mean \(\sum a_i\mu_i\).
Calculate the variance \(\sum a_i^2\sigma_i^2\).
Write \(N(\text{mean},\ \text{variance})\).

Self-check: The variance uses \(w_i^2\). The mean uses \(w_i\). The variance is positive.

Example · Six head sizes with supposed parameters Added

The true parameters are unknown. Suppose they equal the brainhead estimates:

\(\beta_0=335.4323150\) and \(\beta_1=0.2608207\) (Lecture 2 · p.38).

\(\sigma=72.35\), the \(\hat\sigma\) of Slides p.45.

The \(x_i\) and \(w_i\) are the six values of Section 02.4.

Steps 1–3: \(a_i=w_i\), and \(\mu_i=\beta_0+\beta_1x_i\). Observation 1:

\[\mu_1=335.4323150+0.2608207\times4512\]

\[=335.4323150+1176.8229984\]

\[=1512.2553\]

\(\mu_1,\dots,\mu_6=1512.2553,\ 1310.3801,\ 1446.7893,\ 1320.5521,\ 1424.8804,\ 1270.4745\): the fitted values on Lecture 2 · p.38.

Step 4:

\[\sum w_i\mu_i=(7.7276\times10^{-4})(1512.2553)+\dots+(-6.4951\times10^{-4})(1270.4745)\]

\[=1.16861-0.54350+0.56086-0.46870+0.36873-0.82518\]

\[=0.26082\]

Step 5:

\[\sum w_i^2=5.9716\times10^{-7}+1.7203\times10^{-7}+1.5028\times10^{-7}+1.2597\times10^{-7}+0.66967\times10^{-7}+4.2186\times10^{-7}\]

\[=1.5343\times10^{-6}\]

\[\sigma^2\sum w_i^2=72.35^2\times1.5343\times10^{-6}\]

\[=5234.5225\times1.5343\times10^{-6}\]

\[=0.00803\]

Step 6:

For these six head sizes, \(\hat\beta_1\sim N(0.26082,\ 0.00803)\).

The mean equals the supposed \(\beta_1=0.2608207\). Section 02.6 shows that this is always true.

Figure 2-2 uses all 236 head sizes and the same supposed parameters. A computer made 10000 new sets of \(y\) from the model. Each set gives one \(\hat\beta_1=\sum w_iy_i\).

Figure 2-2 (Added): histogram of 10000 simulated \(\hat\beta_1\). The shape is the Normal bell. The center is near the supposed \(\beta_1\) (orange line).

Why Slides p.13

The rule gives the Normal shape. The chains give the mean and the variance.

The mean:

\[E[\hat\beta_1]=E\Big[\sum_{i=1}^n w_iy_i\Big]\]

Section 02.4.

\[=\sum_{i=1}^n w_iE[y_i]\]

Linearity of expectation; each \(w_i\) is a constant.

\[=\sum_{i=1}^n w_i(\beta_0+\beta_1x_i)\]

Model: \(E[y_i]=\beta_0+\beta_1x_i\).

The variance:

\[\mathrm{Var}[\hat\beta_1]=\mathrm{Var}\Big[\sum_{i=1}^n w_iy_i\Big]\]

\[=\sum_{i=1}^n\mathrm{Var}[w_iy_i]\]

The \(y_i\) are independent.

\[=\sum_{i=1}^n w_i^2\,\mathrm{Var}[y_i]\]

\(\mathrm{Var}[aY]=a^2\mathrm{Var}[Y]\).

\[=\sum_{i=1}^n w_i^2\sigma^2\]

Model: \(\mathrm{Var}[y_i]=\sigma^2\).

\[=\sigma^2\sum_{i=1}^n w_i^2\]

∎

02.6The mean of \(\hat\beta_1\): unbiasedness Slides p.14–18

What Slides p.14–18

Mean of \(\hat\beta_1\) · Lecture 3 · p.14–18 (slide 9/25) \[\begin{aligned}E[\hat\beta_1]&=\sum_{i=1}^n w_i(\beta_0+\beta_1x_i)\\&=\sum_{i=1}^n\frac{(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}(\beta_0+\beta_1x_i)\\&=\beta_0\frac{\sum_{i=1}^n(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}+\beta_1\frac{\sum_{i=1}^n x_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\\&=0+\beta_1\frac{\sum_{i=1}^n(x_i-\bar x)(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\\&=\beta_1\end{aligned}\] Hence it is unbiased.

Over many repeated samples, the mean of \(\hat\beta_1\) is \(\beta_1\). One \(\hat\beta_1\) can still differ from \(\beta_1\).

Pages 14–18 build one slide, one line per page. The quote copies p.18.

How Added

Show that the linear estimator \(\sum w_iy_i\) is unbiased

Write \(E[\hat\beta_1]=\sum w_iE[y_i]=\sum w_i(\beta_0+\beta_1x_i)\).
Split the sum: \(\beta_0\sum w_i+\beta_1\sum w_ix_i\).
Show \(\sum w_i=0\) with \(\sum(x_i-\bar x)=0\).
Show \(\sum w_ix_i=1\): the numerator \(\sum x_i(x_i-\bar x)\) equals the denominator \(\sum(x_j-\bar x)^2\).
Substitute: \(\beta_0\cdot0+\beta_1\cdot1=\beta_1\).

Self-check: The result contains no \(\beta_0\) and no \(x_i\).

Example · Six head sizes with supposed parameters Added

Data: Section 02.4, with the supposed \(\beta_0=335.4323150\) and \(\beta_1=0.2608207\).

Step 3: \(\sum w_i=0\) (check (2), Section 02.4). \(\beta_0\sum w_i=335.4323150\times0=0\).

Step 4: \(\sum w_ix_i=1\) (check (3), Section 02.4). \(\beta_1\sum w_ix_i=0.2608207\times1=0.2608207\).

Step 5: \(\sum w_i(\beta_0+\beta_1x_i)=0+0.2608207=0.2608207\), the supposed \(\beta_1\).

Section 02.5 got 0.26082 term by term. The results agree.

The 10000 simulated values of Figure 2-2 have mean 0.26063, near \(\beta_1=0.2608207\).

Why Slides p.14–18

Slides p.18 gives no reason for two steps: "\(=0\)", and the change from \(\sum x_i(x_i-\bar x)\) to \(\sum(x_i-\bar x)(x_i-\bar x)\).

Helper result: \(\sum x_i(x_i-\bar x)=\sum(x_i-\bar x)^2\).

\[\sum_{i=1}^n(x_i-\bar x)(x_i-\bar x)=\sum_{i=1}^n\left[x_i(x_i-\bar x)-\bar x(x_i-\bar x)\right]\]

\[=\sum_{i=1}^n x_i(x_i-\bar x)-\bar x\sum_{i=1}^n(x_i-\bar x)\]

\(\bar x\) has no index \(i\).

\[=\sum_{i=1}^n x_i(x_i-\bar x)-\bar x\cdot0\]

Helper result of Section 02.4.

\[=\sum_{i=1}^n x_i(x_i-\bar x)\]

Check on the six observations: the terms \(x_i(x_i-\bar x)\) are \(2272544-1010506+1076612.667-873746+704520.667-1517650\). Their sum, \(651775.333\), is \(S_{xx}\).

The main chain (p.18):

\[E[\hat\beta_1]=\sum_{i=1}^n w_i(\beta_0+\beta_1x_i)\]

Section 02.5. p.14.

\[=\sum_{i=1}^n\frac{(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}(\beta_0+\beta_1x_i)\]

Definition of \(w_i\). p.15.

\[=\sum_{i=1}^n\frac{\beta_0(x_i-\bar x)+\beta_1x_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=\sum_{i=1}^n\frac{\beta_0(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}+\sum_{i=1}^n\frac{\beta_1x_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=\beta_0\frac{\sum_{i=1}^n(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}+\beta_1\frac{\sum_{i=1}^n x_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

Constants have no index \(i\). p.16.

\[=\beta_0\frac{0}{\sum_{j=1}^n(x_j-\bar x)^2}+\beta_1\frac{\sum_{i=1}^n x_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

Helper result of Section 02.4.

\[=0+\beta_1\frac{\sum_{i=1}^n x_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=0+\beta_1\frac{\sum_{i=1}^n(x_i-\bar x)(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

Helper result above. p.17.

\[=\beta_1\frac{\sum_{i=1}^n(x_i-\bar x)^2}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=\beta_1\]

Same sum, different index letter. p.18. ∎

With Section 02.5: \(\hat\beta_1\sim N(\beta_1,\ \sigma^2\sum w_i^2)\).

02.7Practice Added

Q1. A data set has three pairs: \((x_i,y_i)=(1,2),\ (3,3),\ (5,7)\). Calculate \(w_1,w_2,w_3\). Calculate \(\hat\beta_1=\sum w_iy_i\) and compare it with \(S_{xy}/S_{xx}\).

Answer

Step 1: \(\bar x=(1+3+5)/3=9/3=3\).

Step 2: \(x_i-\bar x=-2,\ 0,\ 2\). \(S_{xx}=(-2)^2+0^2+2^2=4+0+4=8\).

Step 3: \(w_1=-2/8=-0.25\), \(w_2=0/8=0\), \(w_3=2/8=0.25\).

Step 4: \(\sum w_iy_i=(-0.25)(2)+(0)(3)+(0.25)(7)=-0.5+0+1.75=1.25\).

Step 5: \(\bar y=(2+3+7)/3=12/3=4\), and \(y_i-\bar y=-2,\ -1,\ 3\). \(S_{xy}=(-2)(-2)+(-1)(0)+(3)(2)=4+0+6=10\). \(S_{xy}/S_{xx}=10/8=1.25\).

Self-check: \(\sum w_i=-0.25+0+0.25=0\). \(\sum w_ix_i=(-0.25)(1)+(0)(3)+(0.25)(5)=-0.25+0+1.25=1\).

Q2. Let \(w_i=\frac{(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\). Show that \(\sum_{i=1}^n w_i=0\) and \(\sum_{i=1}^n w_ix_i=1\).

Answer

\[\sum_{i=1}^n w_i=\frac{\sum_{i=1}^n(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

The denominator has no index \(i\).

\[=\frac{\sum_{i=1}^n x_i-n\bar x}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=\frac{n\bar x-n\bar x}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\(\sum x_i=n\bar x\).

\[=0\]

\[\sum_{i=1}^n w_ix_i=\frac{\sum_{i=1}^n x_i(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=\frac{\sum_{i=1}^n x_i(x_i-\bar x)-\bar x\sum_{i=1}^n(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

The new term is \(\bar x\cdot0=0\).

\[=\frac{\sum_{i=1}^n(x_i-\bar x)(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\]

\[=1\]

Q3. Use the three \(x\) values of Q1. Suppose that \(\beta_0=1\), \(\beta_1=1\), and \(\sigma=2\). Write the sampling distribution of \(\hat\beta_1\). Which two parts of the model make \(\hat\beta_1\) exactly Normal?

Answer

Means: \(\mu_i=1+x_i\), and \(\mu_1=2\), \(\mu_2=4\), \(\mu_3=6\).

Mean of \(\hat\beta_1\): \(\sum w_i\mu_i=(-0.25)(2)+(0)(4)+(0.25)(6)=-0.5+0+1.5=1=\beta_1\).

\(\sum w_i^2=0.0625+0+0.0625=0.125\).

Variance of \(\hat\beta_1\): \(\sigma^2\sum w_i^2=4\times0.125=0.5\).

Result: \(\hat\beta_1\sim N(1,\ 0.5)\).

(1) Each \(y_i\) is Normal. (2) The \(y_i\) are independent. With constant \(w_i\), \(\sum w_iy_i\) is a linear combination of independent Normals.

Unit 03Variance and distribution of \(\hat\beta_1\) and \(\hat\beta_0\)

Plan · Slides p.19–24

Step 1: Calculate \(\mathrm{Var}[\hat\beta_1]=\sigma^2/S_{xx}\) (Slides p.19–23).

Step 2: State the distribution of \(\hat\beta_1\) (Slides p.24, top).

Step 3: State the distribution of \(\hat\beta_0\) (Slides p.24, bottom).

Step 4: Practice questions (Added).

Model (Slides p.12): \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\ \sigma^2)\), \(i=1,\dots,n\).

\(\hat\beta_1=\sum_{i=1}^n w_iy_i\), with \(w_i=\dfrac{x_i-\bar x}{\sum_{j=1}^n(x_j-\bar x)^2}\) (Slides p.12). Each \(w_i\) is a constant.

From Unit 02: \(\mathrm{Var}[\hat\beta_1]=\sigma^2\sum_{i=1}^n w_i^2\) (Slides p.13) and \(E[\hat\beta_1]=\beta_1\) (Slides p.18).

Examples use the first 6 brainhead head sizes (Lecture 2 · p.10). Added

\(i\)	1	2	3	4	5	6
\(x_i\) (cm³)	4512	3738	4261	3777	4177	3585

The formulas use only \(x\) and \(\sigma\). We suppose \(\sigma=72.35\) g (\(\hat\sigma\) on Slides p.45), so \(\sigma^2=72.35^2=5234.5225\).

03.1Variance of \(\hat\beta_1\) Slides p.19–23

What Slides p.19–23

Variance of \(\hat\beta_1\) · Lecture 3 · p.19–23 \[\begin{aligned} \mathrm{Var}[\hat\beta_1]&=\sigma^2\sum_{i=1}^n w_i^2\\ &=\sigma^2\sum_{i=1}^n\left[\frac{(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\right]^2\\ &=\sigma^2\sum_{i=1}^n\frac{(x_i-\bar x)^2}{\left[\sum_{j=1}^n(x_j-\bar x)^2\right]^2}\\ &=\sigma^2\frac{\sum_{i=1}^n(x_i-\bar x)^2}{\left[\sum_{j=1}^n(x_j-\bar x)^2\right]^2}\\ &=\sigma^2\frac{1}{\sum_{j=1}^n(x_j-\bar x)^2}=\frac{\sigma^2}{S_{xx}} \end{aligned}\]

Slides p.23 writes \(\sum_{j=1}^n(x_i-\bar x)^2\). The index is \(j\), so the correct term is \(x_j\).

Larger \(\sigma^2\) (points farther from the line): larger variance.

Larger \(S_{xx}\) (wider spread of \(x\)): smaller variance.

Each term of \(S_{xx}\) is 0 or more. An added observation cannot make \(S_{xx}\) smaller.

How Added

Calculate \(\mathrm{Var}[\hat\beta_1]\)

Calculate \(\bar x=\frac1n\sum_{i=1}^n x_i\).
Calculate each \(x_j-\bar x\).
Square each value. Add the squares to get \(S_{xx}\).
Get \(\sigma^2\) from the question.
Calculate \(\mathrm{Var}[\hat\beta_1]=\sigma^2/S_{xx}\).
Take the square root to get the standard deviation \(\sigma/\sqrt{S_{xx}}\).

Self-check: With \(w_i=(x_i-\bar x)/S_{xx}\), \(\sum_{i=1}^n w_i^2=1/S_{xx}\).

Example · Six head sizes Added

Mean of \(x\) Step 1

\[\bar x=\frac{4512+3738+4261+3777+4177+3585}{6}=\frac{24050}{6}=4008.3333\]

Distances from \(\bar x\) Step 2

\[503.6667,\quad -270.3333,\quad 252.6667,\quad -231.3333,\quad 168.6667,\quad -423.3333\]

\(x_1-\bar x=4512-4008.3333=503.6667\).

Squares and their sum Step 3

\[253680.1111,\quad 73080.1111,\quad 63840.4444,\quad 53515.1111,\quad 28448.4444,\quad 179211.1111\]

\[S_{xx}=253680.1111+73080.1111+63840.4444+53515.1111+28448.4444+179211.1111=651775.3333\]

Rounded squares give 651775.3332; exact is 651775.3333.

Weights and the self-check Self-check

\[w_1=\frac{503.6667}{651775.3333}=7.7276\times10^{-4},\qquad w_1^2=5.9716\times10^{-7}\]

\[w_2^2=1.7203\times10^{-7},\ w_3^2=1.5028\times10^{-7},\ w_4^2=1.2597\times10^{-7},\ w_5^2=6.6967\times10^{-8},\ w_6^2=4.2186\times10^{-7}\]

\[\sum_{i=1}^6 w_i^2=1.5343\times10^{-6},\qquad \frac{1}{S_{xx}}=\frac{1}{651775.3333}=1.5343\times10^{-6}\]

Self-check: The two values are equal.

Variance Steps 4–5

\[\mathrm{Var}[\hat\beta_1]=\frac{\sigma^2}{S_{xx}}=\frac{5234.5225}{651775.3333}=0.0080312\]

Standard deviation Step 6

\[\frac{\sigma}{\sqrt{S_{xx}}}=\sqrt{0.0080312}=0.089617\]

The standard deviation is 0.089617 g/cm³. Unit 02 got the same variance, 0.00803, from \(\sigma^2\sum w_i^2\). ▲

With all 236 observations, Slides p.45 prints 0.01307. Slides p.45

The full data have a much larger \(S_{xx}\), so the value is much smaller.

Slides p.45 uses \(\hat\sigma=72.35\) for \(\sigma\). Unit 04 calls this the standard error, \(\mathrm{SE}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}\).

Why Slides p.19–23

The \(y_i\) are independent, so \(\mathrm{Var}[\sum w_iy_i]=\sum w_i^2\mathrm{Var}[y_i]=\sigma^2\sum w_i^2\).

\[\mathrm{Var}[\hat\beta_1]=\sigma^2\sum_{i=1}^n w_i^2\]

Slides p.19; Unit 02.

\[=\sigma^2\sum_{i=1}^n\left[\frac{(x_i-\bar x)}{\sum_{j=1}^n(x_j-\bar x)^2}\right]^2\]

Put in \(w_i\). Slides p.20.

\[=\sigma^2\sum_{i=1}^n\frac{(x_i-\bar x)^2}{\left[\sum_{j=1}^n(x_j-\bar x)^2\right]^2}\]

Square top and bottom. Slides p.21.

\[=\sigma^2\frac{\sum_{i=1}^n(x_i-\bar x)^2}{\left[\sum_{j=1}^n(x_j-\bar x)^2\right]^2}\]

Denominator has no \(i\). Slides p.22.

\[=\sigma^2\frac{\sum_{j=1}^n(x_j-\bar x)^2}{\left[\sum_{j=1}^n(x_j-\bar x)^2\right]^2}\]

Rename the index \(i\) to \(j\).

\[=\sigma^2\frac{1}{\sum_{j=1}^n(x_j-\bar x)^2}\]

Cancel one factor. Slides p.23.

\[=\frac{\sigma^2}{S_{xx}}\]

Definition of \(S_{xx}\). ∎

03.2Distribution of \(\hat\beta_1\) Slides p.24

What Slides p.24

Distribution of \(\hat\beta_1\) · Lecture 3 · p.24 So the distribution of \(\hat\beta_1\) is: \[\hat\beta_1\sim N\Big(\beta_1,\ \frac{\sigma^2}{S_{xx}}\Big)\]

\(\sim\) means "has the distribution".

The second item in \(N(\cdot,\cdot)\) is the variance, not the standard deviation.

How Added

Write the distribution of \(\hat\beta_1\)

Write \(\beta_1\) as the mean, because \(E[\hat\beta_1]=\beta_1\).
Calculate \(S_{xx}\) (Section 03.1).
Write \(\sigma^2/S_{xx}\) as the variance.
Write \(N\) as the family (Slides p.12–13).

Self-check: \(\hat\beta_1\) is in g/cm³, so the variance is in (g/cm³)².

Example · Six head sizes Added

\[\hat\beta_1\sim N\Big(\beta_1,\ \frac{5234.5225}{651775.3333}\Big)=N\big(\beta_1,\ 0.0080312\big)\]

The standard deviation is \(\sqrt{0.0080312}=0.089617\). 95% of the values of \(\hat\beta_1\) are within \(1.96\times0.089617=0.175649\) of \(\beta_1\). ▲

Figure 3-1 uses the numbers of Slides p.45: \(\hat\beta_1=0.26082\), estimated standard deviation 0.01307. Added · Slides p.45

Tick marks are \(0.26082+k\times0.01307\), \(k=-3,\dots,3\). For example, \(0.26082-3\times0.01307=0.26082-0.03921=0.22161\).

Figure 3-1 (Added): estimated sampling distribution \(N(0.26082,\ 0.01307^2)\) of \(\hat\beta_1\), 236 brainhead observations (Slides p.45). Orange: center. Green: ±1 standard deviation. Almost all area is between 0.22 and 0.30, far from 0.

Why Added

\[\hat\beta_1=\sum_{i=1}^n w_iy_i,\qquad y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\]

Slides p.12.

\[\hat\beta_1\sim N\Big(\sum_{i=1}^n w_i(\beta_0+\beta_1x_i),\ \sigma^2\sum_{i=1}^n w_i^2\Big)\]

Linear combination of independent Normals. Slides p.13.

\[\hat\beta_1\sim N\Big(\beta_1,\ \sigma^2\sum_{i=1}^n w_i^2\Big)\]

\(\sum w_i(\beta_0+\beta_1x_i)=\beta_1\). Slides p.18.

\[\hat\beta_1\sim N\Big(\beta_1,\ \frac{\sigma^2}{S_{xx}}\Big)\]

Section 03.1. ∎

03.3Distribution of \(\hat\beta_0\) Slides p.24

What Slides p.24

Distribution of \(\hat\beta_0\) · Lecture 3 · p.24 Question: what about \(\hat\beta_0\)? \[\hat\beta_0\sim N\Big(\beta_0,\ \sigma^2\Big(\frac1n+\frac{\bar x^2}{S_{xx}}\Big)\Big)\]

\(\hat\beta_0\) is Normal and unbiased.

\(\frac1n\) decreases when \(n\) increases.

\(\frac{\bar x^2}{S_{xx}}\) increases when \(\bar x\) is farther from 0. \(\beta_0\) is the mean of \(y\) at \(x=0\).

How Added

Calculate \(\mathrm{Var}[\hat\beta_0]\)

Calculate \(n\), \(\bar x\), and \(S_{xx}\) (Section 03.1).
Calculate \(\frac1n\).
Calculate \(\bar x^2\), then \(\frac{\bar x^2}{S_{xx}}\).
Add the two terms.
Multiply the sum by \(\sigma^2\) to get the variance.
Take the square root to get the standard deviation.

Self-check: The sum is at least \(\frac1n\), because \(\frac{\bar x^2}{S_{xx}}\geq0\).

Example · Six head sizes Added

Basic values Step 1

\[n=6,\qquad \bar x=4008.3333,\qquad S_{xx}=651775.3333\]

First term Step 2

\[\frac1n=\frac16=0.166667\]

Second term Step 3

\[\bar x^2=4008.3333^2=16066736.1111\]

\[\frac{\bar x^2}{S_{xx}}=\frac{16066736.1111}{651775.3333}=24.650727\]

Sum Step 4

\[\frac1n+\frac{\bar x^2}{S_{xx}}=0.166667+24.650727=24.817394\]

Self-check: 24.817394 > 0.166667.

Multiply by \(\sigma^2\) Step 5

\[\mathrm{Var}[\hat\beta_0]=5234.5225\times24.817394=129907.21\]

Square root Step 6

\[\sigma\sqrt{\frac1n+\frac{\bar x^2}{S_{xx}}}=\sqrt{129907.21}=360.43\]

\(\bar x\) is far from 0, so the second term is almost all of the sum. The standard deviation of \(\hat\beta_0\) is 360.43 g. ▲

For all 236 observations, Slides p.45 prints 47.77644 (with \(\hat\sigma\) for \(\sigma\)): the standard error of \(\hat\beta_0\). Slides p.45

Why Added

The slides give no derivation. \(\hat\beta_0=\bar y-\hat\beta_1\bar x\) is also a linear combination of the \(y_i\). It is Normal with mean \(\beta_0\), as for \(\hat\beta_1\).

Optional — not on the slides

Outline only. For the exam, use the slide result.

Covariance \(\mathrm{Cov}[U,V]=E[(U-E[U])(V-E[V])]\) measures how two random variables change together.

\(\mathrm{Var}[U-V]=\mathrm{Var}[U]+\mathrm{Var}[V]-2\mathrm{Cov}[U,V]\).

\(\hat\beta_0=\bar y-\bar x\,\hat\beta_1\) gives \(\mathrm{Var}[\hat\beta_0]=\mathrm{Var}[\bar y]+\bar x^2\mathrm{Var}[\hat\beta_1]-2\bar x\,\mathrm{Cov}[\bar y,\hat\beta_1]\).

\(\mathrm{Var}[\bar y]=\sigma^2/n\), and \(\mathrm{Var}[\hat\beta_1]=\sigma^2/S_{xx}\).

\(\mathrm{Cov}[\bar y,\hat\beta_1]=\sum_{i=1}^n\frac1n w_i\sigma^2=\frac{\sigma^2}{n}\sum_{i=1}^n w_i=0\), because \(\sum_{i=1}^n(x_i-\bar x)=0\).

Together: \(\mathrm{Var}[\hat\beta_0]=\sigma^2\big(\frac1n+\frac{\bar x^2}{S_{xx}}\big)\). ∎

03.4Practice Added

Q1. Use \(y_i\overset{indep}{\sim}N(\beta_0+\beta_1x_i,\sigma^2)\), \(\hat\beta_1=\sum_{i=1}^n w_iy_i\), \(w_i=\frac{x_i-\bar x}{\sum_{j=1}^n(x_j-\bar x)^2}\). Show \(\mathrm{Var}[\hat\beta_1]=\sigma^2/S_{xx}\). Where does the derivation use independence?

Answer

\(\mathrm{Var}[\hat\beta_1]=\mathrm{Var}[\sum w_iy_i]=\sum\mathrm{Var}[w_iy_i]\). This step uses independence.

\(=\sum w_i^2\mathrm{Var}[y_i]=\sigma^2\sum w_i^2\).

\(=\sigma^2\sum_i\frac{(x_i-\bar x)^2}{[\sum_j(x_j-\bar x)^2]^2}=\sigma^2\frac{\sum_i(x_i-\bar x)^2}{[\sum_j(x_j-\bar x)^2]^2}=\sigma^2\frac{1}{\sum_j(x_j-\bar x)^2}=\frac{\sigma^2}{S_{xx}}\).

Q2. For the 6 observations, \(S_{xx}=651775.3333\) and \(\sigma=72.35\). A second study measures the same 6 head sizes, so each \(x_i\) occurs two times and \(n=12\). (a) Calculate the new \(S_{xx}\). (b) Calculate the new standard deviation of \(\hat\beta_1\).

Answer

(a) \(\bar x\) stays 4008.3333. Each square occurs two times: \(S_{xx}=2\times651775.3333=1303550.6667\).

(b) \(\sigma^2/S_{xx}=5234.5225/1303550.6667=0.0040156\). \(\sqrt{0.0040156}=0.063369\).

This is the old value divided by \(\sqrt2\): \(0.089617/1.41421=0.063369\).

Q3. For the 6 observations, \(n=6\), \(\bar x=4008.3333\), \(S_{xx}=651775.3333\). (a) Calculate \(\frac1n+\frac{\bar x^2}{S_{xx}}\). (b) Which term gives most of \(\mathrm{Var}[\hat\beta_0]\)? Why?

Answer

(a) \(\frac1n=0.166667\). \(\frac{\bar x^2}{S_{xx}}=16066736.1111/651775.3333=24.650727\). Sum: \(0.166667+24.650727=24.817394\).

(b) The second term: \(24.650727/24.817394=0.99328\), about 99%. The mean head size 4008.3333 is far from \(x=0\), where the intercept is.

Unit 04Confidence interval for \(\beta_1\): known \(\sigma\), unknown \(\sigma\) (t distribution), and standard error

Plan · Slides p.25–35

Step 1 (Slides p.25–26): Standardize \(\hat\beta_1\) to get \(Z\sim N(0,1)\).

Step 2 (Slides p.27–30): Known \(\sigma\): solve \(P(-1.96\le Z\le1.96)=0.95\) for \(\beta_1\).

Step 3 (Slides p.31–33): Unknown \(\sigma\): use \(\hat\sigma\). The new quantity has distribution \(t_{(n-2)}\).

Step 4 (Slides p.34): Replace 1.96 with the t quantile \(q=t_{1-\alpha/2,n-2}\).

Step 5 (Slides p.35): Define \(\mathrm{SD}(\hat\beta_1)\) and \(\mathrm{SE}(\hat\beta_1)\). Write the interval as \(\hat\beta_1\pm t_{1-\alpha/2,n-2}\mathrm{SE}(\hat\beta_1)\).

Unit 05 calculates the brainhead interval (Slides p.36–37).

Start point (results of Units 01–03) · Added

Model (Slides p.3): \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), with \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\), \(i=1,\dots,n\).

Unit 03 (Slides p.24): \(\hat\beta_1\sim N\!\left(\beta_1,\dfrac{\sigma^2}{S_{xx}}\right)\), with \(S_{xx}=\sum_{i=1}^n(x_i-\bar x)^2\).

Slides p.8: \(\hat\sigma^2=\dfrac{1}{n-2}\sum_{i=1}^ne_i^2\), with \(e_i=y_i-\hat y_i\) and \(\hat y_i=\hat\beta_0+\hat\beta_1x_i\).

\(N(0,1)\) is the standard Normal distribution: mean 0, variance 1.

Brainhead data: Lecture 2 · p.38 gives \(\hat\beta_1=0.2608207\). Slides p.45 give \(\hat\sigma=72.35\) on 234 degrees of freedom (\(n=236\)) and \(\mathrm{SE}(\hat\beta_1)=0.01307\).

Four-point data set (Added): \((x_i,y_i)\) = (1, 2), (2, 3), (3, 5), (4, 6), \(n=4\). \(\bar x=10/4=2.5\), \(\bar y=16/4=4\). Deviations: \(x_i-\bar x=-1.5,-0.5,0.5,1.5\) and \(y_i-\bar y=-2,-1,1,2\). \(S_{xy}=3+0.5+0.5+3=7\), \(S_{xx}=2.25+0.25+0.25+2.25=5\). Estimates: \(\hat\beta_1=7/5=1.4\), \(\hat\beta_0=4-1.4\times2.5=0.5\).

04.1Standardize \(\hat\beta_1\) Slides p.25–26

Slides p.25 is the section page "Confidence Interval for \(\beta_1\)". It has no formulas.

What Slides p.26

Confidence interval for β1 · Lecture 3 · p.26 \[\hat\beta_1\sim N\!\left(\beta_1,\frac{\sigma^2}{S_{xx}}\right)\quad\Longrightarrow\quad\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}\sim N(0,1)\]

Confidence interval (CI): an interval from the data that contains the unknown parameter with a given probability, for example 95%.

Standardize: subtract the mean, then divide by the SD. The result has mean 0 and SD 1.

The SD of \(\hat\beta_1\) is \(\sqrt{\sigma^2/S_{xx}}=\sigma/\sqrt{S_{xx}}\). Call the result \(Z=\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}\).

The distribution of \(\hat\beta_1\) contains the unknown \(\beta_1\) and \(\sigma\). \(N(0,1)\) contains no unknown values. Its known probabilities give statements about \(\beta_1\).

How Added

Write \(Z\) for a data set

Mean. The mean of \(\hat\beta_1\) is \(\beta_1\) (Unit 02).
SD. Take the square root of \(\sigma^2/S_{xx}\): \(\sigma/\sqrt{S_{xx}}\).
\(\sqrt{S_{xx}}\). ① Calculate \(\bar x\). ② Calculate each \((x_i-\bar x)^2\). ③ Add them. ④ Take the square root.
Write \(Z\). Numerator \(\hat\beta_1-\beta_1\). Denominator \(\sigma/\sqrt{S_{xx}}\).

Self-check: the denominator contains \(\sqrt{S_{xx}}\), not \(S_{xx}\). It is an SD, not a variance.

Example · Four-point data set Added

Mean of \(x\) How step 3 ①

\[\bar x=\frac{1+2+3+4}{4}=\frac{10}{4}=2.5\]

Squared distances How step 3 ②

\[(1-2.5)^2=2.25,\quad(2-2.5)^2=0.25,\quad(3-2.5)^2=0.25,\quad(4-2.5)^2=2.25\]

\(S_{xx}\) and its square root How step 3 ③④

\[S_{xx}=2.25+0.25+0.25+2.25=5,\qquad\sqrt{S_{xx}}=\sqrt5=2.236068\]

SD of \(\hat\beta_1\) How step 2

\[\frac{\sigma}{\sqrt{S_{xx}}}=\frac{\sigma}{2.236068}=0.447214\,\sigma\]

Write \(Z\) How step 4

\[Z=\frac{1.4-\beta_1}{0.447214\,\sigma}\sim N(0,1)\]

Section 04.3 removes \(\sigma\). ▲

Why Added

A linear transformation multiplies by a constant and adds a constant. A linear transformation of a Normal variable is Normal.

\[Z=\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}=\frac{\sqrt{S_{xx}}}{\sigma}\hat\beta_1-\frac{\sqrt{S_{xx}}}{\sigma}\beta_1\]

Linear transformation: \(Z=a\hat\beta_1+b\), \(a=\sqrt{S_{xx}}/\sigma\), \(b=-\sqrt{S_{xx}}\beta_1/\sigma\). \(Z\) is Normal.

\[E[Z]=\frac{\sqrt{S_{xx}}}{\sigma}E[\hat\beta_1]-\frac{\sqrt{S_{xx}}}{\sigma}\beta_1\]

\(E[aW+b]=aE[W]+b\).

\[=\frac{\sqrt{S_{xx}}}{\sigma}\beta_1-\frac{\sqrt{S_{xx}}}{\sigma}\beta_1=0\]

\(E[\hat\beta_1]=\beta_1\).

\[\mathrm{Var}[Z]=\left(\frac{\sqrt{S_{xx}}}{\sigma}\right)^2\mathrm{Var}[\hat\beta_1]\]

\(\mathrm{Var}[aW+b]=a^2\mathrm{Var}[W]\).

\[=\frac{S_{xx}}{\sigma^2}\cdot\frac{\sigma^2}{S_{xx}}\]

\(\mathrm{Var}[\hat\beta_1]=\sigma^2/S_{xx}\).

\[=1\quad\Longrightarrow\quad Z\sim N(0,1)\]

∎

04.295% CI for \(\beta_1\) with known \(\sigma\) Slides p.27–30

What Slides p.27–30

95% CI for β1 — Known σ · Lecture 3 · p.27–30 Suppose \(\sigma\) is known. \[0.95=P(-1.96\le Z\le1.96)\] \[0.95=P\!\left(\hat\beta_1-1.96\frac{\sigma}{\sqrt{S_{xx}}}\le\beta_1\le\hat\beta_1+1.96\frac{\sigma}{\sqrt{S_{xx}}}\right)\] So a 95% CI for \(\beta_1\) is \(\hat\beta_1\pm1.96\dfrac{\sigma}{\sqrt{S_{xx}}}\)
I.e. the random interval \(\hat\beta_1\pm1.96\dfrac{\sigma}{\sqrt{S_{xx}}}\) will cover the true \(\beta_1\) 95% of the time.

\(P(A)\): the probability of event \(A\), a number from 0 to 1.

Quantile: for a random variable \(W\) and a probability \(p\), the number \(q\) with \(P(W\le q)=p\).

1.96 is the 0.975 quantile of \(N(0,1)\): \(P(Z\le1.96)=0.975\). The exact value is 1.959964.

\(a\pm b\): the interval from \(a-b\) to \(a+b\).

Random interval: its end points come from \(\hat\beta_1\) and change with the data. \(\beta_1\) is fixed.

\(N(0,1)\) is symmetric about 0. Each tail outside \(\pm1.96\) has area 0.025. "95%" is a property of the method: over many samples, about 95% of the intervals contain \(\beta_1\).

How Added

95% CI with known σ

Quantile. Use 1.96.
SD. Divide \(\sigma\) by \(\sqrt{S_{xx}}\).
Half-width. Multiply: \(1.96\times\sigma/\sqrt{S_{xx}}\).
Interval. \(\hat\beta_1-\) half-width to \(\hat\beta_1+\) half-width.

Self-check: the center is \(\hat\beta_1\). The area of \(N(0,1)\) between \(-1.96\) and 1.96 is 0.950004.

Example · Four-point data set with a supposed \(\sigma\) Added

In practice, \(\sigma\) is unknown. For this example only, suppose \(\sigma=0.3\).

Quantile How step 1

\[q=1.96\]

SD of \(\hat\beta_1\) How step 2

\[\frac{\sigma}{\sqrt{S_{xx}}}=\frac{0.3}{2.236068}=0.134164\]

\(\sqrt{S_{xx}}\) from Section 04.1.

Half-width How step 3

\[1.96\times0.134164=0.262962\]

Interval How step 4

\[1.4-0.262962=1.137038,\qquad1.4+0.262962=1.662962\]

\[1.4\pm0.262962=(1.137038,\ 1.662962)\]

▲

Unit 05 shows the meaning of "95% of the time" with the figure on Slides p.39.

Why Slides p.27–30

Each step does the same operation on all three parts. The probability stays 0.95.

\[0.95=P(-1.96\le Z\le1.96)\]

\(P(Z\le1.96)=0.975\). \(N(0,1)\) is symmetric.

\[0.95=P\!\left(-1.96\le\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}\le1.96\right)\]

Definition of \(Z\).

\[0.95=P\!\left(-1.96\frac{\sigma}{\sqrt{S_{xx}}}\le\hat\beta_1-\beta_1\le1.96\frac{\sigma}{\sqrt{S_{xx}}}\right)\]

Multiply by \(\sigma/\sqrt{S_{xx}}>0\). The signs stay.

\[0.95=P\!\left(-1.96\frac{\sigma}{\sqrt{S_{xx}}}\le\beta_1-\hat\beta_1\le1.96\frac{\sigma}{\sqrt{S_{xx}}}\right)\]

Multiply by \(-1\). The signs reverse. The bounds are symmetric.

\[0.95=P\!\left(\hat\beta_1-1.96\frac{\sigma}{\sqrt{S_{xx}}}\le\beta_1\le\hat\beta_1+1.96\frac{\sigma}{\sqrt{S_{xx}}}\right)\]

Last line of Slides p.30. ∎

04.3Unknown \(\sigma\): use \(\hat\sigma\), get the t distribution Slides p.31–33

What Slides p.31–33

95% CI for β1 — Unknown σ · Lecture 3 · p.31–33 In practice, \(\sigma\) is unknown and must be estimated. We have:
1. \(Z=\dfrac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}\sim N(0,1)\)
2. \(V=\dfrac{1}{\sigma^2}\sum_{i=1}^ne_i^2\sim\chi^2_{(n-2)}\)
3. It can be shown that \(Z\) and \(V\) are independent
Note that for independent \(Z\sim N(0,1)\) and \(V\sim\chi^2_\nu\), \(\Longrightarrow\dfrac{Z}{\sqrt{V/\nu}}\sim t_\nu\) \[\Longrightarrow\frac{Z}{\sqrt{V/\nu}}=\frac{\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\sqrt{\frac{1}{\sigma^2}\sum_{i=1}^ne_i^2/(n-2)}}\sim t_{(n-2)}\] \[\frac{\frac{\hat\beta_1-\beta_1}{1/\sqrt{S_{xx}}}}{\sqrt{\sum_{i=1}^ne_i^2/(n-2)}}\sim t_{(n-2)}\] \[\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\]

Degrees of freedom (df) \(\nu\): the parameter that sets the shape of a \(\chi^2\) or t distribution. For SLR, \(\nu=n-2\).

t distribution \(t_\nu\): the distribution of \(Z/\sqrt{V/\nu}\), with independent \(Z\sim N(0,1)\) and \(V\sim\chi^2_\nu\).

\(t_\nu\) is bell-shaped and symmetric about 0, with thicker tails than \(N(0,1)\). When \(\nu\) increases, it comes nearer to \(N(0,1)\).

The slides do not prove item 3.

\(Z\) contains the unknown \(\sigma\). With \(\hat\sigma\) for \(\sigma\), the denominator is random, and the distribution changes to \(t_{(n-2)}\). This distribution has no unknown values.

How Added

Calculate \(\hat\sigma/\sqrt{S_{xx}}\) and the degrees of freedom

Fit. Calculate \(\hat\beta_0\), \(\hat\beta_1\), and each \(\hat y_i\).
Residual sum of squares. ① Calculate \(e_i=y_i-\hat y_i\). ② Square each. ③ Add them.
df. \(\nu=n-2\).
\(\hat\sigma\). \(\hat\sigma^2=\sum e_i^2/(n-2)\). Take the square root.
Denominator. Divide \(\hat\sigma\) by \(\sqrt{S_{xx}}\). Then \((\hat\beta_1-\beta_1)/(\hat\sigma/\sqrt{S_{xx}})\sim t_{(n-2)}\).

Self-check: divide by \(n-2\), not \(n\). The residuals add to 0.

Example · Four-point data set Added

Fitted values How step 1

\[\hat y_i=0.5+1.4x_i:\quad0.5+1.4=1.9,\quad0.5+2.8=3.3,\quad0.5+4.2=4.7,\quad0.5+5.6=6.1\]

Residuals How step 2 ①

\[e_1=2-1.9=0.1,\quad e_2=3-3.3=-0.3,\quad e_3=5-4.7=0.3,\quad e_4=6-6.1=-0.1\]

Self-check: \(0.1-0.3+0.3-0.1=0\).

Residual sum of squares How step 2 ②③

\[\sum_{i=1}^4e_i^2=0.01+0.09+0.09+0.01=0.2\]

df How step 3

\[\nu=n-2=4-2=2\]

\(\hat\sigma\) How step 4

\[\hat\sigma^2=\frac{0.2}{2}=0.1,\qquad\hat\sigma=\sqrt{0.1}=0.316228\]

t quantity How step 5

\[\frac{\hat\sigma}{\sqrt{S_{xx}}}=\frac{0.316228}{2.236068}=0.141421\]

\[\frac{1.4-\beta_1}{0.141421}\sim t_{(2)}\]

\(\beta_1\) is the only unknown. ▲

Brainhead data: \(\hat\sigma=72.35\) on 234 df (Slides p.45). The t quantity has distribution \(t_{(234)}\).

Why Slides p.31–33

\[\frac{Z}{\sqrt{V/(n-2)}}\sim t_{(n-2)}\]

Definition of t with items 1–3 of the slide.

\[\frac{Z}{\sqrt{V/(n-2)}}=\frac{\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\sqrt{\frac{1}{\sigma^2}\sum_{i=1}^ne_i^2/(n-2)}}\]

Definitions of \(Z\) and \(V\).

\[=\frac{\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\frac{1}{\sigma}\sqrt{\sum_{i=1}^ne_i^2/(n-2)}}\]

\(\sqrt{\frac{1}{\sigma^2}A}=\frac{1}{\sigma}\sqrt{A}\), because \(\sigma>0\).

\[=\frac{\sigma\cdot\frac{\hat\beta_1-\beta_1}{\sigma/\sqrt{S_{xx}}}}{\sqrt{\sum_{i=1}^ne_i^2/(n-2)}}\]

\[=\frac{\frac{\hat\beta_1-\beta_1}{1/\sqrt{S_{xx}}}}{\sqrt{\sum_{i=1}^ne_i^2/(n-2)}}\]

\(\sigma\) cancels. Second line of Slides p.32.

\[=\frac{\frac{\hat\beta_1-\beta_1}{1/\sqrt{S_{xx}}}}{\hat\sigma}\]

\(\hat\sigma=\sqrt{\sum_{i=1}^ne_i^2/(n-2)}\) (Slides p.8).

\[=\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\]

Last line of Slides p.33. ∎

04.4Interval with a t quantile \(q=t_{1-\alpha/2,n-2}\) Slides p.34

The steps are those of Section 04.2, with \(q\) for 1.96 and \(\hat\sigma\) for \(\sigma\).

What Slides p.34

95% CI for β1 — Unknown σ · Lecture 3 · p.34 So we can find some quantile \(q\) such that \[0.95=P\!\left(-q\le\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\le q\right)\] \[0.95=P\!\left(\hat\beta_1-q\frac{\hat\sigma}{\sqrt{S_{xx}}}\le\beta_1\le\hat\beta_1+q\frac{\hat\sigma}{\sqrt{S_{xx}}}\right)\] and construct a 95% CI for \(\beta_1\) is \(\hat\beta_1\pm q\dfrac{\hat\sigma}{\sqrt{S_{xx}}}\)
Often written \(t_{1-\alpha/2,n-2}\), as in: \(\hat\beta_1\pm t_{1-\alpha/2,n-2}\dfrac{\hat\sigma}{\sqrt{S_{xx}}}\)

\(\alpha\): the probability that the interval does not contain the true value. The confidence level is \(1-\alpha\). A 95% CI has \(\alpha=0.05\).

\(t_{1-\alpha/2,n-2}\): the \(1-\alpha/2\) quantile of \(t_{(n-2)}\). For a \(t_{(n-2)}\) variable \(T\), \(P(T\le t_{1-\alpha/2,n-2})=1-\alpha/2\).

For a 95% CI, \(\alpha/2=0.025\) and \(q=t_{0.975,n-2}\). Get it from a t table or software.

Each tail outside \(\pm q\) has area \(\alpha/2\). The t tails are thicker than the Normal tails. \(q\) is a little larger than 1.96, and the interval is a little wider.

How Added

Find \(t_{1-\alpha/2,n-2}\) and write the interval

\(\alpha\). \(\alpha=1-\) confidence level. For 95%, \(\alpha=0.05\).
df. df \(=n-2\).
\(q\). Find the \(1-\alpha/2\) quantile of \(t_{(n-2)}\) (table or software).
Interval. \(\hat\beta_1\pm q\,\hat\sigma/\sqrt{S_{xx}}\).

Self-check: ① For 95%, \(q\) is larger than 1.96. When df increases, \(q\) comes nearer to 1.96. ② \(P(T\le-q)=\alpha/2\).

Example 1 · Brainhead data: compare \(q\) with 1.96 Added

\(\alpha\) and df How steps 1–2

\[\alpha=1-0.95=0.05,\qquad\alpha/2=0.025,\qquad\text{df}=236-2=234\]

t quantile How step 3

\[q=t_{0.975,234}=1.970154\]

Compare with the Normal value Self-check ①

\[1.970154-1.959964=0.010190\]

Large df makes the difference small.

Tail area Self-check ②

\[P(T\le-1.970154)=0.025\]

▲

Values of \(t_{0.975,\nu}\): df = 2: \(q=4.302653\). df = 5: \(q=2.570582\). df = 10: \(q=2.228139\).

df = 30: \(q=2.042272\). df = 234: \(q=1.970154\). When df increases, \(q\) comes nearer to 1.959964.

Figure 4-1. Densities of \(N(0,1)\) (blue solid) and \(t_{(234)}\) (orange dashed). Their largest height difference is 0.00058. The green area between \(-q\) and \(q\) is 0.95. Each tail is 0.025. At this scale, the difference between 1.970 and 1.96 is too small to see.

Figure 4-2. Zoom from 1.85 to 2.10. The t line is above the Normal line, and the t tail has more area. To keep the right tail at 0.025, the boundary moves from 1.960 to 1.970.

Example 2 · Four-point data set: 95% CI with unknown σ Added

\(\hat\sigma/\sqrt{S_{xx}}=0.141421\) comes from Section 04.3.

\(\alpha\) and df How steps 1–2

\[\alpha=0.05,\qquad\alpha/2=0.025,\qquad\text{df}=4-2=2\]

t quantile How step 3

\[q=t_{0.975,2}=4.302653\]

Half-width How step 4

\[q\,\frac{\hat\sigma}{\sqrt{S_{xx}}}=4.302653\times0.1414214=0.608487\]

Interval How step 4

\[1.4-0.608487=0.791513,\qquad1.4+0.608487=2.008487\]

\[1.4\pm0.608487=(0.791513,\ 2.008487)\]

With 2 df, \(q\) is large and the interval wide. ▲

Why Slides p.34

\[0.95=P\!\left(-q\le\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\le q\right)\]

\(q=t_{0.975,n-2}\). \(t_{(n-2)}\) is symmetric about 0.

\[0.95=P\!\left(-q\frac{\hat\sigma}{\sqrt{S_{xx}}}\le\hat\beta_1-\beta_1\le q\frac{\hat\sigma}{\sqrt{S_{xx}}}\right)\]

Multiply by \(\hat\sigma/\sqrt{S_{xx}}>0\).

\[0.95=P\!\left(-q\frac{\hat\sigma}{\sqrt{S_{xx}}}\le\beta_1-\hat\beta_1\le q\frac{\hat\sigma}{\sqrt{S_{xx}}}\right)\]

Multiply by \(-1\). The bounds are symmetric.

\[0.95=P\!\left(\hat\beta_1-q\frac{\hat\sigma}{\sqrt{S_{xx}}}\le\beta_1\le\hat\beta_1+q\frac{\hat\sigma}{\sqrt{S_{xx}}}\right)\]

General level: \(1-\alpha\) for 0.95, \(t_{1-\alpha/2,n-2}\) for \(q\). ∎

04.5\(\mathrm{SD}(\hat\beta_1)\) and \(\mathrm{SE}(\hat\beta_1)\) Slides p.35

What Slides p.35

Standard Errors · Lecture 3 · p.35 Standard error of \(\hat\beta_1\) is the estimated standard deviation of \(\hat\beta_1\) \[\mathrm{SD}(\hat\beta_1)=\frac{\sigma}{\sqrt{S_{xx}}}\] \[\mathrm{SE}(\hat\beta_1)=\frac{\hat\sigma}{\sqrt{S_{xx}}}\] Can write a CI as \(\hat\beta_1\pm t_{1-\alpha/2,n-2}\mathrm{SE}(\hat\beta_1)\)

\(\mathrm{SD}(\hat\beta_1)\): the usual distance of \(\hat\beta_1\) from \(\beta_1\) over many samples. It contains the unknown \(\sigma\). We cannot calculate it.

Standard error \(\mathrm{SE}(\hat\beta_1)\): the estimate of \(\mathrm{SD}(\hat\beta_1)\), with \(\hat\sigma\) for \(\sigma\). We can calculate it.

The interval has the form "estimate ± t quantile × standard error".

How Added

Calculate \(\mathrm{SE}(\hat\beta_1)\)

\(\hat\sigma\). \(\hat\sigma=\sqrt{\sum e_i^2/(n-2)}\) (Section 04.3, How steps 2–4).
\(\sqrt{S_{xx}}\). Section 04.1, How step 3.
Divide. \(\mathrm{SE}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}\).

Self-check: SE has the unit of \(\hat\beta_1\). SE uses \(\hat\sigma\), not \(\sigma\).

Example · Four-point data set and brainhead data Added

\(\hat\sigma\), four-point data How step 1

\[\hat\sigma=\sqrt{\frac{0.2}{4-2}}=\sqrt{0.1}=0.316228\]

\(\sqrt{S_{xx}}\), four-point data How step 2

\[\sqrt{S_{xx}}=\sqrt5=2.236068\]

Divide How step 3

\[\mathrm{SE}(\hat\beta_1)=\frac{0.316228}{2.236068}=0.141421\]

Brainhead SE Slides p.45

\[\hat\sigma=72.35,\qquad\mathrm{SE}(\hat\beta_1)=0.01307\]

\[\sqrt{S_{xx}}=\frac{\hat\sigma}{\mathrm{SE}(\hat\beta_1)}=\frac{72.35}{0.01307}=5535.6\]

Approximate: the slide values are rounded.

Interval form Slides p.35

\[\hat\beta_1\pm t_{0.975,234}\mathrm{SE}(\hat\beta_1)=0.2608207\pm1.970154\times0.01307\]

Unit 05 completes this (Slides p.36–37). ▲

Over many samples, \(\hat\beta_1\) is usually about 0.013 from the true \(\beta_1\).

Why Added

\[\mathrm{Var}[\hat\beta_1]=\frac{\sigma^2}{S_{xx}}\]

Unit 03 (Slides p.23–24).

\[\mathrm{SD}(\hat\beta_1)=\sqrt{\mathrm{Var}[\hat\beta_1]}=\sqrt{\frac{\sigma^2}{S_{xx}}}\]

SD is the square root of the variance.

\[=\frac{\sigma}{\sqrt{S_{xx}}}\]

\[\mathrm{SE}(\hat\beta_1)=\frac{\hat\sigma}{\sqrt{S_{xx}}}\]

Slides p.35: \(\hat\sigma\) for \(\sigma\). ∎

04.6Practice Added

Q1. Brainhead data, \(n=236\). For a 90% CI and a 99% CI, give \(\alpha\), \(\alpha/2\), df, and the quantile \(t_{1-\alpha/2,n-2}\). Use \(t_{0.95,234}=1.651391\) and \(t_{0.995,234}=2.597002\). Compare them with the \(N(0,1)\) quantiles 1.644854 and 2.575829.

Answer

df \(=236-2=234\).

90%: \(\alpha=0.10\), \(\alpha/2=0.05\). \(t_{0.95,234}=1.651391\). Normal: 1.644854. Difference: \(1.651391-1.644854=0.006537\).

99%: \(\alpha=0.01\), \(\alpha/2=0.005\). \(t_{0.995,234}=2.597002\). Normal: 2.575829. Difference: \(2.597002-2.575829=0.021173\).

The t quantile is a little larger at both levels (thicker tails). A higher level gives a larger \(q\) and a wider interval.

Q2. \(\sigma\) is unknown. Start from \(1-\alpha=P\!\left(-t_{1-\alpha/2,n-2}\le\dfrac{\hat\beta_1-\beta_1}{\mathrm{SE}(\hat\beta_1)}\le t_{1-\alpha/2,n-2}\right)\). Derive the \(1-\alpha\) CI for \(\beta_1\).

Answer

Write \(t=t_{1-\alpha/2,n-2}\) and \(\mathrm{SE}=\mathrm{SE}(\hat\beta_1)\).

\[1-\alpha=P\!\left(-t\le\frac{\hat\beta_1-\beta_1}{\mathrm{SE}}\le t\right)\]

\[1-\alpha=P\!\left(-t\,\mathrm{SE}\le\hat\beta_1-\beta_1\le t\,\mathrm{SE}\right)\]

\(\mathrm{SE}>0\).

\[1-\alpha=P\!\left(-t\,\mathrm{SE}\le\beta_1-\hat\beta_1\le t\,\mathrm{SE}\right)\]

Multiply by \(-1\). The bounds are symmetric.

\[1-\alpha=P\!\left(\hat\beta_1-t\,\mathrm{SE}\le\beta_1\le\hat\beta_1+t\,\mathrm{SE}\right)\]

CI: \(\hat\beta_1\pm t_{1-\alpha/2,n-2}\mathrm{SE}(\hat\beta_1)\). ∎

Q3. Small data set of Unit 01: \((x_i,y_i)\) = (1, 2), (2, 4), (3, 5), (4, 4), (5, 5). Sections 01.4–01.5 give \(S_{xx}=10\), \(\hat\beta_1=0.6\), and \(\sum e_i^2=2.4\). Calculate \(\hat\sigma\), \(\mathrm{SE}(\hat\beta_1)\), df, and the 95% CI for \(\beta_1\). Use \(t_{0.975,3}=3.182446\).

Answer

\(n=5\). df \(=5-2=3\).

\(\hat\sigma^2=2.4/3=0.8\) and \(\hat\sigma=\sqrt{0.8}=0.894427\).

\(\mathrm{SE}(\hat\beta_1)=0.894427/\sqrt{10}=0.894427/3.162278=0.2828427\).

Half-width: \(3.182446\times0.2828427=0.900132\).

95% CI: \(0.6-0.900132=-0.300132\) and \(0.6+0.900132=1.500132\). The interval is \((-0.300132,\ 1.500132)\).

Unit 05Interval estimation example and interpretation

Plan · Slides p.36–39

Step 1: Calculate the two intervals (Slides p.36).

Step 2: Check the printed intervals (Slides p.37–38).

Step 3: Interpret "95%" (Slides p.39).

Step 4: Practice (Added).

Start point · Added

Brainhead data: \(n=236\), \(\hat\beta_1=0.2608207\), \(\hat\beta_0=335.4323150\) (Lecture 2 · p.38).

\(\hat\sigma=72.35\), with \(n-2=234\) (Slides p.45).

From brainhead.csv (Added): \(\bar x=3637.8644\), \(S_{xx}=30647233.66\), and \(\hat\sigma=72.35066\).

05.1Confidence intervals for \(\beta_1\) and \(\beta_0\) Slides p.36

What Slides p.36

Example: interval estimation · Lecture 3 · p.36 (results only) 95% CI for \(\beta_1\): [0.235, 0.287]
95% CI for \(\beta_0\): [241.305, 429.559]

The \(\beta_0\) interval uses \(\hat\beta_0\sim N\big(\beta_0,\ \sigma^2(1/n+\bar x^2/S_{xx})\big)\) (Slides p.24), with \(\hat\sigma\) for \(\sigma\).

\(\beta_1\): \(\hat\beta_1\pm t_{1-\alpha/2,\,n-2}\,\mathrm{SE}(\hat\beta_1)\), with \(\mathrm{SE}(\hat\beta_1)=\dfrac{\hat\sigma}{\sqrt{S_{xx}}}\)

\(\beta_0\): \(\hat\beta_0\pm t_{1-\alpha/2,\,n-2}\,\mathrm{SE}(\hat\beta_0)\), with \(\mathrm{SE}(\hat\beta_0)=\hat\sigma\sqrt{\dfrac{1}{n}+\dfrac{\bar x^2}{S_{xx}}}\)

How Added

Calculate the CI for \(\beta_1\) or \(\beta_0\)

Find \(t_{1-\alpha/2,\,n-2}\).
Calculate the standard error.
Multiply the quantile by the standard error to get the half-width.
Subtract the half-width from the estimate to get the lower limit.
Add the half-width to the estimate to get the upper limit.

Self-check: ① The midpoint equals the estimate. ② For \(n=236\), the 95% quantile is 1.970. ③ The standard errors agree with Slides p.45: 0.01307 and 47.77644.

Example 1 · CI for \(\beta_1\) Added

Quantile How step 1

\[t_{1-\alpha/2,\,n-2}=t_{1-0.05/2,\,236-2}\]

\[=t_{0.975,\,234}\]

\[=1.970154\]

Standard error How step 2

\[\mathrm{SE}(\hat\beta_1)=\frac{\hat\sigma}{\sqrt{S_{xx}}}=\frac{72.35066}{\sqrt{30647233.66}}\]

\[=\frac{72.35066}{5535.994}\]

\[=0.01306914\]

Slides p.45: 0.01307.

Half-width How step 3

\[t_{0.975,\,234}\,\mathrm{SE}(\hat\beta_1)=1.970154\times0.01306914\]

\[=0.0257482\]

Limits How steps 4–5

\[\hat\beta_1\pm0.0257482=0.2608207\pm0.0257482\]

\[=(0.2608207-0.0257482,\ \ 0.2608207+0.0257482)\]

\[=(0.2350725,\ \ 0.2865689)\]

Rounded: [0.235, 0.287], as on Slides p.36.

Example 2 · CI for \(\beta_0\) Added

Quantile How step 1

\[t_{0.975,\,234}=1.970154\]

Same \(\alpha\) and \(n-2\) as Example 1.

Standard error How step 2

\[\mathrm{SE}(\hat\beta_0)=\hat\sigma\sqrt{\frac{1}{n}+\frac{\bar x^2}{S_{xx}}}=72.35066\sqrt{\frac{1}{236}+\frac{3637.8644^2}{30647233.66}}\]

\[=72.35066\sqrt{\frac{1}{236}+\frac{13234057.39}{30647233.66}}\]

\[=72.35066\sqrt{0.0042373+0.4318190}\]

\[=72.35066\sqrt{0.4360563}\]

\[=72.35066\times0.6603456\]

\[=47.77644\]

Slides p.45: 47.77644.

Half-width How step 3

\[t_{0.975,\,234}\,\mathrm{SE}(\hat\beta_0)=1.970154\times47.77644\]

\[=94.12694\]

Limits How steps 4–5

\[\hat\beta_0\pm94.12694=335.43232\pm94.12694\]

\[=(335.43232-94.12694,\ \ 335.43232+94.12694)\]

\[=(241.30538,\ \ 429.55926)\]

Rounded: [241.305, 429.559], as on Slides p.36.

\(\mathrm{SE}(\hat\beta_0)=47.78\) is much larger than \(\mathrm{SE}(\hat\beta_1)=0.0131\). The cause is \(\bar x^2/S_{xx}=0.4318\): \(\bar x=3637.8644\) is far from \(x=0\).

Why Added

The steps are the same as for \(\hat\beta_1\) on Slides p.33–34.

\[\hat\beta_0\sim N\Big(\beta_0,\ \sigma^2\Big(\frac1n+\frac{\bar x^2}{S_{xx}}\Big)\Big)\]

Slides p.24.

\[\frac{\hat\beta_0-\beta_0}{\sigma\sqrt{\frac1n+\frac{\bar x^2}{S_{xx}}}}\sim N(0,1)\]

Standardize.

\[\frac{\hat\beta_0-\beta_0}{\hat\sigma\sqrt{\frac1n+\frac{\bar x^2}{S_{xx}}}}\sim t_{(n-2)}\]

Replace \(\sigma\) with \(\hat\sigma\), as on Slides p.33.

\[0.95=P\Big(-t_{0.975,\,n-2}\le\frac{\hat\beta_0-\beta_0}{\mathrm{SE}(\hat\beta_0)}\le t_{0.975,\,n-2}\Big)\]

\[0.95=P\big(-t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_0)\le\hat\beta_0-\beta_0\le t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_0)\big)\]

\[0.95=P\big(-t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_0)\le\beta_0-\hat\beta_0\le t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_0)\big)\]

\[0.95=P\big(\hat\beta_0-t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_0)\le\beta_0\le\hat\beta_0+t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_0)\big)\]

∎

05.2The printed intervals Slides p.37–38

What Slides p.37–38

Example: interval estimation · Lecture 3 · p.37–38 (results only) 95% CI for \(\beta_0\): from 241.3053884 to 429.5592416
95% CI for \(\beta_1\): from 0.2350725 to 0.2865689
Does this mean \(P(0.235\le\beta_1\le0.287)=95\%\)? No!

Slides p.37 gives the intervals of Slides p.36 with more decimal places.

How Added

Check a printed interval

Calculate (lower + upper)/2. It must equal the estimate.
Calculate (upper − lower)/2 to get the half-width.
Divide the half-width by the standard error. The result must equal \(t_{0.975,\,n-2}\).

Self-check: For the brainhead data, step 3 gives 1.970154.

Example 3 · Check the printed intervals Added

\(\beta_1\), step 1: \((0.2350725+0.2865689)/2=0.2608207=\hat\beta_1\).

\(\beta_1\), step 2: \((0.2865689-0.2350725)/2=0.0257482\), as in Example 1.

\(\beta_1\), step 3: \(0.0257482/0.01306914=1.97015\).

\(\beta_0\), step 1: \((241.3053884+429.5592416)/2=335.432315=\hat\beta_0\).

\(\beta_0\), step 2: \((429.5592416-241.3053884)/2=94.1269266\). Example 2 gives 94.12694, a rounding difference.

\(\beta_0\), step 3: \(94.1269266/47.77644=1.97015\).

05.3Interpretation of the confidence interval Slides p.39

"95%" describes the method, not one calculated interval.

What Slides p.39

Interval interpretation · Lecture 3 · p.39 If we were to draw samples repeatedly and construct intervals in the same way, any given interval may or may not cover the truth. On average, 95% will cover the truth.

\(\beta_1\) is a fixed number. It has no probability distribution.

Before sampling, \(L=\hat\beta_1-t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_1)\) and \(U=\hat\beta_1+t_{0.975,\,n-2}\,\mathrm{SE}(\hat\beta_1)\) are random, because the \(y_i\) are random.

An interval covers \(\beta_1\) when \(L\le\beta_1\le U\).

"95%" means \(P(L\le\beta_1\le U)=0.95\).

After sampling, \(L=0.235\) and \(U=0.287\) are fixed. This interval covers \(\beta_1\) or it does not.

How Added

Interpret a confidence interval

State the parameter and its unit.
State the interval and the confidence level.
Say that 95% of intervals from repeated samples cover the parameter.

Self-check: ① "95%" describes the intervals, not \(\beta_1\). ② The limits agree with Slides p.36–37.

Example 4 · Interpretation for \(\beta_1\) Added

Interpretation · brainhead \(\beta_1\)

Step 1: \(\beta_1\) is the change in mean brain weight (g) for each 1 cm³ increase in head size.

Step 2: The 95% CI for \(\beta_1\) is [0.235, 0.287] g per cm³.

Step 3: Draw many samples of 236 persons. Make \(\hat\beta_1\pm t_{0.975,\,234}\,\mathrm{SE}(\hat\beta_1)\) from each. About 95% of these intervals cover the true \(\beta_1\).

Example 5 · Simulation of coverage Added

In real data, \(\beta_1\) is unknown. A simulation sets the true values and counts the intervals that cover \(\beta_1\).

Simulation settings

True values: \(\beta_0=2\), \(\beta_1=1\), \(\sigma=2\).

Fixed \(x\): \(1,2,\dots,10\). \(n=10\), \(\bar x=5.5\), \(S_{xx}=82.5\).

Each sample: \(y_i=2+1\cdot x_i+\epsilon_i\), with \(\epsilon_i\sim N(0,2^2)\).

Each interval: the \(\beta_1\) method of Section 05.1, with \(t_{0.975,\,8}=2.306004\).

\(S_{xx}\): the \(x_i-\bar x\) are \(\pm0.5,\pm1.5,\pm2.5,\pm3.5,\pm4.5\). Their squares add to \(2(0.25+2.25+6.25+12.25+20.25)=2\times41.25=82.5\).

Figure 5-1 · 95% CIs from 20 simulated samples (Added). Dot: \(\hat\beta_1\). Red line: true \(\beta_1=1.0\). 19 blue intervals cover it. Interval 1 (orange, [0.061, 0.941]) does not.

\(\beta_1=1\) is fixed. The red line does not move.

Each sample gives a different \(\hat\beta_1\) and \(\hat\sigma\). The center and width of each interval change.

Coverage: \(19/20=0.95\). For 10000 intervals: \(9497/10000=0.9497\).

Why Added

\[\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\]

Slides p.33. \(\hat\beta_1\), \(\hat\sigma\) are random.

\[0.95=P\Big(-t_{0.975,\,n-2}\le\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\le t_{0.975,\,n-2}\Big)\]

Definition of the quantile.

\[0.95=P\Big(\hat\beta_1-t_{0.975,\,n-2}\frac{\hat\sigma}{\sqrt{S_{xx}}}\le\beta_1\le\hat\beta_1+t_{0.975,\,n-2}\frac{\hat\sigma}{\sqrt{S_{xx}}}\Big)\]

Slides p.34.

\[0.95=P(L\le\beta_1\le U)\]

\(L\), \(U\) random; \(\beta_1\) constant.

\[L=0.235,\quad U=0.287\]

One sample: nothing random remains.

Line 4 is a property of the method. Line 5 is one result of it. ∎

05.4Practice Added

Brainhead data: \(n=236\), \(\hat\beta_0=335.4323150\), \(\hat\beta_1=0.2608207\), \(\mathrm{SE}(\hat\beta_1)=0.01306914\), \(\mathrm{SE}(\hat\beta_0)=47.77644\).

Q1. Calculate a 90% CI for \(\beta_1\). Use \(t_{0.95,\,234}=1.651391\).

Answer

\[t_{1-0.10/2,\,234}=t_{0.95,\,234}=1.651391\]

\[1.651391\times0.01306914=0.02158226\]

\[0.2608207\pm0.02158226=(0.2392384,\ 0.2824030)\]

It is narrower than the 95% CI [0.235, 0.287], because 1.651 < 1.970.

Q2. Calculate a 99% CI for \(\beta_0\). Use \(t_{0.995,\,234}=2.597002\).

Answer

\[2.597002\times47.77644=124.0755\]

\[335.4323\pm124.0755=(211.357,\ 459.508)\]

Q3. The 95% CI for \(\beta_1\) is [0.235, 0.287]. Which interpretation is correct?

(A) There is a 95% probability that \(\beta_1\) lies in [0.235, 0.287].

(B) Draw many samples of size 236 and make an interval the same way from each. About 95% of these intervals cover the true \(\beta_1\).

Answer

(B). \(\beta_1\) is fixed. \(L\) and \(U\) are random (Slides p.39).

Unit 06Hypothesis tests for \(\beta_1\): p-value, coefficient table, and one-sided tests

Plan · Slides p.40–49

Step 1: Hypotheses and test statistic (Slides p.40–41).

Step 2: p-value as a tail area of the t density (Slides p.42).

Step 3: Test \(H_0:\beta_1=0\) on the brainhead data (Slides p.43–44).

Step 4: Coefficient table (Slides p.45).

Step 5: p-value meaning and Type-I error rate (Slides p.46–47).

Step 6: One-sided tests (Slides p.48–49).

Step 7: Practice (Added).

Unit 01: \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), \(\epsilon_i\overset{iid}{\sim}N(0,\sigma^2)\); \(\hat\beta_1=S_{xy}/S_{xx}\); \(\hat\sigma=\sqrt{\sum_{i=1}^n e_i^2/(n-2)}\).

Unit 04: \(\mathrm{SE}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}\) and \(\dfrac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\) (Slides p.33–35).

Brainhead (Unit 05): \(n=236\), \(\hat\beta_1=0.2608207\), \(\hat\sigma=72.35066\), \(S_{xx}=30647233.66\).

06.1Test setup and test statistic Slides p.40–41

Slides p.40: section page, no formulas.

Hypothesis test: a method that measures the evidence in the data against a claim about a parameter.

Null hypothesis \(H_0\): the claim that we test, for example \(H_0:\beta_1=\theta_0\). The test assumes it is true.

\(\theta_0\): a given number. Usually \(\theta_0=0\): the mean of \(y\) has no linear relation with \(x\).

Alternative hypothesis \(H_1\): the claim against \(H_0\), for example \(H_1:\beta_1\ne\theta_0\).

Test statistic: a value from the data whose distribution under \(H_0\) is known.

What Slides p.40–41

Hypothesis Testing · Lecture 3 · p.41 Want to test a null hypothesis \(H_0:\beta_1=\theta_0\) vs. some alternative hypothesis \(H_1:\beta_1\ne\theta_0\).
• E.g., often want to test \(H_0:\beta_1=0\).
Goal is to characterize how much evidence we have against \(H_0\)
• How “extreme” are our data relative to \(H_0\)?
• I.e. If \(H_0\) really were true, would these data be really surprising?
Under \(H_0\): \[\frac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\]

The test does not prove \(H_0\) true or false. It asks: are the data rare if \(H_0\) is true?

The statistic is the distance from \(\hat\beta_1\) to \(\theta_0\) in standard errors.

\(T\) is the random statistic. \(t\) is its observed value (Slides p.42).

How Added

Calculate \(t\)

Write \(H_0:\beta_1=\theta_0\).
Calculate \(\hat\beta_1=S_{xy}/S_{xx}\).
Calculate \(\mathrm{SE}(\hat\beta_1)=\hat\sigma/\sqrt{S_{xx}}\).
Calculate \(t=\dfrac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\).
Write the reference distribution \(t_{(n-2)}\).

Self-check: \(t\) has the sign of \(\hat\beta_1-\theta_0\).

Example · Brainhead data Slides p.43

Hypotheses How step 1

\[H_0:\beta_1=0\quad\text{vs.}\quad H_1:\beta_1\ne0,\qquad \theta_0=0\]

Slope estimate How step 2

\[\hat\beta_1=0.2608207\]

Standard error How step 3

\[\mathrm{SE}(\hat\beta_1)=\frac{\hat\sigma}{\sqrt{S_{xx}}}=0.01306914\]

Section 05.1. Slides p.45 prints 0.01307.

Observed value How step 4

\[t=\frac{0.2608207-0}{0.01306914}=\frac{0.2608207}{0.01306914}=19.957\]

Reference distribution How step 5

\[n-2=236-2=234,\qquad T\sim t_{(234)}\ \text{under}\ H_0\]

Self-check: \(\hat\beta_1=0.26>0=\theta_0\), and \(t\) is positive.

\(\hat\beta_1\) is about 20 standard errors from 0. ▲

Why Slides p.41

\[\frac{\hat\beta_1-\beta_1}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\]

Unit 04 (Slides p.33–35), for each \(\beta_1\).

\[\beta_1=\theta_0\]

\(H_0\) is true.

\[\frac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\quad\text{under }H_0\]

Put line 2 into line 1.

∎ Under \(H_0\), this distribution has no unknown parameter.

06.2The p-value Slides p.42

p-value: the probability, under \(H_0\), of a test statistic as extreme as the observed value or more.

Extreme (two-sided test): a large \(|T|\), far from 0 in either direction.

Probability density function (PDF): a curve whose area over an interval is the probability of that interval.

What Slides p.42

Hypothesis Testing – p value · Lecture 3 · p.42 Under \(H_0\): \(\frac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\)
• What’s the probability under the null of a test statistic as extreme or more than what we observe? \[P(|T|\ge|t|)=2P(T\ge|t|)=2[1-P(T\le|t|)]\quad\text{(last term corrected)}\]

Correction: Slides p.42 prints the last term as \(2[1-P(T\le-|t|)]\). This is a typing error. The block uses the correct form.

Figure 6-1 is the plot that Slides p.42 asks for ("DRAW PLOT").

Figure 6-1. \(t_{(234)}\) density (blue). Orange: the tails \(T\ge|t|\) and \(T\le-|t|\), for the demonstration value \(|t|=2\). Each tail has area 0.02332811. Total (two-sided p-value): 0.04665622. The brainhead value 19.957 is far past the right edge.

How Added

Calculate the two-sided p-value

Calculate \(t\) (Section 06.1) and \(n-2\).
Take \(|t|\).
Find \(P(T\ge|t|)\) for \(T\sim t_{(n-2)}\) with a t table or software.
Multiply by 2. The result is \(P(|T|\ge|t|)\).

Self-check: \(2P(T\le-|t|)\) gives the same value.

Example · Demonstration value and brainhead data Added

Demonstration value: right tail How step 3

\[P(T\ge2)=0.02332811,\qquad T\sim t_{(234)}\]

Demonstration value: multiply by 2 How step 4

\[P(|T|\ge2)=2\times0.02332811=0.04665622\]

Brainhead: right tail How step 3

\[P(T\ge t)=1.015856\times10^{-52}\]

Unrounded \(t\) (brainhead.csv): 19.956998.

Brainhead: multiply by 2 How step 4

\[P(|T|\ge|t|)=2\times1.015856\times10^{-52}=2.031713\times10^{-52}\]

Self-check: Slides p.43 prints \(2.031713\times10^{-52}\).

\(|t|=2\) gives about 0.047. \(|t|=19.957\) gives about \(10^{-52}\). A larger \(|t|\) gives a smaller p-value. ▲

Why Slides p.42

\[P(|T|\ge|t|)=P(T\ge|t|)+P(T\le-|t|)\]

The two tails do not overlap.

\[P(T\le-|t|)=P(T\ge|t|)\]

Symmetry about 0.

\[P(|T|\ge|t|)=2P(T\ge|t|)\]

Put line 2 into line 1.

\[P(T\ge|t|)=1-P(T<|t|)\]

Complement.

\[P(T<|t|)=P(T\le|t|)\]

\(T\) is continuous: \(P(T=|t|)=0\).

\[P(|T|\ge|t|)=2[1-P(T\le|t|)]\]

Put lines 4 and 5 into line 3.

∎ Lines 2 and 3 also give \(P(|T|\ge|t|)=2P(T\le-|t|)\).

06.3Example: test \(H_0:\beta_1=0\) Slides p.43–44

Significance level \(\alpha\): the limit, set before the test, below which the p-value rejects \(H_0\). This course uses \(\alpha=0.05\).

Reject \(H_0\): the data give sufficient evidence against \(H_0\).

What Slides p.43–44

Example: Testing a Hypothesis · Lecture 3 · p.43–44 The pval < 0.05. Hence we reject the null hypothesis at the 5% level.
What if pval > 0.05? Would we accept the null? No! Simply would not have enough evidence to reject.

A p-value above 0.05 does not show that \(H_0\) is true.

How Added

Make a decision at level \(\alpha\)

Write \(H_0\), \(H_1\), and \(\alpha\) (default 0.05).
Calculate \(t\) and the p-value (Sections 06.1–06.2).
If p-value < \(\alpha\), write "reject \(H_0\) at the \(100\alpha\%\) level".
If p-value > \(\alpha\), write "not enough evidence to reject \(H_0\)".
Write the result in the question's words.

Self-check: Never write "accept \(H_0\)".

Example · Brainhead data Slides p.43

Hypotheses and level How step 1

\[H_0:\beta_1=0,\quad H_1:\beta_1\ne0,\quad \alpha=0.05\]

Statistic and p-value How step 2

\[t=19.957,\qquad \text{p-value}=2\times1.015856\times10^{-52}=2.031713\times10^{-52}\]

Compare How step 3

\[2.031713\times10^{-52}<0.05\ \Rightarrow\ \text{reject }H_0\text{ at the 5\% level}\]

Result sentence How step 5

The data give very strong evidence that mean brain weight changes linearly with head size (\(\beta_1\ne0\)).

▲

06.4Read the coefficient table Slides p.45

What Slides p.45

Example: Testing a Hypothesis · Lecture 3 · p.45 (numbers only) Intercept row: estimate 335.43231, standard error 47.77644, t value 7.021, p-value \(2.37\times10^{-11}\).
Slope row: estimate 0.26082, standard error 0.01307, t value 19.957, p-value less than \(2\times10^{-16}\).
\(\hat\sigma\): 72.35 on 234 degrees of freedom.

Each row tests \(H_0:\text{coefficient}=0\). The intercept row is \(\beta_0\). The slope row is \(\beta_1\).

Estimate: \(\hat\beta_0=335.43231\), \(\hat\beta_1=0.26082\).

Standard error: slope \(\hat\sigma/\sqrt{S_{xx}}=0.01307\); intercept \(\hat\sigma\sqrt{1/n+\bar x^2/S_{xx}}=47.77644\).

t value: estimate divided by standard error.

p-value: two-sided, \(P(|T|\ge|t|)\), \(T\sim t_{(n-2)}\). For the slope, the table prints only "less than \(2\times10^{-16}\)". Slides p.43 gives \(2.031713\times10^{-52}\).

\(\hat\sigma\): 72.35, with \(n-2=234\) degrees of freedom.

Residuals: minimum −174.88, first quartile −49.03, median −2.40, third quartile 46.72, maximum 242.41.

\(R^2=0.6299\), adjusted \(R^2=0.6283\), F statistic 398.3 on 1 and 234 degrees of freedom. Later lectures explain these.

How Added

Read one test from the table

Select the row.
Use \(H_0:\text{coefficient}=0\) vs. \(H_1:\text{coefficient}\ne0\). The table is only for \(\theta_0=0\).
Read the t value and the p-value.
Compare the p-value with \(\alpha=0.05\) (Section 06.3).

Self-check: Estimate / standard error = t value, to rounding. Degrees of freedom: 234.

Example · Brainhead data Slides p.45

Intercept row: t value Self-check

\[t=\frac{335.43231}{47.77644}=7.020873\approx7.021\]

Intercept row: p-value Section 06.2

\[2P(T\ge7.020873)=2.371264\times10^{-11}\approx2.37\times10^{-11}\]

Slope row: t value Self-check

\[t=\frac{0.26082}{0.01307}=19.9556\]

Unrounded: \(0.2608207/0.01306914=19.957\).

Slope row: p-value Section 06.3

\[2.031713\times10^{-52}<2\times10^{-16}\]

Self-check: Same p-value as Section 06.3.

Both rows reject \(H_0:\text{coefficient}=0\) at the 5% level. ▲

06.5Interpret the p-value Slides p.46–47

Type-I error: the test rejects \(H_0\) when \(H_0\) is true.

Type-I error rate: the probability of a Type-I error.

What Slides p.46–47

Interpreting P-Values · Lecture 3 · p.46–47 Does this mean pval \(=P(\beta_1=0)\)? No! Instead, it means that under the null hypothesis (i.e. assuming \(\beta_1=0\)) the probability of a test statistic as extreme as the one observed is pval. That’s why a small p-value is evidence against the null, since it would be particularly “rare” under the null (and hence surprising).

If we reject the null whenever we get a p-value below 5%, then if the null is true and we repeat the experiment over and over then we will only reject the null incorrectly 5% of the time. This is called the Type-I error rate.

\(\beta_1\) is a fixed constant. The p-value is a probability about the data, not about \(\beta_1\).

How Added

Interpret a p-value in one sentence

Write the condition: "Assuming \(H_0\) is true (\(\beta_1=\theta_0\)), …".
Write the event: "… the probability of a test statistic at least as extreme as the observed \(t\) …".
Write the value: "… is p-value = …".
Write the result: a small p-value is evidence against \(H_0\).

Self-check: The sentence says "assuming \(H_0\)". The probability is about the test statistic.

Example · Brainhead data Added

Condition How step 1

Assuming \(\beta_1=0\) (head size has no linear effect on mean brain weight),

Event and value How steps 2–3

the probability of \(|T|\ge19.957\) is \(2.031713\times10^{-52}\).

Result How step 4

The data give very strong evidence against \(\beta_1=0\).

▲

Why · The Type-I error rate is 5% Slides p.47

Section 04.4 defines \(t_{1-\alpha/2,\,n-2}\). Brainhead: \(t_{0.975,\,234}=1.970154\).

\[P(\text{reject}\mid H_0)=P(\text{p-value}<0.05\mid H_0)\]

Decision rule.

\[\text{p-value}<0.05\iff 2P(T\ge|t|)<0.05\]

Section 06.2.

\[2P(T\ge|t|)<0.05\iff P(T\ge|t|)<0.025\]

\[P(T\ge|t|)<0.025\iff |t|>t_{0.975,\,n-2}\]

Right-tail area at \(t_{0.975,\,n-2}\) is 0.025.

\[P(\text{reject}\mid H_0)=P(|T|>t_{0.975,\,n-2})\]

Under \(H_0\), \(T\sim t_{(n-2)}\).

\[P(|T|>t_{0.975,\,n-2})=0.025+0.025=0.05\]

Symmetry: each tail is 0.025.

∎

06.6One-sided tests Slides p.48–49

Two-sided alternative \(H_1:\beta_1\ne\theta_0\): evidence for \(\beta_1>\theta_0\) or \(\beta_1<\theta_0\) counts.

One-sided alternative, for example \(H_1:\beta_1>\theta_0\): only evidence for \(\beta_1>\theta_0\) counts.

What Slides p.48–49

One-sided hypothesis tests · Lecture 3 · p.48–49 Above we conducted a hypothesis test of the null \(H_0:\beta_1=\theta_0\) vs the two-sided alternative \(H_1:\beta_1\ne\theta_0\)
• Evidence against the null (and in favour of the alternative) is evidence that either \(\beta_1>\theta_0\) or \(\beta_1<\theta_0\)
Sometimes we are interested in evidence one direction: e.g., \(H_1:\beta_1>\theta_0\)
• Evidence against the null in favour of \(H_1\) is evidence that the true \(\beta_1\) is larger than \(\theta_0\)
• How do we compute a p-value here?
  ◦ Consider the test statistic: \(\frac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\sim t_{(n-2)}\)
  ◦ Evidence against the null and in favour of \(H_1\) is evidence that \(\beta_1>\theta_0\).
  ◦ So large values of \(\hat\beta_1-\theta_0\) indicate evidence against the null in favour of \(H_1\)
  ◦ \(\Longrightarrow\) large values of \(\frac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\) indicate evidence in favour of \(H_1\)
  ◦ I.e. p-value only includes one tail

Slides p.48 writes \(H_1:\theta\ne\theta_0\). It means \(H_1:\beta_1\ne\theta_0\).

For \(H_1:\beta_1>\theta_0\), the p-value is \(P(T\ge t)\). Do not take \(|t|\). Do not multiply by 2.

Figure 6-2. One-sided p-value for \(H_1:\beta_1>\theta_0\): the right tail \(T\ge t\) only (green), with \(t=2\). Area 0.02332811, half of the two-sided 0.04665622 in Figure 6-1.

How Added

One-sided test of \(H_0:\beta_1=\theta_0\) vs. \(H_1:\beta_1>\theta_0\)

Calculate \(t=\dfrac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\). Keep the sign.
Find \(P(T\ge t)\) for \(T\sim t_{(n-2)}\).
Do not multiply by 2. This area is the p-value.
Compare the p-value with \(\alpha\) (Section 06.3).

Self-check: When \(t>0\), the one-sided p-value is half the two-sided p-value.

Example · Brainhead data Added

Test \(H_0:\beta_1=0\) vs. \(H_1:\beta_1>0\): a larger head gives a larger mean brain weight.

Statistic How step 1

\[t=\frac{0.2608207-0}{0.01306914}=19.957\]

Right tail How steps 2–3

\[\text{p-value}=P(T\ge19.957)=1.015856\times10^{-52}\]

Self-check: \(2.031713\times10^{-52}/2=1.015856\times10^{-52}\).

Decision How step 4

\[1.015856\times10^{-52}<0.05\ \Rightarrow\ \text{reject }H_0\text{ in favour of }\beta_1>0\]

▲

Why · The p-value uses one tail Slides p.49

\[H_1:\beta_1>\theta_0\]

\[\hat\beta_1-\theta_0\ \text{large}\]

\(\hat\beta_1\) is unbiased for \(\beta_1\).

\[\frac{\hat\beta_1-\theta_0}{\hat\sigma/\sqrt{S_{xx}}}\ \text{large}\]

The denominator is positive.

\[\text{p-value}=P(T\ge t)\quad\text{under }H_0,\ T\sim t_{(n-2)}\]

Only values larger than \(t\) are more extreme.

∎ When \(t>0\), symmetry gives \(P(T\ge t)=\tfrac12P(|T|\ge|t|)\).

06.7Practice Added

Q1. Brainhead: \(\hat\beta_1=0.2608207\), \(\hat\sigma/\sqrt{S_{xx}}=0.01306914\), \(n=236\). Test \(H_0:\beta_1=0.25\) vs. \(H_1:\beta_1\ne0.25\) at the 5% level.

Answer

\(t=\dfrac{0.2608207-0.25}{0.01306914}=\dfrac{0.0108207}{0.01306914}=0.8280\).

With \(t_{(234)}\): \(P(T\ge0.8280)=0.2043\). p-value \(=2\times0.2043=0.4085\).

\(0.4085>0.05\): not enough evidence to reject \(H_0:\beta_1=0.25\) at the 5% level.

Q2. Same data. Test \(H_0:\beta_1=0.24\) vs. \(H_1:\beta_1>0.24\) at the 5% level. Also give the two-sided p-value for \(H_1:\beta_1\ne0.24\).

Answer

\(t=\dfrac{0.2608207-0.24}{0.01306914}=\dfrac{0.0208207}{0.01306914}=1.5931\).

One-sided p-value: \(P(T\ge1.5931)=0.05624\), \(T\sim t_{(234)}\).

\(0.05624>0.05\): not enough evidence to reject \(H_0\) in favour of \(\beta_1>0.24\) at the 5% level.

Two-sided p-value: \(2\times0.05624=0.1125\).

Q3. Slides p.45, intercept row: estimate 335.43231, standard error 47.77644, t value 7.021, p-value \(2.37\times10^{-11}\). (a) Hypotheses? (b) Check the t value. (c) Interpret the p-value.

Answer

(a) \(H_0:\beta_0=0\) vs. \(H_1:\beta_0\ne0\).

(b) \(335.43231/47.77644=7.020873\approx7.021\).

(c) Assuming \(\beta_0=0\), the probability of a test statistic at least as extreme as \(|t|=7.021\) is \(2.37\times10^{-11}\). Reject \(H_0\) at the 5% level.