11 Simple Linear Regression

A progressive guide to modeling quantitative relationships with simple linear regression, interpreting fitted lines, checking assumptions, making predictions, and conducting inference for the population slope.

Model Purpose and Setup

A model studies the relationship between two quantitative variables:

The predictor, or explanatory variable, is denoted by $X$ .
The response variable is denoted by $Y$ .

The population model is:

Y_i=\beta_0+\beta_1x_i+\varepsilon_i,

where $\beta_0$ is the population intercept, $\beta_1$ is the population , and $\varepsilon_i$ is the random error for observation $i$ . The model concerns the mean response at each predictor value:

E(Y\mid X=x)=\beta_0+\beta_1x.

The phrase refers to using one predictor and a linear mean relationship. Regression summarizes an association and can support prediction, but it does not automatically establish causation.

Takeaway: Regression connects a quantitative predictor to the mean of a quantitative response; the data-collection design determines whether a causal conclusion is justified.

The Least-Squares Regression Line

For paired observations $(x_i,y_i)$ , the estimated regression equation is:

\hat y=b_0+b_1x.

The least-squares method chooses $b_0$ and $b_1$ to minimize the sum of squared vertical errors:

SSE=\sum_{i=1}^{n}(y_i-\hat y_i)^2.

The coefficient estimates are:

b_1=\frac{\sum (x_i-\bar x)(y_i-\bar y)}{\sum (x_i-\bar x)^2}

and:

b_0=\bar y-b_1\bar x.

The fitted line always passes through $(\bar x,\bar y)$ . The $b_1$ gives the estimated change in predicted $Y$ for a one-unit increase in $X$ . The intercept $b_0$ is the predicted response when $X=0$ , but that interpretation is useful only when zero is meaningful and within the relevant data range.

For example, if:

\hat y=52+4.5x,

then each additional hour studied is associated with an estimated increase of $4.5$ exam-score points. The predicted score at zero hours is $52$ , although this interpretation may be questionable if zero is outside the observed range or the model is not appropriate there.

Takeaway: Least squares produces the straight line with the smallest possible sum of squared residuals, and its coefficients must be interpreted in the original units.

Fitted Values and Residuals

For observation $i$ , the fitted value is:

\hat y_i=b_0+b_1x_i,

and the is:

e_i=y_i-\hat y_i.

A positive means the observed response is above the fitted line.
A negative means the observed response is below the fitted line.
A of zero means the observation lies exactly on the line.

Suppose the fitted equation is $\hat y=52+4.5x$ . At $x=6$ :

\hat y=52+4.5(6)=79.

If the observed score is $84$ , then:

e=84-79=5.

Thus, the model underpredicted the score by $5$ points. plots help determine whether a linear model is reasonable: random scatter around zero is desirable, while systematic curves or changing spread indicate possible problems.

Takeaway: A measures observed minus predicted response, and its pattern across the data is more informative than any single .

Explained Variation and $R^2$

The $R^2$ measures the proportion of sample variation in the response explained by the fitted regression model. Define:

SST=\sum(y_i-\bar y)^2,

as total variation,

SSR=\sum(\hat y_i-\bar y)^2,

as variation explained by the regression, and:

SSE=\sum(y_i-\hat y_i)^2,

as unexplained variation. Then:

R^2=\frac{SSR}{SST}=1-\frac{SSE}{SST}.

If $R^2=0.64$ , then $64\%$ of the observed sample variation in $Y$ is explained by its linear relationship with $X$ , while $36\%$ is not explained by this model. A high $R^2$ does not prove causation, guarantee that the model is appropriate, or ensure accurate predictions outside the observed range. The value of $R^2$ itself lies between $0$ and $1$ .

Takeaway: $R^2$ describes explained sample variation; it is not a complete measure of model quality or evidence of a causal relationship.

Inference for the Population

The sample $b_1$ estimates the population $\beta_1$ . A common uses:

H_0:\beta_1=0

against:

H_a:\beta_1\ne0.

The test statistic is:

t=\frac{b_1-0}{SE(b_1)},

which follows a $t$ -distribution with:

df=n-2

degrees of freedom under the model assumptions and the null hypothesis. A small p-value provides evidence that the population has a nonzero linear association, but it does not demonstrate causation or practical importance. A directional alternative, such as $H_a:\beta_1>0$ , should be selected only when justified in advance.

A $100(1-\alpha)\%$ confidence interval for the population is:

b_1\pm t^*SE(b_1),

where $t^*$ is based on $n-2$ degrees of freedom. For example, an interval of $(2.1,6.9)$ supports a positive population because every value in the interval is above zero.

Takeaway: Use the test or confidence interval to assess evidence about the population relationship, while keeping statistical significance separate from practical importance and causation.

Prediction and Its Uncertainty

For a predictor value $x^*$ , the fitted response is:

\hat y^*=b_0+b_1x^*.

Two intervals answer different questions:

A confidence interval for the mean response estimates the average value of $Y$ for all units with $X=x^*$ .
A estimates the response for one new unit with $X=x^*$ .

The is wider because it includes both uncertainty in estimating the mean and individual random variation. For a new individual response, its form is:

\hat y^*\pm t^*s\sqrt{1+\frac{1}{n}+\frac{(x^*-\bar x)^2}{S_{xx}}},

where:

S_{xx}=\sum(x_i-\bar x)^2

and:

s=\sqrt{\frac{SSE}{n-2}}.

Predictions are usually most precise near $\bar x$ and less precise as $x^*$ moves farther from the center of the observed predictor values. occurs when a prediction is made outside the observed range of $x$ -values; it is risky because the linear pattern may not continue.

Takeaway: Distinguish mean-response intervals from individual prediction intervals, and make predictions only within a scientifically reasonable range.

Assumptions and Diagnostics

Reliable regression analysis depends on the :

Linearity: The mean response changes linearly with the predictor.
Independence: Errors are independent of one another.
Normality: Errors are approximately normally distributed for each predictor value.
Equal variance: Errors have the same variance across predictor values.

Use the following diagnostics:

A scatterplot of $Y$ versus $X$ should show an approximately straight-line pattern. Curvature suggests nonlinearity.
A residuals-versus-fitted-values plot should look like a random horizontal band centered near zero. A curve suggests nonlinearity, while a funnel or megaphone suggests unequal variance.
A residuals-versus-order or time plot can reveal dependence or autocorrelation.
A normal probability plot or histogram can assess approximate normality, which is especially relevant for small-sample inference and prediction intervals.
Outlier and leverage checks identify unusual observations. A large indicates an outlier in the response direction; a point far from the mean predictor value may have high leverage and strongly affect the line.

Minor nonnormality may have limited impact in large samples, but serious violations, dependence, strong outliers, nonlinearity, or unequal variance require caution or alternative methods.

Takeaway: Check the assumptions with the data rather than treating a fitted line as automatically trustworthy.

A Complete Regression Analysis

A complete analysis can follow this sequence:

Identify the predictor and response variables.
Examine a scatterplot for direction, form, strength, and unusual observations.
Fit the least-squares line $\hat y=b_0+b_1x$ .
Interpret the using the variables and their units.
Examine plots and assess linearity, independence, normality, and equal variance.
Report $R^2$ as a description of the proportion of sample variation explained.
When appropriate, test $H_0:\beta_1=0$ and report a p-value or confidence interval for $\beta_1$ .
Make predictions only within a scientifically reasonable range, distinguishing mean-response confidence intervals from individual prediction intervals.
Avoid causal claims unless the study design supports them.

Final takeaway: A sound analysis combines interpretation of the fitted line, -based diagnostics, measures of explained variation, inference for the , and cautious prediction.