4 Regression

What regression predicts

Regression is a supervised learning task: a model learns patterns from examples with known targets, then uses those patterns to predict targets for new examples. The target is numerical and can vary across a range of values, unlike a classification prediction, which is a category.

A regression dataset contains examples described by features and paired with a numerical target. For instance, a home’s floor area and number of bedrooms can be features, while its sale price is the target. Other example targets include tomorrow’s temperature and the time needed to complete a delivery.

Baseline and linear models

A Mean baseline predicts the average target in the training data for every example. It is a simple reference point; a more useful model should generally do better on data it has not seen.

Linear regression predicts a target as a weighted sum of features. A simplified rent model is:

In this example, the model predicts a base rent of $700$ currency units, plus $18$ currency units for each square meter. With multiple features, the model adds a term for each feature. Ordinary least squares chooses coefficients to minimize the sum of squared differences between observed and predicted targets.

Regularization, trees, and ensembles

Regularized linear regression methods, including Ridge and Lasso, add penalties on coefficient size. These methods can help when there are many features or when features overlap in the information they provide. Lasso can also shrink some coefficients to zero.

Decision-tree regression repeatedly divides examples according to feature values, then predicts a value for each resulting group. It can represent nonlinear patterns, but an overly deep tree can fit training examples too closely.

Ensemble regression includes random forests, which average predictions from many trees, and gradient-boosted trees, which build trees in sequence to improve predictions. These models can capture complex relationships, but they are often less straightforward to interpret than a small linear model.

Interpreting predictions and coefficients

A prediction, written $\hat{y}$, is the model’s estimated target for an example. When the actual value $y$ is known, compare it with the prediction: the residual is $y - \hat{y}$. A positive residual means the model predicted too low; a negative residual means it predicted too high.

For a linear model, a coefficient describes how the prediction changes with a feature when the other features are held fixed. In the rent example, the area coefficient of $18$ means the predicted rent increases by $18$ currency units for each additional square meter, according to this model. This describes the model’s pattern; it does not prove that increasing an area causes rent to rise by that amount. Coefficients depend on the data, the included features, and their units.

Prediction limits and generalization

Predictions are estimates, not guarantees. They may be unreliable when an input differs greatly from the training examples, when important features are missing, or when the data’s patterns change. In particular, a linear model’s prediction far outside the range of training data is an extrapolation and may not be realistic.

Check predictions against their context, and use held-out data to assess how well a model generalizes. Interpret predictions in the target’s units, while keeping in mind that their reliability can be limited for examples unlike those used in training.