4 Regression

Regression basics

Regression is a supervised learning method for predicting a numeric value, such as a home's sale price, a trip's duration, or tomorrow's temperature. A dataset contains examples, each with one or more features (input values) and a numeric target (the value to predict). A regression model learns a relationship between features and target from training data, then uses that relationship to predict targets for new examples.

Linear regression

A common starting point is linear regression. With one feature, the prediction can be written as:

With several features, the model adds a weighted contribution from each one. It learns an intercept and coefficients (weights) by fitting predictions to the training examples. Ordinary least squares chooses coefficients that minimize the sum of squared differences between actual and predicted targets.

For example, a model predicting a home's price from floor area might use:

According to this model, an additional square foot is associated with a $\$250$ increase in predicted price. For a $1{,}000$-square-foot home, the prediction is $\$300{,}000$. The intercept is the prediction at zero square feet; it may have little practical meaning if that value is outside the data's useful range. These are model relationships, not proof that changing a feature causes the target to change.

Training and testing

Split labeled data into a training set and a test set. Fit the model using only the training set, and keep the test set separate so it can estimate performance on examples the model did not train on. In practice, preprocessing steps and choices made during model development should also avoid using test-set information.

This Python example assumes X is a table of numeric features and y contains the corresponding numeric targets:

fit learns the model parameters; predict applies the fitted model to new feature rows. Compare test-set predictions with the actual targets to assess how well the model generalizes.

Residuals and prediction errors

A residual is the actual target minus the predicted target:

A positive residual means the model predicted too low; a negative residual means it predicted too high. Looking at residuals can reveal patterns, such as consistently larger errors for expensive homes or errors that curve with floor area. Such patterns may indicate that the model is missing an important relationship.

Evaluating regression metrics

Mean absolute error (MAE) is the average absolute prediction error. It is expressed in the target's units, so an MAE of $\$20{,}000$ means predictions are off by $\$20{,}000$ on average in absolute terms. Root mean squared error (RMSE) is also expressed in target units, but it penalizes large errors more strongly than MAE.

R² (coefficient of determination) compares prediction errors with the variation around the target's mean. A value of $1$ indicates perfect predictions; $0$ means performance equivalent to predicting the test-set mean; and a value below $0$ means worse performance than that baseline. R² alone does not show the typical size of an error, so it is useful to consider it alongside an error metric.

Generalization and model fit

A model that fits the training examples extremely well may still perform poorly on unseen data; this is overfitting. Compare training and test performance, and prefer a model that performs well on new data rather than one that merely memorizes the training set.

A straight-line model can also miss a curved relationship. Residual patterns and performance on held-out data help identify when the chosen model may be inadequate. Taken together, test performance and residual patterns help show whether a model is useful beyond its training data.