5 Classification

Classification predicts categories

Classification is a prediction task: a model assigns an example to one of a set of categories called classes. It uses the example's features—such as the words in an email—and learns from training examples that already have labels. For instance, an email classifier might label messages as spam or not spam.

Unlike regression, which predicts a numerical quantity, classification predicts a category. A task with two possible classes is binary classification; a task with more than two classes is multiclass classification. An image classifier might choose among cat, dog, and bird. Some classifiers also produce a score or probability for each class rather than only a final label.

How decision boundaries separate predictions

A classifier divides the space of possible feature values into regions and assigns a class to each region. The decision boundary is where the predicted class changes. For example, if fruit is plotted using weight and sweetness as features, one region might be predicted as apple and another as orange; the line or curve between them is the decision boundary.

Real datasets can include many features. In that case, boundaries exist in higher-dimensional spaces and cannot always be drawn on a page. Different models can form different kinds of boundaries: a linear model or linear support vector machine can separate classes with a straight boundary in a two-feature plot, while a decision tree creates regions through feature-based splits. A k-nearest neighbors classifier predicts from nearby training examples and can create more irregular boundaries.

Common classification models

Common classification models use different strategies to predict categories:

• Logistic regression estimates class probabilities. For a binary task, a chosen threshold turns the probability into one of the two labels.

• Decision trees apply a sequence of feature-based rules to reach a class prediction.

• Support vector machines (SVMs) seek a separating boundary, often aiming for a wide margin between classes.

• k-nearest neighbors (k-NN) predicts the class most common among a new example's nearest labeled training examples.

Turning probabilities into labels

For a binary classifier, a threshold determines how an estimated probability becomes a label. For example, if an email model estimates a spam probability of $0.82$ and its threshold is $0.50$, it predicts spam.

Changing the threshold can change which messages receive that label, so the threshold should suit the task. In this way, classifiers that return probabilities or scores can use them to make final category predictions.