5 Classification

What classification predicts

Classification is a supervised learning task: a model learns from examples whose category labels are known, then uses example features to predict labels for new examples. For instance, an email classifier can use words and sender details to predict whether a message is spam or not spam.

Unlike regression, which predicts a numerical value, classification predicts one or more categories. The kind of classification task depends on how many labels are possible and whether an example can receive more than one label.

Types of classification

In binary classification, there are two possible labels. Predicting whether an email is spam or not spam is an example.

In multiclass classification, each example receives one of three or more mutually exclusive labels. For example, a fruit might be classified as an apple, orange, or pear.

In multilabel classification, one example can receive several labels at once. A photo, for example, might be tagged both beach and sunset.

Common classifiers

Classifiers differ in how they use training examples to distinguish categories:

• Logistic regression learns a weighted relationship between features and classes. Despite its name, it is commonly used for classification and can estimate class probabilities as well as predict labels.

• Decision trees make a sequence of feature-based decisions, such as checking whether a message is from an unknown sender. Their resulting rules can be easy to inspect.

• Random forests combine predictions from many decision trees. This often improves accuracy and helps control overfitting.

• Support vector machines (SVMs) seek a boundary that separates classes with a wide margin. Kernels can represent more complex boundaries.

• K-nearest neighbors (KNN) predicts from the labels of nearby training examples, typically using a majority vote.

• Naive Bayes estimates class probabilities using Bayes’ theorem and a simplifying assumption that features are conditionally independent given the class. It is often used for text, including spam filtering.

No classifier is best for every dataset. A suitable choice depends on the data, the task, and practical needs such as speed and interpretability.

Labels and probability estimates

A classifier can return a class prediction, meaning its selected label, or probability estimates for the possible labels. For example, a spam model might estimate a probability of $0.82$ for spam and $0.18$ for not spam. A multiclass model might estimate probabilities of $0.70$ for apple, $0.20$ for pear, and $0.10$ for orange. For mutually exclusive classes, the estimates sum to $1$.

A predicted probability is an estimate, not a guarantee. Not every classifier provides probabilities directly, and probability estimates can be poorly calibrated: predictions assigned a probability of $0.8$ do not necessarily prove correct $80\%$ of the time. Naive Bayes, for example, can classify effectively while producing probability estimates that should be treated cautiously.

How thresholds change labels

In binary classification, a model may use a decision threshold to turn the estimated probability of the positive class into a label. With a threshold of $0.5$, an estimated spam probability of $0.82$ produces the label spam.

A different threshold may be appropriate when the costs of mistakes differ. For example, a medical screening system may use a lower threshold to reduce missed positive cases. Changing the threshold changes the label predictions, even though the underlying probabilities remain the same.