3 Classification Fundamentals

Classification as supervised learning

Classification is a supervised learning task. A model learns from examples that include input data and known category labels, then predicts a category for a new example.

A task with two possible categories is binary classification; one with three or more categories is multiclass classification. A classifier may produce a probability or score for each category, and a decision rule—often a threshold—turns that score into a predicted class.

Examples, features, and labels

A dataset consists of examples. Each example has Features, which are the input information used to make a prediction, and a Label, which is the correct category the model is meant to predict. For an email, features might include its text, sender, or number of links; its label might be spam or not spam.

During training, the model uses examples’ features and labels to learn patterns connecting inputs to categories. At prediction time, it receives features for a new example and estimates its label. Features should be available when the prediction is made. Accidentally including information that reveals the answer is data leakage and can make evaluation misleading.

Training and test data

Training data is used to fit the model. Test data is held back until the model is ready and used to estimate how well it handles unseen examples.

Testing on training examples can give an overly optimistic result: a model may memorize those examples without learning patterns that generalize. Keep test examples separate from training, and avoid repeatedly using the test set to make modeling decisions. If a validation set is used to select settings or compare models, reserve the test set for a final evaluation.

Where appropriate, stratified splitting can help preserve class proportions across sets. Data with time order or related groups may need a split that respects that structure.

Understanding the confusion matrix

For a binary classifier, first choose one class as the positive class. Compare each prediction with the known label to count four outcomes:

• True positive (TP): A positive example correctly predicted positive.

• False positive (FP): A negative example incorrectly predicted positive.

• True negative (TN): A negative example correctly predicted negative.

• False negative (FN): A positive example incorrectly predicted negative.

A confusion matrix contains these counts. It shows both how many predictions were wrong and what kinds of errors occurred.

Calculating and choosing metrics

Common metrics summarize different aspects of binary-classification performance. Using the counts from the confusion matrix, calculate them as follows:

• Accuracy is the fraction of all predictions that are correct.

• Precision is the fraction of predicted-positive examples that truly are positive. High precision means fewer false alarms.

• Recall is the fraction of actual positive examples the model correctly identifies. High recall means fewer missed positives.

• F1 score is the harmonic mean of precision and recall. It provides a single summary that is low when either precision or recall is low.

For an email classifier with $TP = 8$, $FP = 2$, $FN = 4$, and $TN = 6$, accuracy is $\frac{14}{20} = 70\%$, precision is $\frac{8}{10} = 80\%$, recall is $\frac{8}{12} \approx 66.7\%$, and the F1 score is approximately $72.7\%$.

No single metric is best for every task. Accuracy can conceal poor performance when one class is much more common than another. Precision is especially relevant when false positives are costly, while recall matters when missing positive cases is costly. Choose a metric or balance of metrics in light of the consequences of each type of error.