2 Features, Labels, and Data Preparation

Features and labels

A machine-learning dataset is a collection of examples. In supervised learning, each example pairs features—the information given to a model—with a label, the value it is meant to predict. For example, when predicting a home's sale price, its size and location might be features, and the sale price is the label.

A feature should be useful for prediction and available when the model will be used. Labels should represent the intended outcome and be defined consistently. Features are often represented as $X$ and labels as $y$; the model learns from the relationship between them.

Prepare data carefully

Features can be numeric, categorical, text, dates, or other data types. Because a model may need numeric values, preparation can involve encoding categories, scaling numeric values, or converting dates into useful components. For example, a category such as "blue" can be represented with indicator columns. The appropriate transformations depend on the data and the model.

Before training, check that examples are correctly matched with their labels. Identify missing, invalid, or inconsistent values, and look for duplicates. Remove or resolve errors when justified, and decide how to handle missing values.

Avoid features that reveal the answer indirectly or contain information unavailable at prediction time. Such information can create data leakage, making evaluation appear better than performance on genuinely new examples.

Fit preprocessing without leakage

Split the data before fitting transformations that learn from it, such as calculating averages for imputation or means and standard deviations for scaling. Fit these transformations on the training data, then apply the same fitted transformations to validation, test, and future data. A pipeline can help keep this process consistent and prevent leakage.

Separate training, validation, and testing

A dataset is commonly divided into three subsets with distinct roles:

• Training set: used to fit the model and learn patterns.

• Validation set: used during development to compare models and make choices, such as selecting features or settings.

• Test set: held aside for final evaluation after development choices are complete.

A common illustrative division is $70\%$ for training, $15\%$ for validation, and $15\%$ for testing, but no single ratio fits every dataset. Each subset needs enough examples for its purpose. Repeatedly consulting the test set can influence development decisions and weaken its role as an independent final check.

Use training data to fit the model and any learned preprocessing. Use validation results to guide development. Once choices are settled, evaluate on the test set; do not use that result to keep tuning the model.

Choose a split that fits the data

The way data is split should reflect how the model will encounter new data. For classification, stratified splitting can help preserve class proportions in each subset. If multiple rows belong to the same person, household, or other group, keep each group in only one subset. For time-dependent data, split chronologically so that training examples precede validation and test examples.

Example of a stratified split

This example makes an illustrative $70/15/15$ split for a classification dataset. X contains features and y contains labels. The first split reserves $20\%$ for testing. The second split takes $15\%$ of the original dataset for validation: $15\%$ of the $80\%$ remaining after the test set is reserved. Stratification is suitable when each class has enough examples.