7 Model Evaluation Free Online FlashCards

What makes a model evaluation useful?

Model evaluation estimates performance on unseen data; the evaluation data and metric should reflect the model’s intended use.

What are the roles of training, validation, and test sets?

The training set fits the model, the validation set supports model selection or tuning, and the test set provides a final performance estimate.

Why should test-set results not guide repeated model selection?

Repeatedly selecting models using test results leaks test-set information into model selection and can make the reported score overly optimistic.

How does k-fold cross-validation work?

Divide the training data into k folds; train on k−1 folds and validate on the remaining fold, repeating until each fold has served as validation.

What do stratified folds preserve approximately?

Stratified folds approximately preserve class proportions, helping keep classification folds representative of the overall class distribution.

How should repeated observations from one person or group be split?

Keep observations from the same person or group together across folds to prevent information from leaking between training and evaluation.

How should time-dependent data be split for evaluation?

For time-dependent data, train on earlier observations and evaluate on later ones so the split reflects prediction of the future.

How can preprocessing avoid leaking information across splits?

Fit learned preprocessing, such as scaling or feature selection, using only the training portion of each split.

How should common values of R² be interpreted?

An R² of 1 is perfect; 0 means no improvement over predicting the test-set mean, and a negative value means worse performance than that baseline.

What information does a confusion matrix show?

A confusion matrix counts actual versus predicted classes, including true positives, false positives, true negatives, and false negatives in a binary task.

What does precision measure, and when is it especially important?

Precision is the fraction of predicted positives that are truly positive: $\frac{TP}{TP+FP}$. It matters when false alarms are costly.

What does recall measure, and when is it especially important?

Recall is the fraction of actual positives found by the model: $\frac{TP}{TP+FN}$. It matters when missing positive cases is costly.