7 Model Evaluation Free Online FlashCards

What roles do training, validation, and test data serve?

Training data fits the model, validation data supports model comparison and tuning, and a separate test set provides a final, relatively unbiased performance estimate.

What is a regression residual?

A residual is the actual value minus the predicted value: $e_i = y_i - \hat{y}_i$.

What does classification accuracy measure?

Accuracy is the fraction of predictions that are correct: $\frac{TP+TN}{TP+FP+TN+FN}$.

What does a classification threshold control?

A threshold converts classifier scores or probabilities into labels. Changing it changes the confusion matrix and usually trades precision against recall; choose it on validation data.

Why should the test set not guide repeated model choices?

Repeatedly using the test set to make choices can make its performance score overly optimistic.

What does mean absolute error measure?

MAE is the average absolute residual, $\frac{1}{n}\sum_i |e_i|$. It is in the outcome’s units and is less affected by a few very large errors than squared-error measures.

How is mean squared error calculated, and what does squaring do?

MSE is the average squared residual, $\frac{1}{n}\sum_i e_i^2$. Squaring gives large errors more weight, and the measure is in squared outcome units.

What is RMSE, and how does it treat large errors?

RMSE is the square root of MSE, $\sqrt{\text{MSE}}$. It is in the outcome’s units and gives extra weight to large errors.

How should common values of $R^2$ be interpreted?

$R^2=1$ indicates perfect predictions; $R^2=0$ matches the test-set-mean baseline; a negative value is worse than that baseline. It is not an error in outcome units.

Why can accuracy mislead when classes are imbalanced?

With rare positive cases, a model can achieve high accuracy by predicting every case as negative while finding no positives. Class-specific measures reveal this weakness.

What does precision measure, and when might it be prioritized?

Precision is the fraction of predicted positives that are truly positive: $\frac{TP}{TP+FP}$. Prioritize it when false alarms or unnecessary interventions are costly.

What does recall measure, and when might it be prioritized?

Recall is the fraction of actual positives found: $\frac{TP}{TP+FN}$. Prioritize it when missing a positive case is costly.