Evaluations: Metrics, Validation, and Error Analysis

ML validation • 7-9 min read • 1-3 hours to run analyses

TL;DR: Pick metrics that reflect business cost, validate properly (no leakage), and perform targeted error analysis to fix model weaknesses.

Choosing the right metric

• Match metrics to action: if false positives are costly, prioritize precision; if misses are costly, prioritize recall.
• Use business-aligned metrics (lift, revenue per decision) where possible.

Validation strategies

• Cross-validation for IID data.
• Time-based holdout for temporal data.
• Nested CV for hyperparameter selection when data is small.

Confusion matrix and derived metrics

• True positive, false positive, true negative, false negative.
• Precision = TP / (TP + FP)
• Recall = TP / (TP + FN)
• F1 = harmonic mean of precision and recall

Visual: confusion matrix

Replace with a 2x2 confusion matrix SVG showing TP/FP/TN/FN with formulas.

Calibration and probabilistic outputs

• Calibration curves show if predicted probabilities reflect true frequencies.
• Use isotonic or Platt scaling when necessary.

Error analysis workflow

1. Slice the dataset by feature segments (region, device, cohort).
2. Look for systematic errors (model underperforms on subgroup).
3. Check feature distributions and input quality for the failing slices.
4. Iterate: add features, correct labels, or retrain different model families.

Visual checks

• ROC and PR curves for classification.
• Residual plots for regression.
• Learning curves to diagnose under/overfitting.

Practical example (Python snippets)

from sklearn.metrics import confusion_matrix, roc_auc_score, mean_absolute_error

# classification
cm = confusion_matrix(y_test, preds)
print(cm)
print('AUC:', roc_auc_score(y_test, probs[:,1]))

# regression
print('MAE:', mean_absolute_error(y_test, y_pred))

Monitoring in production

• Track chosen metrics on live traffic.
• Monitor data drift (feature distributions) and label drift.
• Automate retrain and rollback strategies.

Conclusion

Good evaluation is iterative: pick business-aligned metrics, validate carefully, and use error analysis to guide improvement. This completes the series’ foundational pieces—next: build a small end-to-end demo pipeline.

Appendix: BigML evaluation quickstart (transcript highlights)

1. Holdout split

   • Reserve a portion of your dataset (commonly 20%) as a test/holdout set. Do not use it while training or tuning hyperparameters.

2. Create an evaluation

   • After building a model, use the platform’s evaluation tools to score the model against the holdout dataset; view accuracy, confusion matrix, ROC/AUC.

3. Interpret confusion matrix

   • Inspect TP/FP/TN/FN cells to understand error types; tune thresholds to balance precision vs recall depending on cost.

4. Calibration and probability thresholds

   • Use calibration plots and Platt/isotonic scaling where probability estimates are important for downstream decisions.

5. Error analysis workflow in BigML

   • Slice & dice predictions by cohort (region, device, buckets) via filters; export failing slices and inspect feature distributions and labels.

6. Monitor & retrain

   • Track live metrics, set alerts for drops, and retrain on rolling windows or when drift is detected.