The scenario
An evaluation desk turns labeled scores into classification reports. The work is to make the report’s decisions and metrics agree with the meaning of its inputs.
A failure to reason about
A dashboard reports a convincing metric while a small hand-calculated cohort shows that some positive predictions were counted in the wrong place.
Illustrative context for the topic. The exercise’s full requirements arrive in the workspace.
Questions to bring to your review
- Which label represents the positive class?
- Which rows belong in the denominator?
- Can a tiny cohort reveal the difference between precision and recall?
Skills you will practice
- Binary classification
- Confusion matrices
- Precision and recall
- Regression testing
Read the existing code, reproduce the reported behavior, review proposed changes, and verify your final implementation. You can use AI assistance while staying responsible for the result.
Prepare with a worked example
- ML evaluation and data leakage: inspect the evidence
Work through a confusion matrix and a historical feature example to spot misleading model metrics and unavailable training data.
See how nrml evaluates an attempt, including the limits of automated evidence.