State what the evaluation measures
A metric becomes meaningful when the evaluated population, decision rule, labels, and time boundary are explicit. Before changing a model, inspect the software that builds its report. A threshold bug or a changed denominator can move a metric without improving a single prediction.
Ask which rows belong in the cohort, which label is positive, how ties at the threshold are handled, and what happens when a label is unavailable. Those choices belong in the report’s contract. Missing ground truth should not silently become a negative label.
Calculate a small report by hand
In an independent example of 100 labeled decisions, suppose there are 8 true positives, 2 false positives, 4 false negatives, and 86 true negatives. Precision is 8 / (8 + 2) = 80%. Recall is 8 / (8 + 4), approximately 66.7%. Accuracy is (8 + 86) / 100 = 94%. The attractive accuracy number does not reveal that one third of actual positives were missed.
Use a tiny cohort like this as an oracle for the report implementation. Then vary one factor at a time: a score exactly at the threshold, a cohort with no predicted positives, or a row without a label. Specify how an undefined ratio is represented instead of letting the runtime choose for you.
Precision: of predicted positives, how many were positive?
Recall: of actual positives, how many were detected?
Accuracy: of all evaluated rows, how many were correct?Separate when an event happened from when it was known
Now consider a prediction made at 10:00. An event happened at 09:40 but arrived in the data platform at 10:20. A historical feature reconstructed from today’s data might include it because its event time precedes the prediction. The live predictor at 10:00 could not have known about it.
This is a point-in-time availability problem. For a historical row, reconstruct what information was available at prediction time, then apply the feature’s event-time window. A correction published at 11:00 creates the same issue: using the latest value everywhere can rewrite what an earlier predictor would have seen. Deletions and revisions need explicit historical semantics too.
09:40 Event occurs
10:00 Prediction is made
10:20 Event first becomes available
The 10:00 feature cannot use information first available at 10:20.Inspect leakage beyond feature timestamps
Preprocessing can also transfer information from evaluation data into training. Split data before fitting transformations such as normalization or feature selection, and fit those transformations using the training partition. Pipelines help keep fitting and transformation attached to the correct stage.
Time-based evaluation and entity separation answer different questions. A temporal holdout can still share near-duplicate entities with training; an entity split can still include information published after prediction time. Choose a split that matches the decision the model will make, and document what the evaluation does and does not simulate.
Make the failure visible before repairing it
For a classifier report, compare the implementation with your hand-calculated cohort. For historical features, hold the prediction time constant while changing when a row becomes available. The expected output should change only when the information crosses the availability boundary defined by the contract.
nrml’s ML exercises focus on the engineering around models: evaluation reports, historical feature reconstruction, and retrieval ranking. You inspect and repair repository code, without needing to train a large model. Explain the data boundary, show a concrete failing case, and verify that existing report or catalog workflows still work after the change.