The scenario
A retrieval workbench combines lexical and vector search evidence. The merged ranking must remain explainable while respecting which catalog documents are eligible.
A failure to reason about
A document appears in both search sources, but the merged preview loses the evidence that should influence its position.
Illustrative context for the topic. The exercise’s full requirements arrive in the workspace.
Questions to bring to your review
- What do the source positions mean?
- When are catalog filters applied?
- How can repeated runs return the same order?
Skills you will practice
- Reciprocal rank fusion
- Retrieval pipelines
- Ranking evidence
- Deterministic ordering
- AI-assisted review
Read the existing code, reproduce the reported behavior, review proposed changes, and verify your final implementation. You can use AI assistance while staying responsible for the result.
Prepare with a worked example
- ML evaluation and data leakage: inspect the evidence
Work through a confusion matrix and a historical feature example to spot misleading model metrics and unavailable training data.
See how nrml evaluates an attempt, including the limits of automated evidence.