nrml
StringsAnthropic

Longest-Match Tokenizer

Repair unknown-character recovery, select the longest vocabulary match, and compact unknown runs.

The scenario

Repair unknown-character recovery, select the longest vocabulary match, and compact unknown runs.

A failure to reason about

Use a small strings scenario to compare current behavior with the requirement.

Illustrative context for the topic. The exercise’s full requirements arrive in the workspace.

Questions to bring to your review

  • Which caller makes the reported behavior visible?
  • What focused regression would distinguish the current behavior from the requirement?
  • What existing behavior must remain intact after the change?

Skills you will practice

  • String matching
  • Cursor invariants
  • Greedy selection
  • API semantics
  • Testing

Read the existing code, reproduce the reported behavior, review proposed changes, and verify your final implementation. You can use AI assistance while staying responsible for the result.

Prepare with a worked example

See how nrml evaluates an attempt, including the limits of automated evidence.