AI interview practice: check that generated tests can fail

Write the expected collection before looking at the tests

Suppose an application accepts a list of tags for a saved search. Trim each string, discard empty results, and remove duplicates using the lowercased trimmed value as the identity. Keep the first surviving spelling and preserve its encounter order. Return a new array without changing the input. This exercise accepts arrays of strings; validation of other input types is a separate requirement.

For the input [" Ops ", "ops", "", " API ", "api"], write ["Ops", "API"] on paper. Those two values settle several questions at once: trimming changes display text, case does not distinguish identity, the first spelling wins, and empty strings do not survive. Do not let an assistant quietly replace that contract with alphabetical sorting or all-lowercase output.

Before asking for tests, predict a second example yourself. ["z", "A", "a"] should return ["z", "A"]. That example separates encounter order from sorting and makes the choice of display spelling visible. Small, independently calculated outputs give you something to defend when the implementation and its generated explanation agree with each other but disagree with the task.

Find the test that accepts an empty answer

A generated suite might loop over the returned tags and assert that every tag is nonempty and trimmed. Those are sensible properties, but a function that always returns an empty array passes them: the loop body never runs. A list of successful assertions can therefore conceal a missing result. Ask which requirement forces the expected tags to exist.

The worked script below keeps that weak check so you can observe the gap. It then compares whole arrays with literal expectations. Node’s strict assertion API provides structural equality checks for arrays and objects; use the assertion appropriate to the value you mean to compare. Node.js strict assertion documentation

Another warning sign is an expected result produced by calling the same helper under test. Changing the helper changes both sides of the assertion. A reference algorithm can be valuable, but it must have an independently justified contract and failure modes. For five tags, a literal answer is easier to inspect than a second clever implementation.

Run literal fixtures against good and broken implementations

Save this self-contained example as tag-audit.mjs and run it with Node.js. The first assertion shows that the weak suite accepts an empty result. The good implementation passes the contract suite. Each named mutation is then required to raise an assertion failure; an unexpected pass stops the script. These are illustrative checks, not evidence about an actual candidate or production feature.

The mutation harness catches only AssertionError. That matters: a syntax mistake, a missing import, or an unrelated TypeError would not show that a requirement check detected the intended defect. The script prints a confirmation only after all four deliberate behavioral mistakes have been rejected. Node’s assert.throws supports checking the thrown error class. Node.js strict assertion documentation

import assert from 'node:assert/strict';

function uniqueTags(input) {
  const seen = new Set();
  const output = [];
  for (const raw of input) {
    const tag = raw.trim();
    const identity = tag.toLowerCase();
    if (tag && !seen.has(identity)) {
      seen.add(identity);
      output.push(tag);
    }
  }
  return output;
}

function weakCheck(fn) {
  for (const tag of fn([' Ops ', 'ops', ' API '])) {
    assert.ok(tag.length > 0);
    assert.equal(tag, tag.trim());
  }
}

function contractCheck(fn) {
  const input = [' Ops ', 'ops', '', ' API ', 'api'];
  const before = [...input];
  assert.deepEqual(fn(input), ['Ops', 'API']);
  assert.deepEqual(input, before);
  assert.deepEqual(fn(['z', 'A', 'a']), ['z', 'A']);
  assert.deepEqual(fn(['', '   ']), []);
  assert.deepEqual(fn([]), []);
}

assert.doesNotThrow(() => weakCheck(() => []));
contractCheck(uniqueTags);
const mutants = {
  empty: () => [],
  duplicates: xs => xs.map(x => x.trim()).filter(Boolean),
  sorted: xs => uniqueTags(xs).sort(),
  mutating: xs => { const answer = uniqueTags(xs); xs.pop(); return answer; },
};
for (const [name, mutant] of Object.entries(mutants)) {
  assert.throws(() => contractCheck(mutant), assert.AssertionError, name);
}
console.log('Good implementation passed; four mutants rejected.');

Connect every rejected mutation to a requirement

The empty mutation violates the requirement to return surviving tags. The duplicates mutation keeps repeated identities. The sorted mutation changes encounter order. The mutating version returns the expected values but removes an input element afterward, so only checking the returned array would miss its side effect. Four failures matter because they represent four different mistakes, not because four is a target score.

Now alter the case-preservation behavior yourself: return lowercased display values while keeping deduplication correct. Predict which literal fixture will fail before running it. If you cannot explain the failure, improve the test name and your contract notes before expanding the suite. A useful interview explanation names the wrong observable behavior, not merely the assertion library that noticed it.

Keep execution coverage separate from behavioral evidence

Node can report code coverage when its test runner is invoked with the documented coverage option. That report describes executed code; it does not supply expected answers or prove that the assertions are strong. Node.js test runner documentation The weak loop in this example illustrates the missing reasoning: its properties describe elements only after the function has chosen what to return.

When moving this script into a repository, place each contract check in a clearly named test and run it through the existing runner. Keep mutants temporary or in an explicit audit harness. Do not leave production code deliberately broken, silently mark a failing regression as skipped, or count an expected-failure marker as evidence that the corrected implementation works.

Manual mutations are selective. They can show that particular faults are detected, but they cannot establish completeness. Our examples do not settle locale-specific matching, Unicode normalization, enormous inputs, or error handling for nonstrings. Add those requirements and corresponding independent expectations only when they belong to the actual public contract.

Ask for a challenge to the oracle, not more test names

A useful follow-up prompt supplies the written contract and asks the assistant to find an implementation that passes the current suite while violating one sentence of it. Require the assistant to identify the violated sentence and the smallest new fixture that exposes the gap. Review that fixture yourself before accepting an implementation change.

Keep the production patch and test-oracle review distinct. If a proposed repair forces a changed expected array, ask whether the requirement changed or the test was wrong. In this example, changing ["Ops", "API"] to lowercase output is a product decision, not a harmless cleanup. During a permitted AI-assisted assessment, explaining that distinction is more useful than producing a long test transcript.

Common questions about verifying generated tests

Must every generated test fail first? No. A regression should expose its target defect, and a mutation check should reject its chosen faulty implementation. Ordinary tests for already-correct behavior can pass immediately. The important point is knowing what each check would reject and why.

Are property checks weaker than examples? Not inherently. Properties can cover broad input spaces, but they must constrain the behavior that matters. Checking that every returned tag is valid does not establish that all required tags are returned. Literal examples and properties can complement each other.

How much of this belongs in a timed interview? Start with the smallest independent answer, then challenge one likely mistake and one side effect if the contract includes it. State remaining gaps honestly. This is a way to make limited testing time informative, not a promise of complete correctness.