AI debugging: measure a performance fix without guessing

State the workload and the output that must survive

Suppose a function filters a dense array of finite integer IDs against an allowed-ID array. Keep matching IDs in their original order, including repeated IDs. Neither input may change. An assistant proposes replacing repeated includes calls with a Set lookup. That is a plausible hypothesis, but it says nothing yet about correctness or the size of a useful improvement.

Start with two questions: how much of the request does this function consume, and what inputs reach it? A slow endpoint might spend most of its time waiting for storage. This exercise measures only an in-memory transformation. Before applying its findings to an application, locate that transformation in an observed profile or request trace.

Reject a faster implementation that changes the answer

For IDs [3, 1, 3, 2] and allowed IDs [3, 2], the required output is [3, 3, 2]. The wrong variant below produces [3, 2] because it builds a Set from the input IDs as well. An improved runtime cannot compensate for losing a required occurrence.

A Set keeps unique values. That property is appropriate for the membership index here, since repeated allowed IDs do not change membership. It is inappropriate for the output sequence, where duplicates are meaningful. The repaired candidate indexes only allowed and filters the original ids array. MDN: Set values, equality, and performance

function baseline(ids, allowed) {
  return ids.filter(id => allowed.includes(id));
}

// Wrong: changes the output contract by removing duplicates.
function wrong(ids, allowed) {
  const lookup = new Set(allowed);
  return [...new Set(ids)].filter(id => lookup.has(id));
}

function candidate(ids, allowed) {
  const lookup = new Set(allowed);
  return ids.filter(id => lookup.has(id));
}

Verify literal answers before comparing timings

Use independent examples for duplicate preservation, empty input, empty allowed IDs, no matches, and repeated allowed IDs. Check the two input arrays afterward. Comparing candidate with baseline on benchmark data is useful, but a shared bug could make them agree. The literal contract cases supply a separate source of expected answers.

The following six tests passed on Node v24.15.0. The final check deliberately confirms that the duplicate-removing mutation fails the literal expectation. Save the functions and tests together in an .mjs file and run node --test on that file before measuring anything.

import assert from 'node:assert/strict';
import test from 'node:test';

const cases = [
  {ids: [3, 1, 3, 2], allowed: [3, 2], expected: [3, 3, 2]},
  {ids: [], allowed: [1], expected: []},
  {ids: [1, 2], allowed: [], expected: []},
  {ids: [1, 2], allowed: [8, 9], expected: []},
  {ids: [2, 1, 2], allowed: [1, 2, 2], expected: [2, 1, 2]},
];
for (const [i, entry] of cases.entries()) {
  test('literal contract case ' + i, () => {
    const ids = [...entry.ids];
    const allowed = [...entry.allowed];
    assert.deepEqual(baseline(ids, allowed), entry.expected);
    assert.deepEqual(candidate(ids, allowed), entry.expected);
    assert.deepEqual(ids, entry.ids);
    assert.deepEqual(allowed, entry.allowed);
  });
}
test('deduplication mutation must fail the literal expectation', () => {
  assert.throws(() => assert.deepEqual(wrong([3, 1, 3, 2], [3, 2]), [3, 3, 2]));
});

Include the work the application must actually perform

The candidate builds its Set inside every invocation. Keep that construction inside the timed region when the real caller supplies a fresh allowed list for each request. Moving construction outside would measure a different operation and could hide a cost that dominates small inputs.

A reusable index may be appropriate when many calls share unchanged allowed IDs. Measure that as a separate scenario, including how the index is built, retained, and refreshed. Reuse introduces memory ownership and freshness questions that this candidate avoids. Do not compare a prebuilt index with a baseline doing all its setup and describe the result as a like-for-like replacement.

MDN documents on-average sublinear Set access and an average membership advantage over comparable includes scans. That supports investigating the change; it does not guarantee constant-time lookup in every implementation or faster complete calls for every input size. MDN: Set values, equality, and performance

Warm up, alternate order, and retain every sample

The harness below is a complete .mjs file. It warms each implementation with 20 untimed calls per scenario, then collects nine rounds. Each round measures repeated calls and divides by that repeat count. Alternating order reduces one simple ordering bias; it does not eliminate runtime noise or prove that warmup is complete.

Input construction and output printing stay outside the timed loop. Set construction remains inside candidate. Summing result lengths consumes the outputs and provides a simple checksum. performance.now supplies a high-resolution millisecond timestamp, so subtraction measures elapsed time within this process. Node v24.15.0: performance measurement APIs

Save raw samples rather than only the best number. The harness prints runtime, engine, platform, architecture, and CPU alongside the data. That context makes a rerun interpretable when a different machine or runtime disagrees. Run separate fresh processes for a stronger comparison; this compact demonstration is not a statistical significance test.

import {performance} from 'node:perf_hooks';
import assert from 'node:assert/strict';
import {cpus} from 'node:os';

function baseline(ids, allowed) {
  return ids.filter(id => allowed.includes(id));
}

// Wrong: changes the output contract by removing duplicates.
function wrong(ids, allowed) {
  const lookup = new Set(allowed);
  return [...new Set(ids)].filter(id => lookup.has(id));
}

function candidate(ids, allowed) {
  const lookup = new Set(allowed);
  return ids.filter(id => lookup.has(id));
}

let checksum = 0;
function sample(fn, ids, allowed, repeats) {
  const start = performance.now();
  for (let i = 0; i < repeats; i++) {
    checksum += fn(ids, allowed).length;
  }
  return (performance.now() - start) / repeats;
}

console.log(JSON.stringify({node: process.version, v8: process.versions.v8,
  platform: process.platform, arch: process.arch, cpu: cpus()[0]?.model}));
for (const size of [16, 2000]) {
  for (const hits of [true, false]) {
    const allowed = Array.from({length: size}, (_, i) => i);
    const ids = Array.from({length: size}, (_, i) => hits ? i : i + size);
    assert.deepEqual(candidate(ids, allowed), baseline(ids, allowed));
    const repeats = size === 16 ? 1000 : 10;
    for (let i = 0; i < 20; i++) {
      baseline(ids, allowed); candidate(ids, allowed);
    }
    const samples = {baseline: [], candidate: []};
    for (let round = 0; round < 9; round++) {
      const order = round % 2 ? ['candidate', 'baseline'] : ['baseline', 'candidate'];
      for (const name of order) {
        const fn = name === 'baseline' ? baseline : candidate;
        samples[name].push(sample(fn, ids, allowed, repeats));
      }
    }
    const median = values => [...values].sort((a, b) => a - b)[4];
    console.log(JSON.stringify({size, hits, repeats, samples,
      medianMs: {baseline: median(samples.baseline), candidate: median(samples.candidate)}}));
  }
}
console.log(JSON.stringify({checksum}));

Inspect the observed results without inventing a speedup

A preparation run used Node v24.15.0 on Windows x64 with an AMD Ryzen 7 9700X. For 16 IDs and 16 allowed values, the median milliseconds per call were approximately 0.000129 for baseline versus 0.000255 for candidate with all hits, and 0.000165 versus 0.000211 with all misses. The candidate was slower in both tiny scenarios.

For 2,000 IDs and 2,000 allowed values, the same run observed approximately 0.864 versus 0.0364 milliseconds per call for all hits, and 1.620 versus 0.0262 for all misses. These are rounded medians of nine batches, with Set construction included. They describe this synthetic run, not an endpoint latency or a promised improvement on another computer.

The checksum was 648000. A changed checksum or failed correctness test should halt the comparison before interpreting timing. Tiny timings are particularly sensitive to batching and runtime effects; keep the raw distribution and repeat the experiment instead of choosing a flattering sample.

Expand the input distribution before accepting the change

All-hit and all-miss inputs are deliberately simple extremes. Add the sizes, hit ratios, duplicate frequencies, and allowed-list lengths that the actual caller sees. A short allowed list with an early match can make a scan cheap; a long list with misses forces different work. Keep deterministic fixtures so another run can reproduce the same workload.

This example limits the contract to finite integers and dense arrays. If the application uses objects, sparse arrays, custom equality, or coercion, review those semantics separately. Also measure memory if a retained Set is under consideration. An elapsed-time comparison alone cannot establish an acceptable memory budget.

Return the measured hypothesis to the real call path

Give the assistant the preserved-output contract, failing duplicate case, complete-call boundary, and raw measurements. Ask for an explanation of why the proposed structure matches that workload. Reject unsupported statements about universal speedups, and check that the final diff leaves Set construction where the experiment measured it.

If the large-input case is common enough to matter, test the candidate through the real caller and measure the end-to-end path again. If requests are dominated by another operation, the local improvement may have little user-visible effect. Preserve the harness and input description so later changes can be assessed against the same question.

Common questions about AI-proposed performance fixes

Is a lower median sufficient? It is one observation. Consider variation, repeated processes, correctness, memory, and the importance of the measured work in the full request. Nine rounds do not supply a confidence interval or prove a production outcome.

Should every call use a Set? No. These measurements show construction can outweigh lookup savings for small arrays. Choose based on representative complete calls and the surrounding contract, rather than a data-structure slogan.

Can AI estimate a speedup before running code? It can suggest a complexity argument and an experiment. Report numerical improvements only from actual measurements with their conditions. The useful result is an evidence-backed decision, including a decision to keep the simpler implementation.