AI coding: regression tests for an off-by-one patch

Write the page contract before generating code

Suppose you ask a coding assistant to add a page helper for a list of records. The request sounds small, but terms such as page size leave room for assumptions. Write a contract first: offset is a nonnegative integer index; size is an integer from one to one hundred; a page returns at most size consecutive items beginning at offset. It preserves order and returns an empty list if that starting index is beyond the available items.

For this illustrative exercise, booleans are rejected even though Python permits them in some integer contexts. An empty source is valid, while a zero-sized request is invalid. Those are deliberate API decisions for the example. They should come from the feature requirements in a real repository. Provide them to the assistant explicitly so the generated implementation and its tests are evaluated against the same written specification, rather than whatever behavior the first draft happens to produce.

Reproduce the extra item with a literal expectation

A generated draft might calculate the stop as offset plus size plus one. With the input [10, 20, 30, 40], offset zero, and size two, that expression returns [10, 20, 30]. The requested output is [10, 20]. This single case is small enough to reason about without involving a database, HTTP route, or production dataset. It identifies the disagreement between the requested count and the actual list before asking for another patch.

Python sequence slicing uses an excluded stop index. For a starting index of zero, stopping at two selects indices zero and one. The documentation also describes how slice bounds beyond the sequence length are handled. These language rules support the repair, but the choice of pagination contract remains ours. Avoid changing the test to expect three items simply because that is what the current implementation returns. Python sequence operations documentation

def broken_page(items, offset, size):
    return items[offset:offset + size + 1]

actual = broken_page([10, 20, 30, 40], 0, 2)
print(actual)  # [10, 20, 30]
assert actual == [10, 20]  # Fails: an extra item is returned.

Review a repair against the entire contract

The corrected slice stops at offset plus size. The input checks below make the chosen argument policy explicit, including rejecting booleans and oversized requests. Review those guards separately from the indexing expression: correcting a slice does not automatically establish validation, and adding validation should not quietly change valid page behavior. The function below is complete enough to copy into page_example.py and execute with the test class in the next section.

Keep the request to the assistant bounded: Repair the excluded-stop calculation, preserve input order, and satisfy these argument rules. Do not redesign storage, add dependencies, or change callers without evidence that they need to change. After receiving the patch, compare it line by line with the contract. A short function still deserves review when the assistant sounds confident; the confidence is not evidence that the requested boundaries are correct.

def page(items, offset, size):
    if type(offset) is not int or offset < 0:
        raise ValueError("offset must be a nonnegative integer")
    if type(size) is not int or not 1 <= size <= 100:
        raise ValueError("size must be an integer from 1 to 100")
    return items[offset:offset + size]

Test boundary partitions using independent answers

Append these tests to the corrected helper and run python page_example.py. Each expected list is written literally. None uses the same slicing expression as the helper to calculate the answer. This avoids a circular oracle: if a mistaken stop calculation appears in both production code and the expected result, the assertion can pass while the feature remains wrong. unittest provides equality assertions for those output checks and exception assertions for invalid requests. Python unittest documentation

The first and middle pages check both size and starting position. The short final page protects valid partial results. Offsets at and beyond the end establish empty-result behavior, and an empty input is tested independently. The invalid-argument method uses subtests so each rejected input is identifiable. All seven test methods were executed successfully against the corrected implementation for this article. That is evidence for the listed examples; it does not exhaust every possible list or argument. Python unittest documentation

import unittest


class PageContractTests(unittest.TestCase):
    def test_first_page(self):
        self.assertEqual(page([10, 20, 30, 40], 0, 2), [10, 20])

    def test_middle_page(self):
        self.assertEqual(page([10, 20, 30, 40], 1, 2), [20, 30])

    def test_short_final_page(self):
        self.assertEqual(page([10, 20, 30, 40], 3, 2), [40])

    def test_offset_at_end(self):
        self.assertEqual(page([10, 20], 2, 2), [])

    def test_offset_beyond_end(self):
        self.assertEqual(page([10, 20], 5, 2), [])

    def test_empty_input(self):
        self.assertEqual(page([], 0, 2), [])

    def test_invalid_arguments(self):
        for offset, size in [(-1, 2), (True, 2), (0, 0), (0, 101), (0, True)]:
            with self.subTest(offset=offset, size=size):
                with self.assertRaises(ValueError):
                    page([10, 20], offset, size)


if __name__ == "__main__":
    unittest.main()

Reintroduce the defect to challenge the suite

Passing tests can be uninformative if they never encounter the defect. As a local check, change only the return expression back to items[offset:offset + size + 1] and rerun the same suite. The first-page and middle-page tests fail because they receive three items. The short final page still passes because slicing beyond the available records cannot create an extra item. That distinction explains why an end-of-list example alone would have missed this bug.

The deliberate defect was tested while preparing this article, and exactly those two test methods failed. Restore the corrected expression afterward. This small mutation exercise is useful when reviewing AI-written tests: it asks whether a known wrong implementation survives the suite. You can also consider wrong starting offsets or accidentally reversed order, but do not claim that testing one mutation establishes complete coverage. Choose additional mutations from plausible mistakes in the code you actually have.

• First page: four available items and size two must return exactly [10, 20].

• Middle page: offset one must begin with 20 and return exactly [20, 30].

• Short final page: offset three may return [40], so it cannot reveal the extra-stop defect by itself.

Separate test generation from patch acceptance

Ask the assistant to derive test cases from the written contract before showing it the repaired indexing expression. Request a table of inputs, literal outputs, and the mistake each row would catch. Then inspect that table yourself against the contract and the language semantics. This separation can reduce the temptation to mirror the implementation, although it cannot guarantee independent reasoning from an AI system.

Treat generated tests as code that also needs review. Look for expectations calculated by calling the helper again, cases that use only one available item, and assertions that check only that a list was returned. Those tests may exercise code without challenging the requested behavior. If the existing repository has callers, run their relevant tests as well. This toy helper does not address concurrent data changes, database ordering, cursor pagination, or performance; those require different contracts and evidence.

When the request adds behavior rather than repairing an existing output, the focused sorting-change example checks the contracts around the feature: filtering, input order, record identity, and stable ties.

For a patch that repeats a remote write, expand the test oracle beyond its returned value. The retry and idempotency walkthrough loses a response after storing a task, then checks both the result and the number of stored tasks. A successful retry can otherwise hide a duplicate side effect.

Frequently asked questions

Should every regression test contain a literal answer? No. Larger systems can use independent reference models, invariants, or metamorphic relationships. For a four-item pagination example, explicit lists are clear and easy to audit. The essential requirement is that the expected answer is justified by the contract rather than copied from the production algorithm.

Does seven passing tests mean the AI patch is safe to merge? It means these seven methods passed for this implementation. Review the diff, check affected callers, and run the repository checks appropriate to the change. The illustrative test count is not a quality score or a substitute for understanding the behavior.

Why deliberately break working code? A temporary local mutation checks whether a known defect would be detected. Keep it out of the final patch and restore the correct helper after observing the failures. The result gives a concrete reason to trust a regression test more than its name alone.