Skip to content

v1.26.1 ·

The suite told nine people somebody else broke it

The problem

A test suite is supposed to tell you that you broke something. This one told nine people in three days that somebody else had, and it was wrong every time.

The mechanism was a number. Two tests in this repository walk the real corpus — every instance's requirements, every roadmap item — count something, and compare the count against a literal. One of those literals did not even live in the test. It lived in a table inside a completed instance's design document, and the test read it from there.

So the failure looked like this. You write an amendment to your own requirements, which the framework explicitly instructs you to do whenever a completed phase's artifact needs changing. The suite goes red. The failing test is named after change-management, an instance you have never opened, and the fix is to go and edit a number in its design document.

× the design records counts that re-running the grammar reproduces
  AssertionError: accepted amendment headings across the corpus:
    expected 10 to be 9

That number was hand-maintained four times in three days. Seven to eight when one instance wrote an amendment. Eight to nine when another did. A census in the sibling test moved from twenty-eight to thirty-three and from one to eight, entirely because an unrelated roadmap item landed on the main branch citing seven upstream issue numbers in prose. And a positive control pinned to an item's status directory failed the day that directory was flattened, reporting "the fixture item is gone" while the item sat one directory away, intact.

Not one of those four was caused by a change to the code under test. The parser never moved. The sanitizer never moved.

The third firing is the one that should have settled it. It happened inside the release script, after the version bump, the tag and the blog post already existed — because the suite runs after tagging. So the failure did not prevent a release. It unwound one, and the whole thing had to be run again. A guard whose expected value is a census of live, unrelated data charges its cost to whoever happens to release next.

By then the pattern had a backlog item of its own, and that item had stopped asking to be recorded. Its closing line read: this is now past the point of re-recording. It wants promoting rather than another occurrence note. The fourth firing arrived the next morning.

What made the class worth fixing rather than re-measuring a fifth time is that re-measuring is not a smaller version of the fix. It is the thing that guarantees a fifth firing. Each hand-update restored green while leaving the mechanism exactly as it was, and every one of them was performed by somebody who had not written the code that failed, did not own the document they were editing, and had no reason to look at either.

How it could be solved

The obvious fix is to replace every toBe with a toBeGreaterThanOrEqual and stop thinking about it. That was the first option, it is what the backlog item suggested, and working through the six numbers in one of the two tests is what ruled it out.

Those six were not six measurements. They were three different kinds of claim wearing one shape, and a uniform floor gets two of the three wrong.

Two were invariants. The count of bare @mentions in prose, and of unbalanced HTML comments in prose, are both zero — because a mention surviving into a projected issue body pings a real person, and an unbalanced comment swallows the rest of the body on GitHub. Zero is not a measurement that drifts. It is the property the sanitizer exists to hold, and flooring it would have said "no more than we already leak", which is not the claim.

One was the input. The count of issue references in prose is what the sanitizer has to defuse. If it ever reaches zero the positive control below it is still green and testing nothing at all — the corpus would have lost the material that makes the assertion evidence. That one wants a floor, but for the opposite reason to the others: not because growth is fine, but because shrinkage is the danger.

Three were a census of live, unrelated data. In-code mentions, in-code references, in-code comments. These move whenever anybody writes a roadmap item, and nothing about the sanitizer changes when they do. Floors, straightforwardly.

The near-miss count in the other test was the interesting one, because it is the only genuinely bidirectional number in either file, and a floor on it would have been an actively wrong fix. Near-miss counts headings the amendment grammar warns about instead of accepting. If somebody fixes a malformed heading anywhere in the corpus, near-miss falls and accepted rises in the same edit — so a floor would have failed on a legitimate improvement, which is precisely the behaviour being removed, reintroduced one line down.

It was deleted instead. Not because the property does not matter, but because the property was already asserted, six lines below, by a test that needs no count: nothing Amendment-shaped in the corpus is silently dropped — every heading is either parsed or warned about, never neither. That assertion is the one that has found three real parser defects, it is indifferent to how large the corpus grows, and it is untouched by this release. The equality above it was adding fragility without adding coverage.

Two options were considered and rejected outright.

Delete both corpus walks. They are the reason four of these numbers were ever right, and the walk — not the number — is what caught the parser's near-misses. All three were found by running it over real files rather than by reading the pattern. Removing the walk to stop the noise would have thrown away the only part with a track record.

Regenerate the design document's table on every release. This keeps the number accurate and misses the point entirely. The defect is not that the number goes stale; it is that a stranger has to edit a completed instance's design document to unbreak their own work. Automating the edit would have made the wrong thing reliable.

What shipped instead moves the expectation into the test that asserts it, with the date it was measured, as a floor. And because a floor that can no longer fail is worse than the equality it replaced, five controls came with it, asserting failure in both directions: the floors fail against an empty corpus, and the two prose invariants fail against a body that really does carry a bare mention, a bare reference and an unclosed comment. This project has recorded six occasions where an absence check passed because the query was wrong rather than because the thing was absent, and a floor is an absence check with better manners.

One thing was deliberately left broken. The suite still runs after the tag is created, so a late failure still unwinds a real release — this release removed the most frequent cause of that, not the ordering that makes it expensive. The reason it was not fixed here is that it is not small: the blog post you are reading is generated from the tag, and the suite is ordered after generation so that the build and the tests validate the generated post. Moving the suite means splitting it, or reusing the throwaway tag the blog dry run already creates. That is a release-script decision with its own failure modes, and putting it in the same commit as a test-fragility fix would have bundled two unrelated risks. It is captured as a roadmap item, with the constraint written down, rather than left as a sentence in a record nobody re-reads.

How AIDLC solves it

Nothing that users can observe, and that is the honest headline. No file under packages/cli/src/ changed, no content package changed, and the published CLI behaves exactly as v1.26.0 did. This release exists to record the work in the trail, not to deliver a capability.

What changed is what the test suite asserts. Four assertions across two files compared a live walk of this repository against a number frozen into a completed instance's design document. Any instance that wrote an amendment, and any roadmap item that cited an issue number in prose, made the suite red — under change-management's name, for people who had never touched it. Those numbers were hand-maintained four times in three days, and once the failure arrived after a release had already been tagged, so the whole release unwound.

The repair was not to widen the numbers, which is the action that had already been taken four times. It was to notice that the six-field census was three different kinds of claim wearing one shape, and to give each the form it actually needed. Two of the six were invariants: mentionProse and commentProse are zero because a bare mention or an unbalanced comment surviving into prose is the defect the sanitizer exists to prevent, and zero does not drift, so those stay exact. One was the sanitizer's own input: refProse must stay above zero or the positive control has nothing left to defuse, so it became a floor. Three were a census of live, unrelated data, and they became floors too.

The near-miss count in change-contract.test.ts was removed rather than floored. It is the one genuinely bidirectional number in the set: fixing a malformed heading anywhere in the corpus lowers near-miss and raises accepted in the same edit, so a floor on it would have failed on a legitimate improvement. Its property is already held exactly by the sibling assertion that nothing Amendment-shaped is silently dropped, which needs no count and is the assertion that found three real parser defects. That one is untouched.

The accepted count is now a dated floor declared in the test that asserts it. Nobody edits another instance's design document to unbreak their own work any more. .aidlc/state/change-management/design.md keeps its measurement as the historical record AC-72 required, with one note added saying no test reads it — the table still looked live, and without the note the next red suite would send somebody here to change a number.

Five controls were added, and they assert failure in both directions. The floors fail on an empty corpus, and the prose invariants fail on a body that really carries @octocat, #99 and an unclosed comment. A floor that can no longer fail would be worse than the equality it replaced.