v1.26.1 ·
The suite told nine people somebody else broke it
A test suite is meant to tell you that you broke something. This one told nine people in three days that somebody else had, and it was wrong every time.
The cause was a number. Two tests walk the real corpus, every instance's requirements and every roadmap item, count something and compare the count to a literal. One literal did not even live in the test. It sat in a table in a completed instance's design document, and the test read it from there.
So you write an amendment to your own requirements, which the framework tells you to do. The suite goes red, naming change-management, an instance you never opened, and the fix is to edit a number in its design document.
× the design records counts that re-running the grammar reproduces
AssertionError: accepted amendment headings across the corpus:
expected 10 to be 9
That number was updated by hand four times in three days. Seven to eight, eight to nine. A census in the sibling test went from 28 to 33 and from 1 to 8 because an unrelated roadmap item cited seven upstream issue numbers in prose. A positive control pinned to a status directory broke the day that directory was flattened. None of the four came from a change to the code under test.
The third one fired inside the release script, after the version bump, the tag and the blog post already existed, because the suite runs after tagging. It did not prevent a release. It unwound one. The backlog item for it had already said it wanted promoting, not another note. The fourth firing came the next morning.
What changed
Not every toBe became a floor. The six numbers in one test were three kinds of claim in one shape.
Two were invariants: bare mentions and unbalanced comments in prose are zero, because either one surviving into a projected issue body pings a real person or swallows the rest of the body. Zero does not drift, so those stay exact. One was the sanitizer's input, the count of issue references. If it reached zero, the positive control would pass while testing nothing, so it became a floor. Three were a census of live, unrelated data, so they became floors too.
The near-miss count in the other test I deleted. It is the one number that moves both ways: fixing a malformed heading lowers near-miss and raises accepted in one edit, so a floor would fail on an improvement. The property is already held, six lines below, by a check that nothing amendment-shaped is silently dropped. That one found three real parser defects, and it needs no count.
The accepted count is now a dated floor in the test that asserts it. The design document keeps its old measurement as history, with a note that no test reads it.
Five controls came with it and fail in both directions. The floors fail on an empty corpus. The prose invariants fail on a body carrying @octocat, #99 and an unclosed comment. A floor that cannot fail is worse than the equality it replaced.
What it does not fix
Nothing a user can see changed. No file under packages/cli/src/ moved, and the CLI behaves as v1.26.0 did.
The suite still runs after the tag, so a late failure can still unwind a release. I removed the most frequent cause, not the ordering. The blog post is generated from the tag, and the suite runs after generation so that the build validates the post. Moving it means splitting the suite or reusing the throwaway tag the blog dry run creates, a release-script decision with its own risks. It is a roadmap item now.
I rejected deleting the corpus walks, which caught the parser's near-misses, and regenerating the design table each release, which makes the wrong thing reliable.