v1.14.3 ·
The clock that was measuring the wrong thing
The problem
A test suite is worth having only if a red run means something went wrong. This one had two ways of going red for reasons that had nothing to do with the code.
Two website test files shared the real packages/website/content/blog/
directory. One ran the blog generator into it; two others snapshotted it and
asserted it had not changed. Nothing serialized them, because vitest runs test
files in parallel workers, and the root pnpm test's --workspace-concurrency=1
separates the two package suites rather than the files inside one. So they raced,
and one of them lost on 2026-08-25 with no assertion diff captured. The same test
was also vacuous on every checkout that is not the primary one on main: the
generator refuses to write there, exits 2, and the test discarded its result — so
it compared two identical reads of files nothing had touched.
Separately, six performance budgets in the knowledge graph's suite were asserted on wall-clock elapsed. Under six-times CPU oversubscription those readings inflated by up to 6.9× while the work's own CPU cost stayed flat — one of them went from 3% of its budget to 22% with no change to the code at all. Even on an idle machine the wall figures were over twice the CPU figures, so more than half of what those budgets defended was disk-I/O wait rather than the parser they were written to watch.
Both matter because pnpm test gates scripts/release.sh. This project has
already spent ten days and a blocked release retrying a defect whose output was
never read, precisely because the suite had trained everyone that a failure means
run it again. Every load-dependent assertion pays into that habit, and the habit
is what makes a real regression cost four runs to notice instead of one.
How it could be solved
Both reports arrived with a suggested fix attached, and in both cases the suggestion was smaller than the problem.
The wall-clock report named one check and offered four ways out. Assert on work done rather than on milliseconds. Keep the intent but measure scaling instead of speed — a hundred instances should cost no more than about three times ten. Set an explicit limit where the default one had landed by accident. Or move the timing checks out of the suite entirely, into a separate job that is allowed to be noisy and does not block anything.
The report also said, in a note to itself, do not simply raise the limit — that keeps the coin flip and moves it.
Two of those four turned out to be already done by someone else. The check the report named had been converted to processor time a week earlier by a different fix, and the timeout it named alongside had turned out to be a real defect — a lookup spawning one subprocess per item — rather than a load artefact at all. So the report's own two examples were closed. What it left standing was the category, and the category's largest untouched member was somewhere the report never looked: six limits on the knowledge graph, none of them measuring the code's own cost.
That is worth stating plainly, because it decided the shape of this whole release. A report naming one place is a sample, not the population. Both halves of this release turned out that way: one reported timing check was six, and one reported directory conflict was three files using the same directory at the same time.
Between the four options, measurement chose. Idle, with nothing else running at all, two of the six limits were reading more than twice the work's actual cost — over half of what they defended was the machine waiting for a disk rather than running the code. A scaling ratio would have kept measuring that same mixture, only relative to itself. Processor time takes the waiting out of the number entirely and leaves the part a regression would change. And it is provably safe to adopt: for work like this, processor time can never exceed elapsed time, so no limit that was passing could start failing. Nothing had to be loosened to make the change land, which is the property a weaker fix would have quietly given up.
The fourth option — a separate, non-blocking job — was declined for a reason worth writing down. It trades a check that fails at the wrong times for a check nobody reads. This project already has a receipt for that: a defect that failed in milliseconds survived ten days and a blocked release because the retries' output was never read. Removing a gate to stop it being annoying removes the gate.
One limit was deliberately left on the wall clock. It times the blog generator, which runs as a separate process, and a separate process's processor cost is not visible from the test harness — the measurement available here counts only the parent. There was no substitute to convert it to, so it kept its limit and gained two things instead: a deterministic check beside it, asserting the property the limit was originally bought with, and a written-down exemption in the new scanner that names the reason. An exemption you can read is different from an oversight you cannot.
The directory conflict came with its own suggested fix: point the writing test at a temporary directory, if that is as free as it looks. It was nearly free, and it was not sufficient.
Pointing one test somewhere else fixes one test. The tempting shortcut was cheaper still — tell the runner not to run files in parallel, and the conflict disappears. That was rejected: it hides the conflict rather than removing it, and it costs the whole suite its parallelism to do so. What shipped instead protects the directory itself. The run compares it before and after, so any test that writes there fails, including one written next year by somebody who never read this. The rewritten test also had to be strengthened rather than merely relocated, because it turned out to have been asserting nothing at all: it discarded the generator's result, and on every branch the generator refuses to write and exits with an error. Moving it to a temporary directory without checking the exit status would have produced a test that still proved nothing, in a new location.
The new flag that makes the relocation possible was the one place restraint mattered. A test needs the generator to write somewhere it controls; a release needs the generator to write only into the real content directory, and only from the main branch of a real checkout. So the flag is honoured only in the modes that already write nowhere real, and asked to redirect an actual release write it refuses and says why. This script runs in continuous integration, where a flag that accepted any destination would stop being a test seam and start being a way to write anywhere.
How AIDLC solves it
Two ways the test suite could fail for reasons that had nothing to do with the code it was checking. Both are gone.
The first was a directory two tests were fighting over. The project generates one
blog post per release into packages/website/content/blog, and one test ran that
generator for real — into that exact directory — while two other tests took a
snapshot of the same directory and asserted nothing had changed underneath them.
Nothing kept the three apart, because the test runner deliberately runs different
files at the same time in different processes. So they raced, and one of them lost
on 25 August with no record of what it had seen.
Reproducing it took a throwaway clone of the repository, because the generator
refuses to write into that directory unless it is running on the main branch of a
real checkout — which is exactly the situation the release script creates when it
runs the suite. In the clone, with one post removed, the generator reported
written and the directory changed. The race was real and it was in the release
path.
The same test turned out to be doing nothing everywhere else. On any branch, the generator stops at that write gate and exits with an error — and the test threw that result away without looking at it. So it compared two identical reads of files nothing had touched, and passed. It had been passing that way since it was written.
The fix has three parts. The generator gained a way to write somewhere else, but only in the modes that already write nowhere real; asked to redirect an actual release write it refuses and says why, so it cannot be used to sneak past the branch gate. The test now builds its own directory, copies the real posts into it, deliberately leaves one out, and then checks the exit status and the reported outcome for every single entry — so it fails if the generator skips everything and fails if it overwrites anything. And the whole website suite now compares that directory before and after every run: any test that writes there, by any means, fails the run. That last one was verified by writing there on purpose. The run failed, named the file, and exited non-zero, even though every individual test in it had passed.
The second failure mode was a set of performance limits measured with the wrong clock. Six checks on the knowledge graph asserted that an operation finished within a number of milliseconds of real time. Real time includes every moment the machine spent doing something else. Measured on an eighteen-core laptop under heavy load, one of those readings went from three per cent of its limit to twenty-two per cent, while the work's actual processor cost did not move at all. Even on a completely idle machine, two of the six were reading more than twice their real cost, because over half of what they were timing was waiting for the disk rather than running the code.
All six now measure processor time — the time this process itself spent working —
taking the best of three attempts. Every limit is unchanged, and none of them
could have been made stricter by the change, because processor time can never
exceed elapsed time for work like this. Nothing was loosened to make it pass. A
deliberately inserted regression still trips the limit, and now says so with the
number it measured: 539.8ms CPU against a 500ms budget.
Fixing those six would only have moved the problem, so the class is now checked directly. A new test scans both suites for anything timing itself against the wall clock, and fails naming the file and line. It carries exactly one exemption, written down with its reason: the check that times the blog generator, which runs as a separate process whose processor time this one cannot see. A deterministic check was added next to that one instead, so the timer is no longer the only thing watching what it was bought to watch.
That new scanner caught its own first false positive within a minute of being written — a comment in the file explaining why it no longer uses the wall clock. It now skips prose, and both halves of that distinction are asserted with examples, because a check that flags legitimate code is a check somebody deletes.