v1.14.3 ·
The clock that was measuring the wrong thing
A test suite is worth having only if a red run means something broke. This one had two ways to go red that had nothing to do with the code.
The first was a directory fight. One website test ran the blog generator for real, into packages/website/content/blog. Two other tests took a snapshot of that same directory and asserted nothing had changed under them. Nothing kept the three apart, because vitest runs test files in parallel workers. The root pnpm test uses --workspace-concurrency=1, but that separates the two package suites, not the files inside one. So they raced, and one lost on 25 August with no record of what it had seen.
The same test was also doing nothing on every branch except the main one in a real checkout. There the generator refuses to write and exits with an error, and the test threw that result away. It compared two identical reads of untouched files and passed. It had passed that way since it was written.
The second was the wrong clock. Six performance budgets in the knowledge graph's tests were asserted on elapsed milliseconds. Under heavy load one reading went from 3% of its budget to 22% while the work's own processor cost did not move. On an idle machine, two of the six read more than twice their real cost, because over half of what they timed was waiting on the disk.
This matters because pnpm test gates the release script. I had already spent ten days and a blocked release retrying a failure whose output nobody read, because the suite had taught everyone that red means run it again.
What changed
The six budgets now measure processor time, best of three attempts. Every limit is unchanged. Processor time cannot exceed elapsed time for work like this, so no passing limit could start failing. A regression I inserted on purpose still trips, and says 539.8ms CPU against a 500ms budget.
The directory test now builds its own temporary directory, copies the real posts in, leaves one out, and checks the exit status and outcome of every entry. And the whole website suite compares the real directory before and after every run. Any test that writes there fails the run. I checked that by writing there on purpose: the run failed and named the file, though every individual test had passed.
A new test scans both suites for anything timing itself on the wall clock and fails with the file and line. Its first false positive was a comment explaining why a file no longer used the wall clock, so it now skips prose.
Choices
The reports each suggested a small fix. They were a sample. One reported timing check was six, and one reported directory conflict was three files sharing a directory.
I turned down two shortcuts. Telling the runner not to parallelize would hide the race and cost the suite its speed. A separate, non-blocking timing job would swap a check that fails at the wrong times for a check nobody reads.
The generator gained a flag to write elsewhere, but it honours it only in modes that already write nowhere real. Asked to redirect a real release write, it refuses and says why. This script runs in CI, and a flag that accepted any destination would stop being a test seam.
What it does not fix
One limit stays on the wall clock. It times the blog generator, which runs as a separate process, and its processor cost is invisible from the test harness. It kept its limit and gained a deterministic check beside it. The scanner carries one exemption for it, with the reason written down.