v1.14.4 ·
The refresh that rewrote everything it read
The problem
aidlc knowledge rebuild is documented as rebuilding the SQLite index from the
YAML sources. The index is a derived cache — gitignored, disposable, regenerable
at any time — so the command should have nothing to say to the tracked files it
reads. It rewrote all of them.
Two causes, and the interesting one is not the one that shows up in the diff.
The visible cause is a writer bug. These files are multi-document YAML streams,
and the writer builds one from two kinds of part: documents it re-serializes, and
documents it copies from what it just parsed. The serializer emits no ---
document marker; the copy carries its own. Joining both with a separator doubles
the marker on everything copied, which inserts an empty document between each
pair of real ones. Valid YAML, a different document stream, and — because a
rebuild also stamped a timestamp into every file's header — a 685-line diff
across seven files for a command that changed nothing anybody asked it to change.
The deeper cause is that the timestamp existed at all. _meta.lastIndexed was
written in two places and read in none. Keeping it current is what turned an
index refresh into a whole-graph write, and it bought a value nothing consults.
The knowledge module's own requirements already said this could not happen: the
index is derived entirely from the sources, only the affected file is ever
touched, and exactly one command is allowed to rewrite a whole file. Three
constraints, and a single line in the rebuild path contradicted all of them.
The consequence is not corruption — nothing failed, because the parser tolerates empty documents. It is that a diff stopped being readable. Any real edit to the knowledge graph now arrived buried in a few hundred lines of separator churn, and the only way to know a rebuild meant nothing was to read all of it and confirm. A record nobody can review is not much better than no record.
How it could be solved
The report suggested three ways out and ranked them smallest first: fix the doubled separator and leave everything else alone; or make the rebuild stop writing the sources and add a separate formatting command if reformatting them is actually wanted; or give the whole class of index-disagrees-with-sources a name and a repair in the health-check command.
All three got taken up in some form, and each one shrank on contact.
The separator fix had an obvious shape and a wrong one. The obvious shape is to re-serialize every document, so that everything in the file goes through one code path and comes out the same way. That would have worked, and it would have quietly thrown away every comment anybody had written by hand and requoted every value the serializer would spell differently. Which is the same reformatting noise the whole exercise exists to remove — just moved from one command to another. So the joiner instead adds a separator only where one is missing, and passes the first document through exactly as it found it. That makes it repeatable: run it on its own output and nothing changes.
The second option — a separate formatting command — turned out to be a command that already exists. There is one sanctioned way to rewrite these files wholesale, it already normalizes separators as a side effect of how it works, and it is already documented as the deliberate deletion path. Adding a second one would have meant two commands that rewrite the same files for overlapping reasons, and a user having to know which. So the rebuild simply stopped writing, and the timestamp it was maintaining was deleted rather than moved somewhere safer. That was the easiest call in the release: the value was written in two places and read in none, so there was nothing to preserve. A method whose only job is to rewrite every tracked file, with no callers left, is a trap waiting for the next person who needs something a bit like it.
The third option was a repair in the health-check command, for the state where the index remembers an entity the sources no longer have. That state is real — it is what the original reporter hit, and the workaround was hand-editing files. But the repair also already existed: rebuilding the index throws away everything first and reads the sources fresh, so it fixes exactly this, in exactly this direction. What was missing was not a mechanism. It was a sentence. The failure named the file it could not find the entity in, which reads like a corrupted file, and said nothing about the index being the half that was ahead. So the message now says which half is stale and which command reconciles it, and no new check was added. This project has learned to be suspicious of adding machinery that watches the lifecycle: every new check is a cost paid by all future work, and making an existing one unnecessary is worth more than adding a good one.
Then running the fix against real data changed the answer again. Normalizing this repository's own two damaged files should have deleted a hundred and eight lines and touched nothing else. It rewrote eight hundred and twenty-nine. The copy-the-other-documents branch was not copying them at all — it was re-serializing them at whichever line width happened to be in force, which was not the width they had been written at, so every long description in the file got rewrapped. The doubled separator was a seventh of the diff it produced and had taken all of the attention.
The near-miss is worth naming, because it looks like a fix. Set the width to match what the writers use, and the two agree; the reformatting stops for anything written since. It still reflows every document written at any other width, which is precisely what a file with history is full of. Copying the exact bytes out of the file, using the position the parser already recorded, reformats nothing ever — and it is fewer lines than the version that gets it nearly right. That distinction is the one thing here a test can prove rather than assert: the same test that passes now fails against the nearly-right version.
How AIDLC solves it
aidlc knowledge rebuild no longer rewrites the files it reads. It refreshes the
search index and nothing else, which is what its one-line description always
claimed and what the knowledge module's own requirements always said: the index is
a throwaway cache, only the file being changed is ever written, and exactly one
command is permitted to rewrite a whole file. A rebuild used to stamp a timestamp
into all seven, and the stamping was done by a writer that doubled the document
separator on everything it copied — so a command that changes nothing produced a
685-line diff and left an empty document between each pair of real ones.
Three things changed to make that true.
The writer now assembles a multi-document file by adding a separator only where one is missing, instead of adding one unconditionally to parts that already carry theirs. The two writers that were never affected — the whole-file rewriter and the merge driver — go through the same helper and produce byte-for-byte the same output as before, which is asserted rather than assumed.
An in-place update copies every document it is not changing straight out of the file, using the position the parser already recorded, rather than putting it back through the serializer. This was the part the original report did not contain and the part that mattered most. Re-serializing reflows long text to whatever line width is in force, and these files were written at two different widths over several weeks, so the branch meant to preserve documents was rewrapping all of them: on this repository's own files, 829 changed lines where the doubled separator accounted for 108. Copying the exact bytes reformats nothing, ever.
The timestamp that motivated the whole write is gone, along with the method that maintained it. It was written in two places and read in none, so keeping it accurate was buying nothing and costing a write to every tracked file on every rebuild. The field stays in the file format, because seven committed files carry it, documented as recording when the file was created and not maintained after.
Two smaller changes came out of the same incident. When the index holds an entity its source file does not — which is what happens after somebody reverts a source file, and is how the original reporter got stuck — the failure now says which half is stale and which command reconciles it, instead of naming a file and leaving the reader to infer corruption. And this repository's own graph was carrying 108 empty documents in two committed files, put there by a rebuild weeks ago; the earlier workaround restored the damaged committed state rather than removing it. Those are normalized here, verified by comparing the parsed contents before and after rather than by reading the diff, which is 108 deletions and no changed content.
No new command and no new health check were added. Both were on the table. The command that normalizes these files already exists and already does this; the command that reconciles the index to the sources already exists and already does this. What was missing was a message, and a message is cheaper than a gate that every future release has to pass.