Documentation Is a Claim. Code Is Evidence.

Code has a mechanism for staying honest. Documentation does not.

Code that is wrong eventually says so. It fails, tests break, alarms fire, a user complains. Being wrong has a cost that arrives on its own schedule and forces a correction.

A sentence about code has no such mechanism. It is written once, at the moment of maximum understanding, and then never asked to earn that status again. It sits in the repository, formatted and confident, while the thing it describes moves underneath it. The only event that could contradict it is a person reading the sentence and the implementation on the same day and noticing they disagree. In most organisations that is nobody's job. In a one-person team it is nobody's job by definition.

I ran the audit on the cleanest codebase I had

I wanted to know how large that gap actually is, so I ran the audit on my own two products: every technical claim in the documentation, checked by hand against the code that was supposed to implement it. Not a spot check. Every sentence that asserted something a reader could rely on.

I chose my own codebase deliberately, and not because I expected it to look bad. I chose it because it is close to the best case available. One author. Written in the last eighteen months. Documented on purpose rather than under pressure. No inherited decisions, no departed engineers, no comments describing a system that was replaced before anyone joined. If documentation drift were mainly a problem of scale, turnover and neglect, this codebase should have been clean.

Six claims came back wrong.

Two were behavioural. The documentation described the rollback as automatic, triggered by accuracy floors. Detection was automated. The response was not, and never had been: the rollback was performed by a human from a runbook. Separately, a package described itself as running on a schedule. There was no schedule. Nothing invoked it.

Two were historical. A feature gate removed eight months earlier was still documented as active. Two modules documented as deleted had actually been placed behind a flag, which was the better engineering decision and the worse sentence, and the difference between the two was not academic: one is reversible without a deploy and the other is not.

One was regulatory. Both products carried a transparency obligation under the EU AI Act that the documentation did not reflect, which is the category where a stale sentence stops being an inconvenience and becomes exposure.

And one had propagated. It had left the repository and reached downstream documents that other people read, which is the property that makes this different from an ordinary stale README. Claims do not stay where they are written. They get copied into decks, specifications, contracts and answers given in meetings, and every copy carries the authority of the original without any of its context.

None of these were lies. Every one of them was true on the day it was written.

The ratio moved

Here is what has changed recently and why I think this gets worse rather than better. AI made writing code dramatically faster. It did not make verifying code faster in the same proportion. Generation got a step change; verification got autocomplete. More changes land per week, and every change is an opportunity for a sentence somewhere to quietly stop being true. This is not a discipline problem that more discipline solves. The ratio moved.

The framing I ended up with is this. Documentation is a claim. Code is evidence. We have built serious infrastructure for testing whether code does what code says, and almost nothing for testing whether code does what prose says. The claim and the evidence live in the same repository, arrive in the same pull requests, and are never once compared.

Writing less is not the fix

The common advice is to write less documentation so there is less to go stale. That trades one failure for another. Instead of confident wrong sentences you get no sentences, and the system's behaviour lives in one person's head until that person leaves. The problem is not that claims exist. It is that they are unverified, and that nothing in the toolchain treats them as claims at all.

I ran that audit by hand. It took days on the easiest possible case and it found six things.

Then I started building the thing that does it.