Recently I wrote about documentation drifting away from the code it describes. This is the same problem one layer down, and it is the harder one.
Every confidence value a system shows a user is a claim about the future. Sixty-eight percent means that when this system says sixty-eight percent, it should be right about sixty-eight times in a hundred. That is a falsifiable statement. It can be checked. In most production AI systems it never is.
Nothing in a normal pipeline grades a probability
We check whether the model returns a well-formed answer. We check latency, cost, uptime, and whether users complain. We do not check whether the number the system attached to its own answer was honest, because doing so requires waiting for reality to arrive and then going back to grade what we said before it did. Nothing in a normal deployment pipeline does that. The claim is made, displayed, and forgotten inside the same second.
So I built the thing that grades it
Once a week, every prediction my system made is scored against what actually happened, and hit rate is reported per confidence band. Not aggregate accuracy, which hides everything that matters, but band by band: of the calls made at high confidence, how many landed. Of the ones at medium. The distance between what the system claimed and what occurred is the number I care about.
accuracy
Two of my modules failed that test
Not marginally. They showed measured negative skill: outcomes predicted at a given confidence occurred noticeably less often than the confidence implied. The system was not merely imprecise, it was confidently wrong in a consistent direction.
Two details make this uncomfortable rather than academic. Nobody was complaining. And they were among the most-used features in the product.
Overconfidence does not generate a support ticket.
That combination is the whole problem. A user who acts on a seventy percent call and sees it fail does not conclude that the number was dishonest. They conclude they were unlucky, because that is what seventy percent means. The failure mode is invisible from the outside and stays invisible for as long as nobody grades it. This is why the measurement has to exist before the complaint, not after: there will not be a complaint.
A flag, not a deletion
I put both modules behind a flag with a documented closed state and made a deterministic calculation over real underlying data the default. Not deleted, and the distinction matters more than it sounds. Deletion is not reversible. A flag reverses without a deploy, which means the decision stays a decision instead of hardening into an accident. I also bounded displayed probabilities at both ends, because the honest answer is never zero and never one hundred, and a system that shows either is telling you something about itself rather than about the world.
Revenue fell the following month. Retention went up.
I do not present that as a clean trade. It is one product, one month, and I would not build a theory on it. What I will say is that the alternative was continuing to ship a number I had measured and knew to be dishonest, and no framing of growth makes that a defensible position.
The harness lies too, in the other direction
There is a second half to this, and it is the part I would push back on if someone told it to me as a success story. An evaluation harness will also lie to you, and it lies in the opposite direction: it reports a problem before it has the evidence to justify one.
A drift alarm fired on my system out of season. Eighty events in the window, ten of them in the high-confidence bucket. The honest reading of ten events is that they tell you almost nothing. Had I refit the model on that signal I would have taught it the shape of one quiet week rather than the shape of the domain, and I would have done it while believing I was being responsive to data.
I left it alone. Two weeks later, at four hundred events, calibration was healthy.
So the alarms carry minimum sample sizes now, and calibration error is measured against a live baseline rather than an absolute target. The reason for the second choice is that an absolute target is a number you get to pick, and you will pick one you can hit. A baseline is a number the system produced when you were not trying to look good, which makes it the only comparison that can embarrass you.
Knowing that your evaluation is not yet telling you anything is a skill, and it is the one I see missing most often. The instinct to act on every alarm feels like rigour. It is closer to superstition.
The through-line
Documentation is a claim about the system. Confidence is a claim the system makes about itself. Both are asserted with total fluency, both are believed by default, and neither has any mechanism that forces it to stay true. The difference is that a stale sentence in a README waits quietly for someone to read it, while a dishonest probability is being served to users right now, thousands of times a day, each instance carrying the authority of a number.
We have spent a decade building infrastructure to test whether code does what code says. The claims are still unchecked.