The most honest answer was the one we never logged.

Our support agent has two honest moves when a question has no answer in the knowledge it’s grounded on: stay silent and hand the thread to a human, or say plainly, “I don’t have information on that.” We instrumented the first and not the second. So for a stretch, every time the agent chose the more human form of honesty — saying it, rather than saying nothing — the knowledge-gap ledger recorded nothing at all. The learning loop had a blind spot exactly where the agent was being most honest. This is Post-mortem 002: how a production probe found it, why the bug shaped itself to hide, and what it taught us about instrumenting honesty.

The finding up front: silence and a spoken “I don’t know” are the same signal in different clothes — and we had only wired up one of them.

The loop as designed

Knowledge-gap capture is the flywheel behind the Insights screen, and it is the part of Iris we are proudest of as a system: a question the AI can’t answer from the published knowledge gets logged as a gap; gaps rank by how often they’re asked; the owner writes one article; that question is answered forever after. Ask, miss, log, write, never miss again. The whole self-improving shape of the product hangs on the log step firing every time the AI genuinely didn’t know.

The trigger we built listened for silence. Staying quiet when unsure is a core Iris behavior — we’ve written a whole essay on why a support AI that answers everything is the dangerous kind — so “the agent declined to reply” was the natural, obvious definition of a knowledge miss. No reply went out, therefore the AI didn’t know, therefore log a gap. Clean.

Two kinds of not-knowing

But the agent isn’t a binary gate; it’s a conversational system, and it has more than one way to not know something. Sometimes the grounding check fails hard and the agent stays silent. And sometimes the agent composes a reply about its own ignorance — a polite, complete, honest sentence: “I don’t have information on that — let me bring in a teammate who can confirm.”

From the customer’s side, that second form is strictly better. They were acknowledged instead of ignored; the honesty is spoken instead of implied by absence. It is the answer we’d want a human agent to give. And from the ledger’s side it was invisible: a reply went out, so as far as the capture path could tell, the question was answered. The agent had just said, in as many words, “this is a gap in my knowledge” — and the system whose entire job is collecting gaps in the knowledge filed it under answered.

The more gracefully the agent admitted ignorance, the less the system learned from it. That’s the shape of the bug, and it’s worth staring at: the failure wasn’t in the sloppy path, it was in the exemplary one. Every improvement we made to the agent’s manners widened the blind spot.

The probe that caught it

No unit test found this, and we want to be honest about why none would have. It surfaced in a production e2e probe — one of the standing checks we run against the live system end to end. The probe asks an intentionally un-answerable question: something no published article covers, phrased the way a real customer would phrase it. The expected outcome is the designed behavior — no confident invention, and a new row in the gap ledger.

What came back was an honest, polite no-answer — and an empty ledger. The probe failed, and the failure was the finding: the reply itself was fine, better than fine; it was the assertion on the gap record that tripped. The agent had been honest with the customer and the system had thrown the honesty away.

Unit gates couldn’t have caught it because every component was correct in isolation. The reply path was right to reply — that’s a legitimate, humane way to handle an unknown. The capture path was right to log silence — that was its spec. The bug lived in the seam: in the unexamined assumption that silence was the only costume honesty would ever wear. A bug that lives between two correct components is invisible to every test that checks one component at a time. Only asking the whole production system a real question, and checking both of its outputs — the reply and the ledger — put the seam under load.

The fix: honesty, self-reported

We didn’t touch the reply. The customer should absolutely still get the honest sentence — taking it away to make the logging easier would have been fixing the instrument by breaking the behavior it measures.

Instead, the agent now self-reports. Alongside each reply it produces a small verdict on its own work: did this reply actually answer the question, or is it an “I don’t know” in some phrasing? A reply that is a no-answer — however polite, however fluent — is marked as one, and the capture path treats that mark exactly the way it treats silence: log the gap, count the ask, and let the reply flow to the customer untouched. Two honesty modes, one ledger. The distinction between saying nothing and saying “I have nothing” now exists only where it should — in the customer experience — and not in the learning loop, where it was never anything but noise.

What we keep

Honest behavior needs its own instrumentation. Silence and spoken honesty are the same signal in different clothes, and a capture path keyed to the costume instead of the meaning will lose whichever form it wasn’t written for. The rule we carry forward: instrument failures by what they mean, not by what they look like on the wire — and whenever a behavior has two expressions, assume your logging covers exactly one of them until proven otherwise.

E2E probes against production catch what unit gates can’t. Every component here passed its own tests, because every component was right. The defect was an assumption shared across a seam, and seams only fail under end-to-end load. A handful of standing probes that ask the live system real questions — including questions designed to have no answer — turned out to be worth more than another hundred unit assertions, because they test the sentence the components make together, not the words each knows alone.

The fix made the agent’s honesty feed its learning instead of hiding from it. This is the one we like most. Before, the agent’s best behavior starved the flywheel — the better its manners, the fewer gaps we captured, the fewer articles got written. Now every admitted unknown, silent or spoken, becomes a ranked entry the owner can turn into an article, and the agent’s honesty is the engine of its own improvement. An admission of ignorance is the most valuable sentence a support AI produces — as long as somebody is writing it down.

The probe that caught this still runs, on the same schedule, against production. It has never been more boring, which is the point: it now checks both roads — the silence and the spoken no — and both end at the same ledger. The blind spot cost us some weeks of un-logged gaps and taught us a permanent habit: whenever we teach the agent a new way to be honest, the first question is no longer “does it say the right thing” but “does the system hear it.”