The exam that writes itself.

Every time an owner changes their support agent’s behavior — a new tone rule, a stricter refund line, a “reply in the customer’s language” — the system now writes the regression test for that change itself. It reads what the change touches, digs through the workspace’s own past customer questions for the ones most likely to expose a break, invents realistic probes only where the history has nothing to offer, and files the result as golden test cases that re-run on every future change, forever. The owner never authors a test, and never sees a test unless one fails. This is Eval Files 004: how the exam writes itself, and why the generation turned out to be the easy half.

The claim up front: a behavior change without a regression test is a promise you have no way of keeping. The previous Eval Files were about gating the model — certifying it before it talks to a customer, re-running the harness when it drifts. This one is about gating the configuration, because in practice the model changes rarely and the owner’s rules change constantly, and a rule is exactly as capable of breaking your agent as a model swap is.

The problem: owners tune, nobody tests

A support agent under a real business is edited all the time. The owner tightens the refund language in March, adds a language-mirroring rule in April, softens the escalation tone in June. Each edit is made in good faith against the conversation the owner is looking at right now — and each edit can quietly bend behavior the owner locked in months ago. The June softening can sand the edge off the March refund line. Nobody notices, because nobody re-asks the March questions in June.

The traditional answer is a regression suite, and it is a non-answer here: the people tuning these agents are business owners, not engineers, and no owner is going to hand-write test cases after every settings change. The test suite that nobody writes protects nobody. So the requirement inverted: if the owner won’t write the exam, the change itself has to.

Step one: read the change for blast radius

When a change lands, the first pass is extraction — what does this edit actually impact? A rule about refund tone touches refund conversations and nothing else; a language-mirroring rule touches every conversation that isn’t in the workspace’s default language; a new escalation instruction touches the angry tail. This scoping matters because the budget downstream is finite. Testing everything on every change is how a system drowns in its own diligence; testing what the change can plausibly break keeps the exam sharp.

Step two: mine the past before inventing anything

Here is the part we consider the actual idea. Before synthesizing a single artificial question, the system mines the workspace’s own history — the real customer questions this business has already been asked — for the ones relevant to what just changed. A refund-tone change pulls real refund demands, in the customers’ own phrasing, with their own typos and their own mixed languages and their own product names.

The best test cases for your support agent were already written — by your customers. A synthetic probe is a guess about how people talk; a mined one is evidence. Real questions carry all the texture a generator flattens out, and a regression caught on a question a real customer actually sent is a regression that would actually have been seen.

Step three: synthesize only where history is silent

Mining fails in one honest case: the rule governs behavior no past customer ever triggered. An owner whose entire history is English adds “mirror the customer’s language” — there is nothing to mine, and an untested rule is exactly the thing this system exists to prevent. So only there, where history is silent, the system synthesizes: realistic probes shaped to exercise the new rule — Hindi and Hinglish openers for the language-mirroring case — written to sound like the customers this workspace actually has, at roughly 3 cases per changed rule. Synthesis is the fallback, never the default; a generated question has to earn its place by covering ground the real ones can’t.

Golden means permanent

The mined and synthesized cases are persisted as golden test cases — and golden means they outlive the change that created them. When the owner edits anything in June, the cases born from March’s change run again, against the new configuration. That is the whole loop: every change is examined by the accumulated exams of every change before it. Regression protection the owner never had to author, compounding with every edit they make. The suite grows the way the business’s actual rulebook grows, one change at a time, and the protection is always aimed at exactly the behaviors this owner cared enough to configure.

Then came the three tensions, which is where the design earned its keep.

Tension one: never evict what a human wrote

The case set is capped — 25 cases per workspace — because an unbounded suite gets slower and noisier with every change until nobody reads its results. A cap means eviction, and eviction has a bright line: the system may forget its own ideas; it may never forget yours. When the budget is full, only auto-generated cases compete for space. A case an owner wrote by hand — or edited, which is the same act of authorship — is permanent. The asymmetry is deliberate: an auto-mined case is a statistical guess about what matters, but a human-authored case is a direct statement of it, and a system that silently deletes its owner’s stated intent to make room for its own guesses has the priorities exactly backwards.

Tension two: the dead must stay dead

Synthesized candidates don’t enter the suite unexamined — a judge screens them first, and some get rejected as unrealistic or redundant. The subtle failure we hit: rejection wasn’t sticky. The next edit to the same rule would regenerate a near-identical candidate, which would sail toward the judge again, and a case that had been thrown out could resurrect through sheer persistence — the generator only needed to get lucky once. The fix is a seen-set: every candidate ever proposed is remembered, and dedupe runs against everything ever seen, not just the cases currently alive. A test case you rejected yesterday should not be able to reapply under a new haircut. Without that memory, the cap plus regeneration turns the suite into a lottery where rejected cases eventually win.

Tension three: grade the behavior, never the wording

The last tension is the oldest one in LLM testing. A golden case can’t store an expected reply, because the agent will never phrase anything the same way twice — exact-match would fail every honest run. So each case stores an expected behavior — “declines and offers to bring in a teammate,” “replies in the customer’s language,” “states the documented window and nothing more” — and at re-run time a judge reads each fresh transcript, in batches of 5, and answers one question: did the reply do this? Two different sentences can both be the right answer, and the exam has to know that. Grading wording produces a suite that cries wolf until it’s muted; grading behavior produces the only signal worth waking an owner for — the agent no longer does the thing you told it to do.

That’s the file. An owner changes a rule at 11pm; the change mines its own witnesses, invents the few it couldn’t find, and joins a standing exam that every future 11pm edit will have to pass. The owner experiences none of this — just, occasionally, a flag that says a change broke something they decided months ago, caught before a customer ever saw it. The exam writes itself so the owner never has to — and never has to trust an unexamined change again.