The Iris Lab
Learning through building, with receipts.
The engineering publication behind Iris — eval files, post-mortems, and the real cost math of building AI-first support. First-person, numbers-forward, and honest about what broke. Written by the team building it.
The exam that writes itself.
When an owner changes how their support agent behaves, nobody writes a regression test — so we made the system write it. It reads the change for blast radius, mines the workspace’s own past customer questions for the ones most likely to break, synthesizes realistic probes only where history is silent, and files the result as permanent golden cases that re-run on every future change. The hard part wasn’t the generation — it was three design tensions: a bounded budget that never evicts anything a human wrote, a seen-set so rejected cases stay dead, and a judge that grades behavior, never wording.
Read the write-up →The lab notebook
The most honest answer was the one we never logged.
Our knowledge-gap capture listened for the agent staying silent on questions it couldn’t ground. But when the agent instead replied with a polite, honest “I don’t have information on that” — the purest knowledge gap there is — nothing was logged. The learning loop was blind exactly where the agent was being most honest. A production e2e probe caught it: an intentionally unanswerable question that drew an honest no-answer instead of silence, and an empty ledger. The fix makes the agent self-report when a reply is an “I don’t know,” so spoken honesty feeds the same ledger silence does.
July 6, 2026Certifying a model before it talks to a customer.
A benchmark score can tell you a model is smart. It cannot tell you the model won’t invent your refund policy, impersonate a human, or “issue” a refund on its own — because the benchmark never asked. So before any model is allowed near customer traffic, it passes a fixed, private harness of owner-style conversations and tool-calling scenarios, graded blind on five failure modes, conversation and hands scored apart. Model swaps ship on a pass, never a benchmark. Here is the method — and why we don’t publish the test.
July 6, 2026The Indic model file: what months of running an Indian-language model taught us.
We evaluated and ran an India-built language model — Sarvam — extensively against real support conversations, and kept a file on it. The honest dossier: its self-reported confidence is worthless (a fluent non-sequitur arrived at 0.9), its Hindi has a default gender nobody chose, and two of the three worst bugs we chased were our prompt, not the model. It is also warmer in Hinglish than anything frontier we tried, and roughly one-thirtieth the price — which changes which ideas are affordable. A capability file, not a review.
July 6, 2026We filmed a UI bug frame-by-frame to catch it lying.
A chat panel flashed when you opened it — a blink you could feel but never point at. The layout-shift meter swore nothing moved. So we recorded the bug at sixty frames a second, caught a single skeleton frame in the stack, and followed it home to a cache that wiped itself whenever a quiet poll returned an empty page. A post-mortem about instruments that lie in different ways.
July 6, 2026WhatsApp-grade images on a chat widget, with zero image servers.
A 12-megapixel phone photo is 4–12 MB. It has no business crossing the wire at that size, and it never does — a few lines of canvas code re-encode it to 250–400 KB before it leaves the browser, strip the GPS metadata as a privacy bonus, and hand a tiny WebP to an object store that caches it forever. The whole image stack: no resize server, no thumbnail worker, no transform CDN. Here are the numbers.
July 6, 2026Self-hosting the observability stack: the real math.
LLM tracing is priced per unit — every trace, span and score is a billable event — and at real volume that meter climbs fast. We ran the numbers on self-hosting the open-source option (Langfuse) instead: a ~$731 cloud month against a flat $100–175 to run it yourself, a 4–6× crossover. But the honest decider isn't the bill. It's that your traces are full of customer conversations — and that self-hosting hands you ClickHouse to run at 3am.
July 6, 2026We tested two frontier models on a real customer-facing support agent. Here’s who lied.
We ran GPT-4.1 and GLM-5.2 through the same support-agent system prompt, the same six owner-style conversations, and three tool-calling scenarios — grading identity honesty, refusal of abusable requests, never inventing policy, grounded answers, and safe refund handling. The surprising part: the cheaper open model was more honest about what it didn’t know, and more dangerous until we guardrailed it.