The Indic model file: what months of running an Indian-language model taught us.

For a stretch of months we ran an Indian-language model through everything we could throw at it — a fixed evaluation harness, shadow tests against real support conversations, and long evenings reading its output line by line against what a human agent would have said. The model is Sarvam, an India-built LLM tuned for Indic languages (Sarvam AI), and this is the file we kept: what it is genuinely good at, where it quietly fails, and the two or three lessons that cost us the most to learn. We are naming it because the specifics are more useful than coyness, and because everything here is a property you would rediscover yourself if you ran the same tests. This is a capability dossier, not a review — and to be exact about it, this is a model we evaluated and ran extensively, not a claim about any particular thing that talks to customers today.

The finding up front, before the detail: the model is warmer and more natural in Hinglish than any frontier model we tried — and almost everything that went wrong was either our prompt or a number the model made up about itself. Two of the three worst bugs we chased were not the model’s fault at all. The third was, and its fix looks nothing like the fix for the other two. Telling those apart turned out to be the entire skill.

Its confidence score is worthless

Start with the lesson that saved us from the most embarrassing mistakes: do not trust a model’s report of its own confidence. Like most models, this one can be coaxed into emitting a self-assessed confidence alongside an answer, and the intuition — the hope — is that a low number flags the shaky answers so you can gate on it. It does not. The confidence is uncalibrated: it is a fluent number the model generates about a fluent answer the model generated, and the two are barely correlated.

The single observation that broke the intuition for good: a fluent, grammatical non-sequitur — an answer that was confidently, completely off-topic — arrived at confidence 0.9. Nothing in the number warned us; the sentence read like every correct sentence. Wrong answers cluster at high confidence just as comfortably as right ones. A model’s confidence score is a number it made up about a number it made up — treat it as prose, not as a signal, and gate quality on correctness measured against ground truth instead. Everywhere we had leaned on self-reported confidence as a safety valve, we ripped it out and replaced it with an external check.

Its Hindi has a default gender

The second lesson is subtler and, in production, more embarrassing. Hindi verbs agree with the grammatical gender of the speaker — “main karti hoon” if the speaker is feminine, “main karta hoon” if masculine — and the model has a default. Left to itself, with no gender pinned, its Hindi replies drifted toward feminine verb endings regardless of the persona it was supposed to be wearing. A support agent configured to sound like a specific brand voice would quietly misgender itself in every other sentence, fluently, without any error to catch.

The model had a default gender, and nobody chose it. This is not a bug you can grade with an English rubric or a benchmark; you only see it if a Hindi speaker reads the transcripts. The fix is boring and mandatory: pin the verb gender explicitly in the prompt, or — better where the brand voice allows — steer toward gender-neutral phrasing that sidesteps the agreement entirely. The general lesson travels beyond Hindi: a model’s “neutral” default is never neutral; it is just a choice you didn’t make on purpose, and in a gendered language that unmade choice is legible to every native reader.

Half the failures were our prompt wearing the model’s face

Now the lesson that reframed everything else. When an answer came back bad, our reflex was to blame the model — it is the obvious suspect. But when we started stripping the prompt back to nothing and re-asking, a startling share of the failures evaporated. Half the failures we blamed on the model were bugs in our prompt wearing the model’s face.

The sharpest example: a sales-primed prompt measurably hurt the model’s comprehension of casual Hinglish. When we framed the system prompt to be persuasive and conversion-minded, the model’s reading of ordinary customer phrasing got worse — at one point it misread “shukriya,” a plain thank-you, because the sales framing had it hunting for objections to overcome instead of just listening. Strip the framing back to a neutral, minimal instruction and the same model understood the same message perfectly. The model was better than its live output suggested; the prompt had been standing on its comprehension.

This is why the distinction in that figure matters more than any single fix. A prompt-induced failure and a model-ceiling failure look identical on screen — a wrong answer is a wrong answer — but they have opposite, non-interchangeable remedies. The prompt-induced one is free to fix and yours to own; the model-ceiling one needs a heavier, costlier intervention. Reach for the wrong fix and you either pay for a bigger model to paper over your own prompt bug, or you endlessly re-tune a prompt against a wall the model was never going to clear. Before you blame the model, ask it again with the framing stripped away — the answer to “whose bug is this” changes what you should do next.

The economics change which ideas are affordable

Then the number that reframes the whole exercise. A value-tier Indic model like this one runs at roughly one-thirtieth of frontier pricing — its published rates sit in the low single-digit rupees per million input tokens against the frontier’s couple-of-dollars (Sarvam pricing; per-token comparison, 2026). That is not a rounding-error saving; it is a different order of magnitude, and order-of-magnitude cost changes are qualitative, not quantitative.

A value-tier model at one-thirtieth the price doesn’t just cut your bill — it changes which ideas are affordable. Things you would never dream of doing with a frontier model per message — classifying every inbound in two languages, drafting a reply the human may never send, re-checking an answer with a second pass — stop being extravagances and become defaults, because the unit cost fell through the floor. The interesting question a cheap-but-capable model poses is not “how much do we save on the current workload” but “what did we refuse to build because the tokens were too expensive.” That is where the leverage actually lives.

Where it shines, and where you need a belt

So the honest ledger. Where it genuinely shines: natural Hinglish warmth and cultural register. Code-mixed Hindi-English, the actual texture of how Indians message support, the small registers of politeness — it lands these more naturally than the frontier models we compared it against, which tend to produce Hindi that is correct and slightly foreign, like a very good textbook. For the conversational surface of Indian support, that warmth is a real, felt edge.

Where it needs a belt: long tool chains. On extended, multi-step reasoning where the model has to call a tool, read the result, call another, and keep the thread coherent across many hops, it was more likely to lose the plot — to not converge, to wander, to drop a constraint it was holding three steps ago. This is a capability-ceiling failure, not a prompt bug, and the fix is architectural, not textual. The pattern we settled on is generic and worth stating plainly: run the cheap, warm model as the default, detect non-convergence, and retry the hard cases on a stronger model. A fallback belt. The cheap model handles the common case it is genuinely good at, and the expensive model catches the long tail it isn’t — so you pay frontier prices only for the fraction of conversations that actually need frontier reasoning. The cheapest capable model for the easy nine-in-ten, a stronger one for the tenth, and a detector deciding which is which.

That belt-and-default shape is, in the end, the whole file compressed to a sentence. A model is not a single verdict — good or bad, in or out — but a map of where it is strong, where it is thin, and what has to sit around it to make the strengths safe to use. The warmth is real and worth having; the uncalibrated confidence, the unchosen gender, and the wandering on long chains are real and worth engineering around. Running a model in production is mostly the discipline of knowing which is which — and building the plain, unglamorous scaffolding that lets the good parts show and quietly catches the rest before a customer ever sees it.