Looks right, but isn’t: the silent-error problem in AI extraction

Ask anyone who has run an AI-extraction pilot in insurance intake what went wrong, and you’ll rarely hear about the document that failed to parse. Those are easy — they throw an error, they land in a queue, someone looks. The stories that stay with people are different: the submission where the each-occurrence limit was read as $1,000,000 instead of $10,000,000 and quoted that way; the renewal bound to the wrong effective date; the loss run where a sub-limit was picked up as the aggregate. None of these looked wrong. Every one of them was a clean, confident, well-formatted value. That is the silent-error problem, and it is the single most important thing to understand before you trust an extraction system with underwriting decisions.

An obvious error announces itself. A silent error is a plausible wrong answer — syntactically valid, correctly typed, in the expected range — that no downstream check questions because nothing about it looks off. It is the most dangerous class of error precisely because the system’s job is to reduce human review, and a value that looks right is exactly the value a human reviewer skims past.

Why fluent models produce confident wrong answers

Traditional OCR and template-based extraction failed loudly. When a field didn’t match the template, you got a blank, a garbled string, or a hard error — visibly broken, easy to route for review. Large language models changed the failure mode. An LLM will almost always return a well-formed answer, because generating fluent, plausible text is what it does. When the source is ambiguous, low-quality, or simply doesn’t contain the value, the model doesn’t stop — it produces its best guess, formatted exactly like a correct answer. The output’s polish is uncorrelated with its truth.

In insurance documents the conditions that produce silent errors are everywhere:

• Structural ambiguity. A dec page lists per-occurrence, aggregate, sub-limits, and SIR in a dense block. Which number is “the limit” depends on context the model may not fully resolve — and a sub-limit read as the aggregate is a perfectly plausible number.
• Look-alike fields. Effective date vs. expiration date vs. quote date vs. loss-run valuation date — four dates in the same format, one field, easy to swap.
• Multi-document packets. The same insured’s name and limits appear on the ACORD, the broker email, and the expiring policy. Pull the revenue figure from last year’s workbook tab instead of this year’s and you get a real number from the wrong source.
• Transposition and unit slips. $1,250,000 vs. $12,500,000; a limit stated in thousands on one schedule and dollars on another; a comma read as a decimal.
• Missing data filled in. When a field genuinely isn’t in the document, the safest answer is “not present” — but a model optimized to be helpful may infer a value that fits.

The anatomy of a silent error

These aren’t random. They cluster into recognizable types, each with a characteristic cause — and knowing the type is the first step to catching it.

Type of silent error	Trigger	P&C example
Transposition / magnitude	Formatting	Each-occurrence limit read as $1M instead of $10M; premium off by a factor of ten from a thousands-vs-dollars mismatch.
Wrong field, right type	Look-alike	Expiration date captured into the effective-date field; both are valid dates, so nothing flags it.
Sub-limit as the limit	Structural	A products/completed-ops sub-limit picked up as the general aggregate; a defense-inside-limits value read as the full limit.
Wrong-source value	Multi-doc	Revenue pulled from the prior-year tab; insured name taken from the broker email when the ACORD is authoritative.
Inferred / hallucinated fill	Missing data	A NAICS code or attachment point invented to complete the record when the document never stated it.
Stale or mis-dated value	Temporal	A loss run’s prior valuation date used as current; open reserves read from an out-of-date snapshot.

Why a single confidence number doesn’t save you

The instinctive fix is “have the model tell us how confident it is, and review anything below a threshold.” It sounds right. In practice a bare numeric confidence — confidence: 0.87 — is close to useless for catching silent errors, for three reasons.

First, the number is not calibrated. An LLM’s self-reported confidence is itself a fluent generation, not a measured probability. A 0.87 on one field and a 0.87 on another may reflect wildly different actual error rates. Worse, models are frequently most confident exactly when they’re wrong in a plausible way — the transposed limit gets a high score because the answer “looks” clean to the model too.

Second, a scalar collapses different problems into one axis. “The value is clearly printed but I’m unsure I read the right field” and “the value isn’t in the document so I guessed” are completely different risks that demand different actions — but they can produce the same 0.87. A single number tells you how much to worry, never what to worry about.

Third, it isn’t actionable. A reviewer handed “limit: $1,000,000 (0.87)” has nothing to check against. Check it against what? The number offers no path to verification — no page, no snippet, no reason. It moves the guess from the model to the human without giving the human anything to work with.

What actually catches silent errors

The goal is not a model that never errs — no extraction system, human or machine, achieves that. The goal is a system that surfaces its own uncertainty in a way you can act on, so that wrong values get caught before they reach a decision. Four mitigations do the real work, and they compound.

1. Evidence and citations back to the source

Every extracted value should carry a pointer to where it came from — the document, the page, and ideally the exact text span or cell. This is the single highest-leverage change, because it converts an unverifiable claim into a checkable one. “Each-occurrence limit: $10,000,000” is a claim; “Each-occurrence limit: $10,000,000, from Umbrella Dec, p.2: ‘Each Occurrence Limit $10,000,000’” is evidence. A reviewer confirms it in seconds, and a wrong-source or hallucinated value becomes obvious because the citation either doesn’t support the number or doesn’t exist.

2. A confidence taxonomy, not a lone number

Replace the scalar with a small set of categories that describe how the value was obtained — which tells the reviewer what kind of check to run. A workable taxonomy for insurance extraction:

Confidence category	What it means	Default routing
Exact	The value is stated verbatim in the source and the citation matches it directly.	Auto-accept
Implied	Clearly supported by the source but reformatted or normalized (e.g. “10M” → $10,000,000).	Auto-accept with note
Inferred	Derived by reasoning across fields or documents, not stated outright.	Review recommended
Uncertain	Ambiguous source, conflicting values, or the field may not be present at all.	Human review required

The difference from a bare number is that each category maps to an action. “Inferred” tells the reviewer to check the derivation; “uncertain” tells them the source itself is the problem. A scalar 0.6 tells them neither. And unlike a self-reported probability, a taxonomy grounded in how the value was obtained is far harder for a fluent model to fake — either there’s a verbatim citation or there isn’t.

3. Self-checking and validation

The best defense against a plausible wrong value is a second, independent check that the value must survive. This is where the system audits itself before a human ever sees the record:

• Cross-field invariants. Does the attachment point plus underlying limits line up with the requested excess layer? Does per-claim incurred roll up to the stated total incurred? Is the effective date before the expiration date? Do the loss-run yearly totals sum correctly? A transposed or wrong-source number usually breaks one of these relationships even when it looks fine in isolation.
• Reference-data validation. Check codes and coded values against authoritative tables — NAICS/SIC/ISO GL class codes, state and country codes, currency codes, carrier/AM Best identifiers. A hallucinated class code fails the lookup.
• Range and format rules. Limits, premiums, and rates that fall outside plausible bands for the class get flagged rather than accepted.
• Targeted re-extraction. When a field fails a check, re-extract just that field — a fresh, focused pass on the specific value, or a cross-read against another document in the packet — rather than trusting the first pass or discarding the whole record.

Cross-checks turn the model’s fluency against itself: a wrong value has to be plausible in every relationship at once, which is a far higher bar than looking clean on its own.

4. Routing uncertain values to humans

The point of the first three mitigations is to make the fourth selective. Instead of reviewing everything (too slow) or nothing (too risky), you route the small set of values the system itself flags — the inferred, the uncertain, the ones that failed a cross-check. A human sees the value, its citation, its category, and the specific rule it violated, and resolves it in seconds. Reviewer attention goes where the risk actually is, and the correction becomes a signal the system can learn from.

Every value earns its confidence category by surviving validation — only the exact and implied auto-accept; everything uncertain routes to a human with its evidence attached.

Why it matters to the business

Silent errors are a governance problem, not just a data-quality one. Left unmanaged, they erode exactly the trust that makes automation worth having:

• Priced risk you didn’t intend. A wrong limit, attachment, or exposure base flows straight into rating. The cost of one silent error on a large account can dwarf the efficiency gains from the whole intake program.
• False confidence scales the damage. The more you trust an unverified system, the less you review — so the better the automation looks, the more silent errors slip through. Verifiability has to scale with automation, not lag it.
• Audit and reserving integrity. Numbers that reach the loss-cost model, the reserving actuary, or a regulator need provenance. “The AI said so” is not an audit trail; “here is the source page for every value” is.
• Reviewer trust is the real adoption gate. Underwriters abandon tools that make confident mistakes. A system that says “I’m sure about these 46 fields and here’s why, and I need your eyes on these 4” earns the trust that gets it used.

The uncomfortable truth is that any fluent extraction system will produce plausible wrong values — the question is whether the system is honest about which ones. A product that returns a flat record and a single accuracy stat is asking you to trust it blindly. A product that cites every value to its source, labels each with a confidence category that reflects how it was obtained, validates against cross-field invariants and reference data, and routes only the genuinely uncertain values to a human is doing the opposite: it is showing you where it might be wrong so you never have to find out the hard way. In P&C underwriting, where one transposed limit can outweigh a year of efficiency, that difference — between looking right and being verifiably right — is the whole game.