There are two ways to get automated extraction wrong. The first is to trust it blindly — let every value flow straight into your policy admin system regardless of how sure the model was, and quietly ship a mis-read total premium into a bound quote. The second is to trust nothing — route every field to a human for verification, which reproduces the manual process you were trying to escape. Both fail. The right answer sits in between, and it hinges on a single idea: the system should know when it doesn’t know.
That is what a confidence score is for. When extraction produces not just a value but an honest estimate of how certain it is, you can automate the clean majority and route only the genuinely uncertain minority to a reviewer. The human stops doing data entry and starts doing what humans are uniquely good at — judgment on the hard, ambiguous, or garbled cases. This post is about where that line belongs and how it works in practice on real P&C documents.
Confidence is a per-field property, not a per-document one
The most common mistake is treating a whole document as “good” or “needs review.” In reality, a single ACORD 125 can have thirty fields the model is completely sure about and one it is genuinely unsure of. Marking the entire document for review because of one field wastes a reviewer’s time on twenty-nine correct values. Passing the whole document through because most of it is fine ships the one bad value.
InsightXtract scores confidence per field. The named insured might come back at 0.98, the mailing address at 0.96, and the total premium — sitting in a slightly skewed scan with an ambiguous digit — at 0.61. Only that one field needs a human. The rest go straight through. Routing is a decision made value by value, which is what keeps review volume low without letting errors slip.
The decision flow: one threshold, three outcomes
The routing logic is deliberately simple, because simple logic is auditable logic. Each field is compared against a confidence threshold. Above it, the value is accepted and flows straight through. Below it, the value is held and the field is routed to the reviewer queue. A middle band can be configured for “accept but flag” on lower-stakes fields where a soft warning is enough. The threshold itself is tunable by field and by line of business — you can demand near-certainty on total_premium and named_insured while being more permissive on a free-text description.
flowchart TD
F["Extracted field
value + confidence"] --> T{"Confidence vs
field threshold"}
T -- "≥ high" --> A["Accept
straight through"]
T -- "borderline" --> W["Accept & flag
soft warning"]
T -- "< low" --> R["Route to
reviewer queue"]
R --> D{"Reviewer
decision"}
D -- "Approve" --> A2["Value confirmed"]
D -- "Correct" --> C["Corrected value
+ reason captured"]
D -- "Request changes" --> Q["Send back /
re-extract"]
A --> REC["Submission record"]
W --> REC
A2 --> REC
C --> REC
Notice that the reviewer has three actions, not one. They can approve the value as-is (the model was right, it was just cautious), correct it and record why, or request changes — sending it back for re-extraction or flagging it for a colleague. This matters because “low confidence” does not mean “wrong.” Often the model was correct but unsure, and a one-click approve is the entire interaction.
Example 1: a low-confidence total premium on an ACORD
Take a concrete case. A general-liability submission arrives with an ACORD 125 where the annual premium sits in a table cell that was slightly cut off during scanning. The model reads $152,500 but marks it 0.58 because the leading digit is ambiguous — it could plausibly be $152,500 or $132,500. On a bound quote, that $20,000 difference is not a rounding error; it is a pricing error.
Because total_premium is a high-stakes field with a strict threshold, the value is held and routed. The reviewer opens the queue, sees the flagged field with its bounding-box citation pointing to the exact cell, glances at the source page, and confirms it reads $152,500. They approve — or, if the scan genuinely shows $132,500, they correct it and enter a reason: “leading digit misread on skewed scan; verified against broker cover email.” The corrected value, the reason, and the reviewer’s identity are all captured. The interaction took under a minute, and a real pricing error never reached the underwriter.
Example 2: a garbled scanned insured name
Now a messier case. A submission comes in as a faxed, re-scanned PDF, and the named-insured line reads something like SVMMlT L0GISTICS LLC — OCR noise has turned Summit Logistics into gibberish. The model returns the string but scores it 0.44 because it recognizes the character-level uncertainty. This is exactly the kind of value you never want flowing silently into clearance, where it would fail to match the correct account and create a duplicate or a broken renewal link.
Routed to a reviewer, the fix is obvious and fast: they read the source, correct the name to Summit Logistics LLC, and record the reason as “OCR garble on faxed scan.” Critically, that reason code is different from a systematic error — it is noise, not a fixable instruction problem — which keeps it from polluting the signal that drives agent improvement. (The distinction between noise and systematic error is what makes the correction loop trustworthy; more on that in the related post below.)
How to set the line: high stakes, tight threshold
Where you draw the confidence line is a business decision, not a technical one, and it should vary by field. The principle is straightforward: the higher the downstream cost of an error, the tighter the threshold. A mis-read premium or limit is expensive; a mis-read secondary contact phone number is not.
| Field | Stakes if wrong | Threshold posture |
|---|---|---|
total_premium, requested_limits | Pricing / binding error | Strict — route on any doubt |
named_insured, FEIN | Clearance / duplicate risk | Strict — route on any doubt |
effective_date, NAICS | Coverage / classification | Moderate — flag borderline |
business_description, notes | Low — informational | Permissive — accept & flag |
This is what lets a UW-ops leader tune the tradeoff explicitly. Tighten thresholds and more routes to review — higher accuracy, more review minutes. Loosen them and more flows straight through — higher throughput, more reliance on the model. Because the setting is per-field, you can be uncompromising exactly where it matters and efficient everywhere else, rather than accepting one blunt setting for the whole document.
The design principle
Automate on certainty, escalate on doubt. The system should be honest about what it doesn’t know, route those cases to a person, and make the human’s job verification — not transcription. Done well, reviewers touch a small, well-chosen minority of fields and catch the errors that actually matter.
Why this beats both extremes
Full automation without a human gate optimizes for throughput and pays for it in silent errors — the mis-read premium nobody caught. Full manual review optimizes for accuracy and pays for it in cost and cycle time — the very problem you set out to solve. Confidence-based routing is not a compromise between the two; it is strictly better than both, because it spends human attention only where the machine is uncertain. Accuracy stays high because doubtful values are always seen; throughput stays high because certain values are never re-keyed.
The line between AI and human, in other words, is not fixed and it is not a wall. It is a threshold — visible, tunable, and enforced field by field — that moves human effort precisely to where it changes the outcome. That is where AI should stop and a human should step in.
Related: turning those corrections into fewer corrections →
Every correction a reviewer makes is a signal. The companion post covers self-improving extraction — how reasons captured at review time become reviewer-approved, validated configuration changes so the same field stops getting flagged.