“How did it decide that?” — the audit question that stalls AI pilots in insurance

There is a well-worn pattern in insurance AI: a promising pilot, a polished demo, and then quiet death. The reason is rarely that the model was wrong. It is that no one could explain, to the standard a regulated business requires, why the model produced the answer it did — or reconstruct that reasoning six months later when an examiner, an auditor, or a plaintiff’s attorney asks. In a line of business governed by rate filings, market-conduct exams, and fair-treatment obligations, “the AI said so” is not a defensible position. It is a liability.

The uncomfortable truth is that accuracy and auditability are different problems. A system can be right most of the time and still be unshippable, because a regulated carrier cannot put a decision it cannot explain into a process that affects pricing, coverage, or claims. The pilots that ship are the ones that treat explainability, audit trails, and governance as first-class requirements — not as features to add after the model works. Below, the real blockers, and what “good” actually looks like.

Why “trust me, it’s 95% accurate” fails an auditor

A headline accuracy number feels reassuring in a demo and means almost nothing in an audit. An auditor is not asking whether the system is usually right. They are asking whether this specific decision, on this specific document, on this specific date, can be reconstructed and justified. Aggregate accuracy says nothing about the case in front of them.

There are three problems with leaning on an accuracy figure. First, it is a claim about a population, and an examiner reviews individuals — the 5% is exactly the set they will land on. Second, it is unverifiable without a labeled ground-truth set, an evaluation method, and a date, none of which usually accompany the number. Third, and most damaging, it is silent on how: even where the answer is right, an accuracy figure cannot show the evidence the system relied on, so it cannot distinguish a correct answer for the right reason from a correct answer by luck. Auditors treat those very differently.

The question behind the question is always the same: show me the basis. That is not something you can bolt on afterward. Either the system recorded the basis at the moment it decided, or the basis is gone.

The five blockers that actually stall pilots

None of these is about model quality. Every one is about whether a regulated business can stand behind the output. This is the checklist that decides whether a pilot becomes production.

Blocker	Concern	What it demands from the system
Explainability	Evidence	Every extracted value cites its source — the document, the page, and the location on the page it was read from.
Audit trail	Record	An append-only log of what was read, what was decided, what was validated, by whom, and when — retained and tamper-evident.
Separation of duties	Accountability	The system proposes; a named human with the right role reviews and approves. Roles, permissions, and sign-off are enforced, not assumed.
Reproducible output	Stability	The same input, under the same configuration version, yields the same result — and the config version is recorded with the output.
Data residency & governance	Control	Documented control over where data lives, who can access it, how long it is retained, and how models handle it.

1. Explainability: evidence, not confidence theater

The first thing an auditor wants is not a score — it is a pointer. When the system reports that the each-occurrence limit is $5,000,000, it must be able to answer: where did that come from? Good extraction answers with a citation: this value came from page 3 of the ACORD, from the field in the upper-right of the declarations block. That citation lets a reviewer verify in seconds and lets an auditor trace the decision back to the primary record. A confidence percentage without a citation is theater; it tells you the system’s opinion of itself, not the evidence.

2. Audit trail: an append-only record of what happened

Regulated processes are reconstructed after the fact. That is only possible if the system kept a durable, append-only record as it worked — not a log you can quietly overwrite, but a trail that captures what was read, what the model proposed, which rules and reference data validated it, what a reviewer changed, and who approved it, each entry stamped with an actor and a timestamp. When an exam arrives, the answer to “walk me through this decision” is a record, not a reconstruction from memory.

3. Separation of duties: the machine proposes, a person is accountable

In a regulated line, accountability cannot live inside a model. Someone with the authority to make the decision has to own it. That means the system’s role is to propose a structured, cited result and the human’s role is to review and approve it — with roles and permissions enforced so that the person who built the configuration is not silently the same person who approves its output on a live account. This is the ordinary control environment of insurance, applied to AI: maker and checker, with the checker named.

4. Reproducible output: the same input yields the same result

An auditor who re-runs a case expects to see what was seen before. If the system returns a different answer on Tuesday than it did on Monday for the same document, it cannot be governed — there is no stable object to audit. Reproducibility means the extraction is pinned to a versioned configuration: the schema, the rules, the reference data, and the prompts that produced a result are recorded alongside it, so the exact behavior can be reproduced and, when configuration changes, the change is itself a logged, reviewable event.

5. Data residency and governance: control you can document

Finally, the questions that never appear in a demo but always appear in procurement and compliance review: where does the data live, who can reach it, how long is it kept, and what does the model do with it? A shippable system has documented answers — residency you can point to, access controlled by role, retention you can configure, and a clear statement on whether inputs are used to train anything. These are not model questions. They are the difference between a pilot and a signed contract.

Black box vs. glass box: the same audit questions, two outcomes

The clearest way to see why some pilots die and others ship is to put the auditor’s questions side by side with what each kind of system can actually say in the room.

The auditor asks…	A black box says…	A glass box shows…
“Where did this limit value come from?”	“The model extracted it.”	A citation to page 3 of the ACORD, with the field highlighted on the page image.
“How confident is the system, and why?”	“Overall accuracy is ~95%.”	A per-field confidence, plus the evidence and the validation rules that passed or failed.
“Who reviewed and approved this?”	“It ran automatically.”	A named reviewer, their role, what they changed, and a timestamped sign-off.
“Re-run it — do you get the same result?”	“It may vary.”	The same output, pinned to a recorded configuration version.
“Show me the full history of this record.”	“We have application logs.”	An append-only trail: read, proposed, validated, reviewed, approved — each with actor and time.
“Where is this data, and who can see it?”	“In the cloud somewhere.”	Documented residency, role-based access, and a stated retention and model-use policy.

What good looks like: glass-box traceability by design

The systems that clear the audit gate share a design principle: they record the basis for every decision as they make it, because you cannot reconstruct evidence you never captured. Concretely, that means an agentic pipeline where every stage leaves a trace — perception of what the document says, a plan for what to extract, the extraction itself with per-field provenance, validation against rules and reference data, and an append-only record of the whole run.

Glass-box traceability — every stage records what it did, so the answer to “how did it decide that?” is already on file.

Three properties turn that pipeline from a black box into something a regulated carrier can stand behind:

• Per-field provenance and confidence. Not one accuracy number for the run, but a citation and a confidence for each value — so a reviewer works the low-confidence and rule-failing fields first, and every field can be traced to the page it came from.
• An append-only audit trace. A durable record of what was read, decided, validated, and approved, with actor and timestamp on each step. When the exam comes, the history is retrieved, not reconstructed.
• Governed configuration. The schema, rules, reference data, and prompts are versioned, reviewable, and role-controlled — so outputs are reproducible, changes are logged events, and the people who build a configuration are not silently the ones who approve its results in production.

Why it matters to the business

Treating explainability and audit as requirements rather than afterthoughts is not compliance overhead — it is what determines whether AI extraction ever leaves the sandbox:

• Pilots that ship. The gap between a good demo and a production deployment is almost always the audit conversation. A system that can answer “how did it decide that?” on day one clears the gate that kills most pilots.
• Defensible decisions. When an exam, a dispute, or a lawsuit lands on a specific record, an append-only trail with per-field citations is the difference between a defensible position and an exposure.
• Faster, safer review. Per-field confidence and provenance let reviewers spend their attention where it matters — the uncertain and rule-failing fields — instead of re-keying everything, which is both faster and more accountable.
• Trust that scales. Governed, versioned configuration means the same behavior across accounts, teams, and time — the consistency that lets compliance sign off on rollout rather than case-by-case.
• A real procurement answer. Documented residency, access control, and retention turn the questions that stall contracts into questions you have already answered.

The lesson from every stalled insurance AI pilot is the same: the model was never the hard part. The hard part is being able to prove, for any given decision, what the system read, why it concluded what it did, who checked it, and that the same input would produce the same result again. Build that in from the start — glass-box traceability, per-field provenance and confidence, an append-only audit trail, and governed configuration — and “how did it decide that?” stops being the question that ends the meeting. It becomes the question you were built to answer.