Auditable Multi-Document Extraction

Executive summary

An extracted value is only useful to a regulator or an auditor if you can point to where it came from.

A submission arrives as an email with an ACORD form, a broker application, an exposure workbook, and a loss run stapled to it. Downstream systems want one clean record. But the moment several documents are merged into one, three audit questions appear: which document supplied each final value, what happened when two documents disagreed, and can a reviewer see the original page? InsightXtract is built so that every consolidated field answers all three — by construction, not by after-the-fact reconstruction.

1. Why merged extraction is an audit problem

Single-document extraction has an easy audit story: one file in, one set of fields out, one place to look. A submission breaks that story. The same fact — the named insured, the requested limit, an annual revenue figure — can appear in the email body, the application, the ACORD form, and the audited financials, sometimes with different values. When you flatten all of that into one record, the provenance disappears unless you deliberately keep it.

That loss of provenance is exactly what fails an audit. Three failure modes matter to compliance and underwriting leadership:

• Untraceable values. A merged field with no source pointer is an assertion, not evidence. When an examiner asks "where did this revenue number come from?", "the model produced it" is not an answer.
• Invisible conflicts. If two documents disagree and the system quietly keeps one, the discrepancy is gone — and so is the chance for a human to catch a fraud signal or a stale figure.
• Unexplained decisions. Even when the system picks the right value, an audit needs the reason. A choice with no recorded rationale can't be reviewed, only trusted.

InsightXtract treats each of these as a first-class requirement. The rest of this paper walks the pipeline in order — because, as section 6 explains, the order is itself part of the audit trail.

2. Per-document extraction: right modality, real citations

Before anything is merged, every document is handled on its own terms. An incoming submission is classified so that each file is assigned a role — submission email, application form, ACORD form, exposure workbook, loss run — and each role is extracted independently through its own document-class spec. Nothing is extracted "as a submission"; the submission is assembled later, from clean per-document results.

Each document type takes the path that preserves the most signal and the most traceability:

Format	How it's read	Citation it produces
PDF / scans	Vision extraction over rendered pages	Page number + bounding box
Word / office	Converted to PDF first, then vision over rendered pages	Page + bbox on the generated PDF
Excel	Deterministic layout analyzer (metadata blocks vs. tables)	Sheet / cell position
Email (.eml/.msg)	Exploded into body + attachments; each part extracted as a first-class document	Per-part, inherits the part's modality

The key design choice is that PDF and Word both go through vision over rendered pages. Word is converted to PDF before extraction, so a native PDF and a Word document produce the same kind of evidence: a page number and a bounding box on a rendered artifact a reviewer can actually look at. Excel takes a deterministic layout analyzer instead of an LLM guessing cell positions, and email is exploded so that each attachment becomes a document in its own right rather than an opaque blob. Because every part is extracted independently, a problem in one document never contaminates another — and every extracted value is born with a pointer back to its source.

3. Consolidation by priority

Merging is not a guess. The master configuration's output_schema defines, for each output field, a priority-ordered list of sources. Each source is a small object — {role, field, priority} — naming which document role supplies the value, which field within that role's extraction to read, and where it sits in the priority order.

Consolidation is then a deterministic rule: for each output field, walk its sources in priority order and take the first non-null candidate. If the highest-priority source didn't contain the field, the next one is tried, and so on. The logic is explicit and inspectable — there is no hidden heuristic deciding which document "wins".

Every consolidated field carries provenance alongside its value: which role and file it came from, the page, and the model's confidence. That object is not a report generated later — it is attached at the moment the value is chosen, so the merged record and its evidence never drift apart.

In a measured Directors & Officers run rendered as Word→PDF, all 38 consolidated fields carried page citations tied to the generated PDF. Every value in the merged record could be pointed back to a page a reviewer could open.

4. When documents disagree

Priority ordering resolves the common case — the highest-priority source has the value, and that's the answer. But sometimes more than one document supplies a distinct, non-null value for the same field. That's a genuine conflict, and it is where most systems quietly fail: they pick one and move on, erasing the disagreement.

InsightXtract escalates instead. When two sources disagree, an LLM adjudicator resolves the conflict, and its reasoning is recorded on the field. The adjudicator follows a stable rule of thumb — prefer audited financial statements over application forms, and application forms over broker emails; detail beats summary — but it explains, in each case, why it chose what it chose. The rationale is stored, so a reviewer sees not just the winning value but the argument for it.

The governing principle for anything the system can't confidently reconcile — a validation failure, a value that violates a rule — is "flag, don't fix." Rather than silently overwriting a suspicious value, the system surfaces the discrepancy for a human. That is why the count of silent overwrites is zero: a conflict either resolves with recorded reasoning or is raised for review. Nothing gets quietly changed behind an auditor's back.

5. Citations you can click

Provenance says which document and page; a citation lets a reviewer see it. Because PDF and Word→PDF are extracted with vision over rendered pages, every value produced from those documents can point to a page — and a bounding box — in the source PDF. That PDF is the exact artifact shown to the reviewer, so the highlight sits on the same page image a human reads.

This is what makes the output defensible. When an examiner questions a figure, the answer is not a probability score — it's the page, with the value boxed on it. And because Word documents are converted to PDF before extraction, a submission delivered as Word gets the same clickable page-level citations as one delivered as PDF; the D&O run in section 3 was rendered exactly this way, and every one of its 38 consolidated fields resolved to a page in the generated PDF.

• Page + bounding box for every vision-extracted value, tied to the rendered source.
• Same evidence for Word, via Word→PDF conversion before extraction.
• Provenance follows consolidation, so the citation on a merged field points to the document that actually won, not a generic "somewhere in the bundle".

6. The glass-box trace

Provenance and citations explain the output. The glass-box trace explains the process. As the agent works each document, it records an append-only reasoning trace across five steps — perceive → plan → extract → validate → reflect — that is streamed live as the run happens and persisted afterward for audit. Nothing in the trace is rewritten; each step is appended, so the record shows what the agent actually did, in order.

Order is not cosmetic here. Conflict resolution and agent reasoning correct the consolidated output before standardization rules are applied. Standardization — normalizing dates, currencies, state codes, and the like — runs last, on a record whose values have already been chosen and, where necessary, adjudicated. That sequencing means the audit trail reflects the real decision: a reviewer sees the substantive choice (which source, why) separately from the cosmetic formatting that came after it. If standardization ran first, it could mask the very disagreement an auditor needs to see.

InsightXtract — agentic document extraction for specialty P&C. The 38/38 citation figure is a measured run of the production engine on a Directors & Officers bundle rendered as Word→PDF and reflects the configuration at time of writing.