- contact@insightxtract.com
A gold-dataset evaluation methodology for specialty P&C submission intake — and the results it produces across PDF, Excel, and Word documents.
Every extraction vendor claims high accuracy. Almost none publish how they measure it. This paper describes the evaluation methodology InsightXtract uses to hold its own agents to account: synthetic gold datasets generated from known ground truth, scored at both the per-document and consolidated level, across every document type an insurance submission contains. We define the method, the scoring rules, and report measured results on representative Directors & Officers and excess-casualty bundles.
You cannot manage what you cannot measure — and in document extraction, the measurement is the hard part.
A submission is not one document. It is an email, an ACORD form, a broker application, an exposure workbook, sometimes a loss run — each in a different format, each contributing overlapping and occasionally contradictory facts. "95% accurate" means nothing without answers to: accurate on which field, in which document, measured against what truth? InsightXtract answers those questions with a repeatable harness that produces exact, field-level scores.
Three problems make extraction accuracy easy to overstate and hard to verify:
Our methodology is built to remove all three excuses.
Instead of labelling documents, we generate documents from labels. Each gold sample starts as a single structured record of known-correct values — the insured, the coverage limits, the financials, every table row. From that record we render realistic source documents: a multi-page audited financial statement, a supplemental application form, an exposure workbook, a broker email. Because the documents are produced from the record, the ground truth is exact and complete — not an approximation.
Every sample carries two answer keys, because there are two questions to answer:
ground_truth.json — the correct fields and table rows per document. This scores each document against what it, specifically, contains.expected_unified.json — the correct consolidated record after all documents are merged. This scores the output that actually reaches downstream systems.Why generate rather than label? A generated corpus is exact, reproducible, and instantly extensible — add an insured, a line of business, a new document type, and the answer key comes with it. It complements (not replaces) evaluation on real, customer-labelled documents; it is what lets us regression-test every change to the engine before it ships.
Scoring is deterministic and deliberately strict — no LLM grades the output, so the number can't drift.
The tables below report measured runs of the production engine against two representative bundles. Scores use the gold-null-aware scorer described above.
Audited financial statements, a supplemental application, and a renewal application — consolidated into one management-liability record.
| Document / output | Fields | Tables |
|---|---|---|
| Financial statements | 4 / 4 | fiscal_years 100% cells |
| Supplemental application | 43 / 43 | crime_agreements 100% |
| Renewal application | 10 / 10 | benefit_plans (rows 2/2) |
| Consolidated record | 38 / 38 (100%) | all tables 100% cells |
Broker email, application, ACORD form, and an Excel exposure workbook — consolidated into one excess-casualty submission record.
| Document / output | Fields | Tables |
|---|---|---|
| Exposure workbook (Excel) | 6 / 6 | exposure_schedule 100% |
| Application (PDF) | 20 / 20 | — |
| ACORD form (PDF) | 31 / 31 | coverage & underlying 100% |
| Broker email | 12 / 13 | — |
| Consolidated record | 29 / 29 (100%) | all tables 100% cells |
Reading the one "miss." The broker email scored 12/13 because it captured $50,000,000 excess/umbrella where the key held $50,000,000 — a more faithful reading of the source, not an error. The consolidated record is still 100%, because priority sourcing takes that field from the application and ACORD. This is exactly the kind of nuance a single aggregate score would erase.
A submission is multi-format, so the harness — and the engine — must handle every format first-class. Each document type takes the path that preserves the most signal:
| Format | How it's read | What that buys |
|---|---|---|
| PDF / scans | Vision extraction over rendered pages | Layout + tables + bounding-box citations |
| Excel | Deterministic layout analyzer (metadata blocks vs. tables, header-name mapping) | Complex sheets without an LLM guessing cell positions |
| Word / office | Converted to PDF, then vision | Same citations & display as native PDFs |
| Email (.eml/.msg) | Exploded into body + attachments; each part extracted | Attachments become first-class documents |
The results in section 4 span all four paths — the D&O bundle was also run as Word documents (converted to PDF) and reproduced the same 100% consolidated score, with page-level citations on every field.
A benchmark is only useful if it governs the real system. The same harness runs as a regression gate: before any change to the extraction engine ships, it is scored against the full gold corpus, and a drop in any field, in any document, blocks the release. In production, the same per-field, per-document scoring underpins confidence routing — high-confidence fields auto-accept, uncertain ones are flagged for a human — so the measurement discipline that proves accuracy is the same one that decides what needs review.
The claim we're comfortable making. Not "99% accurate." Instead: here is the corpus, here is the scoring, here are the field-level numbers, and here is the harness that keeps them honest on every release. That is a claim you can audit.
The full gold corpus spans 12 lines of business × 3 insureds = 36 submission bundles, totalling 99 source documents across email, PDF application, and Excel workbook formats, plus the specialized D&O and excess-casualty bundles reported here (in both PDF and Word). Each bundle carries per-document and consolidated answer keys and is re-scored on every engine change.
We'll run the same methodology against a sample of your real submissions and show you the field-level scorecard.
Talk to us → Next: Auditable extraction →InsightXtract — agentic document extraction for specialty P&C. Figures in this paper are measured runs of the production engine against the described gold datasets and reflect the configuration at time of writing.