Build vs. buy for document automation: the honest math

Let’s be clear up front: sometimes building is the right answer. If document automation is core to how you differentiate, if you have a standing ML platform team, and if your document mix is narrow and stable, an in-house build can be the better economic choice — and you’ll own the roadmap. This post isn’t an argument against building. It’s an argument for costing the build honestly, because the gap between a working demo and a production system that clears 80% of your submissions without a human is where most of the real work — and most of the surprise — lives.

The reason build-vs-buy is hard here is that the first 70% is easy and the last 30% is most of the cost. A weekend prototype that reads a clean ACORD and returns JSON feels like the problem is solved. It isn’t — it’s the point at which the interesting problems start. Below is what a naive “we’ll just call an LLM” plan tends to underestimate, followed by a cost framing and a decision checklist you can take into a steering meeting.

What the demo leaves out

None of these are exotic. They’re the ordinary, unavoidable requirements of running extraction against real broker traffic. The prototype skips them because a curated test file doesn’t exercise them. Production does, immediately.

What the build underestimates	Effort	Why it bites in production
The long tail of document variability	Ongoing	Every broker sends a different ACORD-ish PDF, scanned loss run, or Excel workbook with merged cells. The demo works on three files; the book has three thousand layouts, and the tail never ends.
Evaluation & gold sets	Upfront + ongoing	Without a labelled ground-truth set and a scoring harness, you can’t tell whether a prompt change helped or quietly broke a field. “It looks right” is not a metric.
Confidence & human-in-the-loop	Upfront	An answer with no confidence is unusable — you can’t decide what to auto-post vs. route to a reviewer. Building calibrated confidence, review queues, and correction capture is a product in itself.
Reference data & glossaries	Upfront + curation	NAICS/SIC/ISO codes, state and currency normalization, carrier and coverage synonyms. Raw model output is inconsistent; someone has to own and version the lookup tables.
Governance & audit	Upfront	Field-level provenance, who-changed-what, PII handling, model-version pinning, and reproducibility. Regulated lines need to defend a number a year later.
Integration & the last mile	Upfront	Getting clean JSON is 60% of value at most. The rest is landing it in the PAS/rating engine/data warehouse, in their schema, reliably, with retries and dead-letter handling.
Versioning & regression safety	Ongoing	Schemas evolve, prompts change, models get deprecated. Without versioned specs and regression tests, every improvement risks silently breaking a field that worked yesterday.
Ongoing maintenance	Ongoing	Documents drift, brokers change templates, model providers ship new versions and sunset old ones. This is not a project you finish; it’s a system you staff.

The cost framing: past the sticker price

Build-vs-buy is usually argued as “license fee vs. free.” That’s the wrong comparison. An in-house build is never free — it’s a standing commitment of your scarcest resource. Four costs matter more than the license line:

• Engineering time — and who those engineers are. Extraction that clears production needs ML/LLM engineering, data engineering, and a review-workflow front end. The honest question isn’t “can we build it” but “can we build and staff it indefinitely” — because someone has to keep it running after the launch team moves on.
• Opportunity cost. The strongest engineers you’d put on this are the same ones you’d put on pricing sophistication, distribution, or claims analytics — things a vendor can’t do for you. Extraction is undifferentiated heavy lifting for most carriers; the differentiation is what you do with the structured data, not the parsing.
• Time-to-value. A bought platform can be running on your documents in weeks. A build that reaches comparable straight-through rates is typically a multi-quarter effort before it touches a real renewal. Every quarter of delay is a quarter of manual intake you keep paying for.
• Total cost of ownership. The build cost is the down payment. TCO is the down payment plus years of maintenance, on-call, model migrations, new-document onboarding, and the review tooling — against a license that folds all of that into one predictable line.

The distance from prototype to production is the part the build estimate usually skips — and it never fully ends.

The metric that actually matters

Most build-vs-buy conversations anchor on accuracy — “the model gets the fields right 95% of the time.” Accuracy matters, but on its own it’s a vanity metric. Two numbers decide whether automation pays for itself:

• Straight-through-processing (STP) rate — the share of documents that go from inbox to system-of-record without a human touching them. This is the number that removes cost. A system that’s 95% accurate but routes 60% of documents to manual review because it can’t tell which 5% are wrong has an STP rate of 40%, and it hasn’t saved you much.
• Cost per document — fully loaded: model/API spend, plus the human-review minutes the system doesn’t avoid, plus amortized engineering. This is the number a CFO can compare against the manual baseline and against a vendor quote on equal terms.

Notice that raw accuracy feeds STP but doesn’t equal it. STP depends on calibrated confidence and a review workflow — knowing which answers to trust — far more than on squeezing another accuracy point. That’s precisely the part a naive build under-invests in, and it’s why a slightly-less-accurate system with excellent confidence routing often has a higher STP rate, and lower cost per document, than a more accurate one without it.

A balanced view: when each answer is right

There’s no universal answer, and anyone who gives you one is selling something. The honest version depends on your situation.

Dimension	Leans build	Leans buy
Strategic role of extraction	Core differentiator	Undifferentiated plumbing you want reliable, not owned
Document variety	Narrow & stable	Broad, messy, and constantly changing broker traffic
In-house capability	Standing ML platform team	Lean team you’d rather point at differentiated work
Time-to-value pressure	Multi-quarter is fine	You need STP lift this quarter, not next year
Governance / audit needs	Willing to build it all	Want provenance, versioning, RBAC out of the box
Appetite for maintenance	Can staff it indefinitely	Don’t want to own model migrations forever
Total cost of ownership	Volume justifies a team	Predictable license beats a standing headcount

A useful gut check: if most of your rows land in the left column, a build may genuinely be the better call — and you should resource it like the multi-year product it is. If most land on the right, you’re likely paying to rebuild undifferentiated infrastructure that a platform already runs, and the buy case is strong. Many carriers also land in a middle path — buy the extraction and workflow substrate, and build the thin layer of proprietary rules and integrations that actually reflect your appetite.

A decision checklist

Before committing either way, get honest answers to these. If you can’t answer them for a build, that’s the signal — not that build is wrong, but that the estimate isn’t finished yet.

• Target STP rate. What straight-through rate do we need to hit for this to pay off, and how will we measure it against a labelled gold set — not against “looks right”?
• The gold set itself. Who builds and maintains the ground-truth data, and how do we score every change against it before it ships?
• Confidence & routing. How does the system decide what to auto-post vs. send to a reviewer, and who builds the review queue and correction capture?
• The long tail. What’s our plan when broker #400 sends a layout we’ve never seen — and who owns that backlog every week?
• Reference data ownership. Who curates and versions the NAICS/ISO/coverage lookups and normalization rules?
• Governance. Can we produce field-level provenance, a model version, and an audit trail for any number, a year later?
• The last mile. Who owns reliable delivery into the PAS/rating/warehouse, with retries and error handling — and is that scoped, or assumed?
• Maintenance staffing. Who keeps this running after launch, through model deprecations and document drift — and is that headcount funded, not borrowed?
• Fully-loaded cost per document. Have we compared build and buy on the same basis — API + human minutes + amortized engineering — against the manual baseline?
• Opportunity cost. What differentiated work do these same engineers not do while they build and maintain extraction?

Why it matters

The stakes aren’t really “license fee vs. free.” They’re about where your best engineering goes and how fast intake stops being a bottleneck. A build that’s under-scoped doesn’t fail loudly — it ships a demo, stalls in the long tail, quietly consumes a team, and leaves STP stuck in the 40s while manual intake keeps running. A build that’s scoped honestly, resourced fully, and aimed at something you truly want to own can be a real advantage. The mistake is neither building nor buying — it’s deciding on the demo instead of the honest math.

Whichever way you lean, the discipline is the same: define the STP target, build the gold set, measure cost per document on equal terms, and be honest about the maintenance tail. Do that and the right answer for your situation usually becomes obvious — and it’s a decision you can defend to a board rather than one you backed into.