The same state shows up as “California,” “Calif,” and “CA.” A premium arrives as “$1,234.5,” a date as “01/15/2024,” a total that nobody actually typed but everybody needs. Raw extracted values are only halfway to useful — if the tool stops there, the standardizing, formatting, computing, and checking all land on your team, in downstream scripts nobody wants to own. That’s the “last mile” of extraction, and it’s where a lot of the real cost hides.

InsightXtract runs that last mile as part of the extraction, driven by declarative rules attached to each document class or output contract. Four kinds of rules turn raw values into clean, standardized, decision-ready data — and every transformation is recorded.

StandardizeMap messy values to canonical ones via glossaries & lookups. Calif → CA
FormatNormalize types & shapes. $1,234.5 → 1234.50, dates to ISO.
DeriveCompute new fields from existing ones. total = base + tax
ValidateCheck constraints & flag issues. required, range, in-list.

The four rule types, with examples

1 · Standardize — one canonical value

Extracted values are matched against glossaries (business terms with aliases) and lookup tables (ISO/standard code sets) and rewritten to the canonical form. This is what makes downstream matching, dedupe, and reporting actually work.

“California” → CA    “Calif” → CA    “N.Y.” → NY

2 · Format — consistent types and shapes

Values are coerced to the type and format your systems expect — dates to YYYY-MM-DD, currency to a fixed decimal number, and so on. No more “is this a string or a number” surprises downstream.

“01/15/2024” → 2024-01-15    “$1,234.5” → 1234.50

3 · Derive — compute what isn’t written down

Fields that should exist but aren’t stated — a premium total, a roll-up — are computed from the values that are extracted, by a formula you declare. High-stakes totals are derived from their parts rather than hand-read, which eliminates a whole class of transcription error (and pairs with the invariant checks from self-checking extraction).

total_premium = base_premium + surcharge + tax

4 · Validate — flag what’s wrong

Constraints run over the cleaned values — required fields present, numbers in range, codes that resolved to a real entry — each with a severity that decides whether it blocks or merely warns.

# post_processing — declared once on the document class / contract
standardize:
  - { field: fields.insured_state, lookup: us_state_codes }
  - { field: exposure_schedule.state, lookup: us_state_codes }
format:
  - { field: fields.effective_date, type: date, format: "%Y-%m-%d" }
  - { field: fields.annual_revenue, type: currency, decimal_places: 0 }
derive:
  - { field: fields.total_premium, formula: "base_premium + surcharge + tax" }
validate:
  - { field: fields.insured_name, rule: required, severity: error }
  - { field: fields.insured_state, rule: lookup_match, lookup: us_state_codes, severity: warning }

Every transformation is recorded

Post-processing isn’t a black box. Each run attaches a standardization report listing exactly what was standardized, formatted, and derived, plus any validation warnings — so an auditor (or an underwriter) can see how a clean value got that way.

// standardization report, attached to the result metadata
{ "standardization": {
  "fields_standardized": ["insured_state", "claims.status"],
  "fields_formatted": ["effective_date", "annual_revenue"],
  "fields_derived": ["total_premium"],
  "validation_warnings": [
    { "field": "insured_state", "rule": "lookup_match", "value": "Unknown State", "message": "Not found in us_state_codes" }
  ] } }
flowchart LR R[raw extracted
values] --> ST[standardize
glossary · lookup] ST --> FM[format
date · currency · type] FM --> DV[derive
totals · roll-ups] DV --> VL[validate
required · range · match] VL --> C["decision-ready record
+ standardization report"]
InsightXtract glossaries — business terms with aliases that power the standardize step, mapping messy extracted values to canonical forms
Glossaries and lookups are the reference data behind the standardize step — governed, versioned, reusable across every agent.

Why it matters

  • Decision-ready, not raw. The output is standardized, typed, complete, and checked — ready for a rating engine or a warehouse without a cleanup pass.
  • No downstream ETL team. The normalization that usually lives in fragile scripts lives in governed, versioned rules instead.
  • Business analysts own the rules. Standardization, formatting, and checks are config in the Rules tab — changed without a deploy.
  • Auditable by construction. The standardization report shows every transformation, so a clean value is never a mystery.

Standardization is what makes data join

A warehouse can only aggregate exposure by state if every “Calif” and “California” has already become CA. Standardizing at extraction — against governed reference data — is what turns a pile of correct-but-messy values into a dataset that actually rolls up across a book.