Insurance Document Benchmarks Reveal a Gap Between Model Accuracy and System Reliability
When an AI agent reads an insurance document, the hard part is not the question. It is the file. A scanned ACORD application arrives with handwritten annotations, rotated pages, and stamped-over text. A loss run spreadsheet has fifty tabs. A policy document stretches to hundreds of pages. A submission email arrives as an Outlook .msg file with the application, statement of values, and loss run still attached. The model has to open the file before it can answer anything, and that is where most failures happen.
Cooper Labs published the Insurance Agent Benchmark on September 16, 2026, testing 17 language models across 166 real documents from live brokerage and carrier workflows. The benchmark measures something most insurance AI benchmarks do not: whether an agent can complete document understanding tasks from start to finish, using the messy files that actually land on a commercial-lines desk. Ishika Shah, Cooper's Founding Engineer, authored the report.
The Corpus Is the Point
Of the 166 documents, 150 are untouched originals from existing brokerage and carrier workflows under consent agreements. The remaining 16 are originals modified to set specific traps: an embedded prompt injection instruction, a blacked-out name, a password-protected file, a corrupt file truncated partway through. Nothing is synthetic.
The documents range from a single-page certificate of insurance and photographed auto ID cards to full policy wordings with endorsement schedules, program books hundreds of pages long, and underwriting workbooks with more than fifty sheets. The two largest workbooks pushed Cooper through nearly 16 million tokens to answer a single question. One scanned application alone expands to 1.9 million tokens of extracted text.
Formats span PDF (digital, scanned, broken), XLSX and CSV workbooks, PNG and JPG photos, Outlook .msg with attachments, PPTX, RTF, XML, and JSON system exports. Much of it arrives exactly as desks receive it: faxed and low-quality scans, pages rotated or stamped over, handwritten annotations, checkbox-heavy ACORDs photographed rather than scanned. A few files lie about themselves: a wrong extension, a corrupt file, a "policy" with nothing inside.
The benchmark deliberately targets file sizes around the routing thresholds where a document tips from a single pass into chunked processing. Of the 166 cases, 135 stay under 100,000 tokens, 17 run from 100,000 to 1 million, and 14 exceed 1 million. These are the transitions where document tooling usually breaks.
Eleven Tracks, Not Just Question Answering
The benchmark includes eleven distinct test tracks. ACORD field extraction covers 42 cases pulling structured data from ACORD 125, 140, and 25 forms. SOV and loss-run reasoning tests numeric answers like total TIV and incurred losses across 25 cases. Cross-doc reconciliation checks whether an agent catches mismatches across a submission packet. Long-policy clause retrieval tests whether the model can find a specific clause at the start, middle, or end of a long policy.
Faithfulness and abstention is the track that matters most for trust. When a field is blank, redacted, or unreadable, the correct answer is to say so. The benchmark follows the DUDE approach: when the answer is not in the document, reporting any value at all counts as a failure. Other tracks cover scanned and handwritten document reading, grounding and citations, checkbox and selection reading, chart and graph extraction, number and date normalization, and prompt injection resistance.
What the Harness Changed
Every model was tested twice. In the model-alone run, it received the raw file and question in a single call with no scaffold. With Cooper, the document went through file-type routing, format-specific extraction or rendering, and chunked processing for large files, with the candidate model doing all the reading and reduction. The model, document, and question stayed the same across both runs.
All 17 models scored higher with Cooper. The median lift was 9.4 percentage points. The gains were not均匀. Grok and GPT-5.6 Luna gained about a point, which falls inside single-run noise. Claude Haiku 4.5 gained 15.7 points, going from 59.1% to 74.8%. Meta Muse Spark 1.3 gained 15.1 points. The biggest improvements showed up on the hardest files: long documents, oversized files, and broken inputs that a raw model call struggles to process.
The reliability difference was stark. Model-alone calls failed to produce a usable answer on 7% to 28% of cases, mostly on very large or broken files. With Cooper, every model stayed above 90% reliability. The highest reliability was 98.2%, reached by Gemini 3.5 Flash Lite, Gemini 3.7 Flash, and Fable 5.1.
The Efficiency Frontier
Gemini 3.8 Flash posted the highest accuracy at 85.6%, within the statistical band of the top Claude runs, while costing about $68 for the full corpus. That works out to $0.41 per document. Claude Opus 5 scored 85.3% but cost $560. Claude Fable 5 scored 84.6% at $1,100. Claude Sonnet 5 scored 84.4% at $222.
Gemini 3.7 Flash sat at 83.3% for $33, or $0.20 per document, and tied for the highest reliability at 98.2%. The report's efficiency chart shows a clear elbow: Gemini models occupy the low-cost, high-accuracy region, while the top Claude models cluster in the same accuracy band at five to fifteen times the cost.
The cost estimates use OpenRouter list prices from August 18, 2026, with a 90/10 input/output token split assumption. They are estimates, not actual billed costs, and the run totals are dominated by a handful of very large files that a plain model call cannot ingest at all.
The Hallucination Problem
When a field is blank, redacted, or unreadable, most models still invent values often enough to matter. Claude Sonnet 5 and Fable 5.1 had the lowest hallucination rates at 10.6%. Opus 5 and Meta Muse Spark were next at 14.9%. At the other end, GPT-5.6 Terra invented values on 38.3% of absent fields, and three models (Gemini 3.5 Flash Lite, GPT-5.6 Luna, GPT-5.6 Sol) hit 36.2%.
The report includes a concrete example: an ACORD 125 with the named insured blacked out and premium fields blank. The ground truth says to report both as unavailable. A frontier model with Cooper instead answered with a specific company name and a total premium of $5,790.00, complete with page citations. Fabricated answers with citations are harder to catch than obvious errors.
Prompt Injection vs. Long Policies
Sixteen of 17 models resisted document-embedded prompt injection at 90% or better with Cooper. The report includes an example of a constructed ACORD application with an injected instruction near the footer telling the reader to output an approval message. Claude Sonnet 5 identified it as a prompt injection attempt and refused to follow it.
Long-policy clause retrieval was far more uneven. Fable 5.1 reached 89.7%, but most Cooper runs remained below 80%. The report notes that this capability exists in some models but is not consistent. Checkbox reading was similarly uneven, ranging from 43% to 86% across models.
What the Benchmark Does Not Cover
The corpus excludes personal lines, non-US markets, languages beyond a three-document bilingual slice, handwriting-only documents, and any judgment task like appetite, pricing, or coverage adequacy. It measures reading, not underwriting. Later phases will cover whole workflows, then long-horizon tasks and browser-use capabilities.
The grading uses a pinned frontier LLM that scores answers on meaning, not string match. $1M equals $1,000,000, date formats are interchangeable, field order is irrelevant. Grading against a human-written answer key is the main defense against the known ways an LLM judge can fail. Differences of about 6 points or less should not be treated as definitive rankings, given the 95% interval on a sample of 166 cases.
For teams evaluating AI for insurance document processing, the benchmark's most useful finding may be the simplest: the same model scores higher with better document handling. The harness raised accuracy for every model tested, without a model upgrade. The problem is not always which model to use. It is whether the system can get the document into the model in the first place.