Why AI Misreads Your 10-K: The PDF and Image Problem Behind Financial Hallucinations

Hand a capable model a clean, machine-readable cash flow statement and it will read the numbers back to you correctly almost every time. Hand it a screenshot of the same statement, or a scanned PDF that someone faxed in 2014, and the same model starts inventing. A 4 becomes a 9. A figure in the “prior year” column gets reported as current. A subtotal in a merged cell migrates into the wrong row. The text didn’t change. The pixels did. And that’s enough to break the answer.

This is the part of financial AI hallucination that prompt engineering can’t touch, because the damage happens before the model reasons about anything. It happens at the moment the document becomes tokens.

The accuracy doesn’t degrade gracefully: it falls off a cliff

There’s a comfortable assumption that image input is “a bit worse” than text. The benchmarks don’t support comfortable. On clean forms and correspondence, vision-language models read at 95-98% accuracy. Point the same models at dense financial documents (the multi-column, merged-cell, footnote-heavy tables that make up an actual 10-K) and reported production accuracy drops into the 75-80% range. When compression of the image gets aggressive, accuracy has been measured falling toward 60%. Chart interpretation is worse still: studies put model accuracy on reading values off charts somewhere between roughly 34% and 62%, depending on the chart and the model.

Put bluntly, the same system that looks reliable on a tidy CSV is operating at coin-flip-adjacent reliability on a financial chart inside a PDF. That’s a swing of fifteen to twenty-plus points in accuracy purely from how the document arrived. For most workflows that gap is invisible, until the one quarter it isn’t, and a misread number is already three slides deep in a board deck.

Why is finance the hard case specifically? Three structural reasons:

• Tables are the worst-performing element across every benchmark. Multi-column spans, merged cells, and nested hierarchies break the assumptions OCR and vision models make about reading order. Researchers consistently flag dense, hierarchical, multi-page tables as the failure zone.
• The errors are plausible. A misread digit produces a number that looks like a real number. There’s no red squiggle, no exception thrown. It flows downstream looking exactly like a correct value.
• Finance is unforgiving about which digit is wrong. In prose, a 1% extraction error is noise. In a financial statement, the wrong digit in the wrong cell is the difference between a covenant breach and a clean quarter.

”Just use a better model” is the wrong instinct

Every six months a new model claims to have solved document understanding, and the headline benchmark numbers are genuinely impressive: 97% on a curated set. The trap is the gap between that benchmark and your filing cabinet. Independent reporting puts production degradation at 15-25% versus benchmark conditions, with table misalignment alone accounting for around 30% of production failures.

So you can chase the frontier model. You’ll get a few points. You will not get auditability, and you will not get a guarantee that the figure the model “read” is the figure on the page. The model gives you a confident answer either way. Confidence is not the same as correctness, and on a scanned image you have no cheap way to tell them apart at the moment it matters.

The honest version of this: better models help. They do not close the gap, because the gap isn’t really about reasoning. It’s about a lossy translation from pixels to numbers that nobody verified against a source of truth.

The fix is a layer, not a prompt

If the failure happens at extraction, the fix belongs at extraction. That means treating the document not as something the LLM glances at, but as something that gets parsed, structured, and reconciled before any AI is allowed to reason over it.

Concretely, this looks like a pipeline rather than a single call:

1. Structured extraction. Specialized table parsing and layout analysis pull figures into cells with known coordinates, not a model’s best guess at reading order. Each extracted value keeps a pointer back to where on the page it came from.
2. Reconciliation against the ledger. Extracted statement figures get tied out. Do the line items sum to the subtotals? Does the balance sheet balance? Does the extracted revenue match what the accounting system reports? A scanned 4-read-as-9 fails arithmetic the moment it’s checked against the rest of the statement. The misread surfaces as a tie-out error instead of a confident wrong answer.
3. One reconciled model as the source of truth. Instead of the AI re-reading the PDF every time it’s asked a question, it queries a single financial model that already ties out to the ledger. The pixels were read once, checked once, and locked.
4. Deterministic calculation on top. When the AI needs a ratio or a growth rate, the arithmetic runs through a calculation engine, not the language model. The LLM decides what to compute and explains the result; it never does the math on a number it eyeballed from an image.

This is the architecture Rexfin is built around. We connect to accounting and financial-data platforms (QuickBooks, Xero, NetSuite, Sage, SAP, Oracle, warehouses) or to uploaded statements, and build one reconciled financial model that ties out to the ledger. AI retrieves figures from that model and runs calculations through a deterministic engine, so an answer traces back to a source you can audit, not to a digit a vision model thought it saw.

The point isn’t that OCR is useless. It’s that OCR output has to be treated as a claim to be verified, not a fact to be trusted. Reconciliation is the verification step that the prompt-only approach skips entirely.

What this means for a finance team

If you’re evaluating an AI finance tool, the question to ask isn’t “what model do you use.” It’s “what happens between the PDF and the answer.” A tool that pipes a document image straight into an LLM and returns a number is shipping you that 75-80% table accuracy with no way to catch the misses. A tool that extracts, reconciles, and ties out to the ledger is shipping you a number that failed loudly if it was wrong.

For a deeper look at the failures that survive even when the extraction is perfect, see our companion piece on AI getting the numbers right and the math wrong. And before any AI figure reaches leadership, run it through our verification checklist for finance teams. Both sit under our broader work on AI hallucinations in financial data.

The takeaway is narrow and it matters: the most dangerous financial hallucinations don’t come from the model reasoning badly. They come from the model reading badly, on a document nobody reconciled. Fix the reading, reconcile the result, and the hallucination has nowhere to hide.

If you want to see extraction-plus-reconciliation running against your own filings, book a demo and bring your messiest scanned PDF.