Reading the Benchmarks Honestly: Why 82% LLM Accuracy Is a Failing Grade in Finance

A vendor slide says their model hits 90% accuracy on financial question answering. The room nods. It sounds like an A-minus. In school, 90% was honor roll.

Run the same number through a finance lens and it stops looking like an A. Ninety percent accuracy means one in ten figures is wrong. Put that in a board deck with forty numbers and, on average, four of them are off. You don’t know which four. Neither does the model, which will defend all forty with the same fluent confidence. That is not honor roll. In a function whose entire job is being right about numbers, it’s a failing grade.

The benchmarks deserve a closer read than the headline figure, because the closer you look, the stronger the case gets for taking math away from the language model entirely.

What the benchmarks actually measure

Start with FinanceBench, the open test suite for financial QA built on public company filings. The setup is generous to the model: questions are paired with the exact documents that contain the answers, drawn from filings of 40 large U.S.-listed companies. This is the easy version of the real job. Nobody is hiding the 10-K.

The results were not generous back. In the original work, GPT-4-Turbo paired with a retrieval system answered incorrectly or refused to answer on the order of 81% of questions in the harder configurations. Across setups, GPT-4 landed correct answers somewhere between roughly 19% and 89% of the time, and that wide spread is the part worth sitting with. Same model, same task family, accuracy swinging by tens of points depending on how the context was arranged. One study found that simply moving the context before the question instead of after it moved GPT-4-Turbo from about 25% to about 78%. The model wasn’t reasoning toward a stable answer. It was reacting to prompt shape.

Now the spreadsheet side, which is closer to how finance teams actually work. FinSheet-Bench evaluates models on financial spreadsheet tasks built on realistic private-equity fund structures (lookups at the simple end, multi-step numeric reasoning at the hard end). The best configuration tested, Gemini 3.1 Pro, reached about 82.4% accuracy across the evaluation files. Behind it, GPT-5.2 and Claude Opus 4.6 with reasoning enabled clustered around 80%.

Read that as a top-of-class score and it’s still roughly one error in every six questions. The paper’s own conclusion is blunt: across the configurations tested from OpenAI, Google, and Anthropic, no standalone model reached error rates low enough for unsupervised use in professional finance. That is not a Rexfin talking point. It’s the finding.

Why “use a better model” doesn’t close the gap

The instinct, every time, is to wait for the next release. Scores have climbed year over year, so surely the line eventually crosses into safe territory.

Two things make that optimism expensive.

First, the errors aren’t random noise that averages out. A model that inverts a ratio, reads a number in thousands as a number in millions, or flips the sign on a cash outflow doesn’t make a small mistake: it makes a confident, structural one, and it makes it in the exact place a human reviewer is least likely to double-check because the surrounding narrative reads cleanly. A 90% score with well-behaved errors is one thing. A 90% score where the 10% includes order-of-magnitude misses is something a controller cannot sign.

Second, benchmark accuracy is measured on tidy, curated data. Your data isn’t tidy. It lives across QuickBooks, a NetSuite instance from an acquisition, three warehouses, and a folder of PDFs someone scanned crooked. Accuracy on clean filings is the ceiling, not the floor. The number you’d actually get on your own messy, multi-source financials is lower than any leaderboard, and no benchmark publishes it because no benchmark has your books.

So the honest framing isn’t “models are bad.” Frontier models are genuinely strong at reading, summarizing, and explaining. The honest framing is narrower and more useful: they are not reliable arithmetic engines, and they were never built to be. Asking an LLM to compute a year-over-year change is asking a system trained to predict plausible text to instead produce an exact, defensible figure. Sometimes the plausible token is also the correct one. Eighteen percent of the time, in the hard cases, it isn’t.

The number that should change the architecture, not the vendor

Here is the move the benchmarks point to. If the language model’s weakness is exact computation on real numbers, stop asking it to compute.

Let the model do what it’s good at (understand the question, find the right figures, write the explanation) and route every actual calculation through a deterministic engine running against a reconciled model of your books. The LLM retrieves and reasons. The math runs as code, the same way it runs in a formula. Same inputs, same output, every time, with a path back to the source line in the ledger.

That single split changes what the benchmark numbers mean for you. The 18% spreadsheet error rate and the wide FinanceBench spread are both measuring the model doing arithmetic. Take arithmetic off its plate and those error modes (the inverted ratio, the wrong magnitude, the flipped sign) stop being probabilistic. They become a function you can test once and trust. The model can still misread a question, and a human still reviews the framing. But it can no longer quietly hand you a number that’s off by a factor of ten and a clean sentence to go with it.

This is the foundation underneath everything else. It’s why a spreadsheet is the wrong place to let AI write numbers: you inherit every silent error instead of removing the class of error. And it’s why an AI number should be replayable the way a formula is: identical inputs producing an identical figure, with a traceable path, is exactly what a deterministic layer gives you and a standalone model cannot. The broader case sits in the pillar on the reliable financial-modeling layer for AI.

The pointed version

The next time a benchmark number lands in a meeting, do the translation out loud. Ninety percent is one wrong figure in ten. Eighty-two percent is one wrong answer in six. Then ask which six numbers in the deck the team is comfortable not checking, because that’s what trusting the raw model output actually means.

The benchmarks aren’t an argument for waiting on a better model. They’re an argument for a different architecture: reasoning in the model, math in a layer that can’t hallucinate. That’s the only configuration where the score on the slide and the number in your report are allowed to be the same thing.

If you want to see arithmetic taken off the model and run against a reconciled source of truth, book a demo and bring your messiest reconciliation. That’s the test that counts.