RAG for Finance Isn't Enough Without a Calc Layer

A retrieval system can hand your AI the exact page of the 10-Q where revenue is reported, the exact cell in the export, the exact line of the GL. And the AI can still tell you the wrong gross margin.

That gap is the whole problem with treating RAG as the answer for finance.

Retrieval-augmented generation earned its reputation honestly. Bolt a vector search onto a language model, feed it your documents, and it stops inventing facts out of thin air. For “what does our refund policy say” or “summarize the MD&A section,” it works. The model grounds its answer in something real, and you can usually click back to the source. For a lot of knowledge work, that’s enough.

Finance is not a lot of knowledge work. Finance is arithmetic that has to tie out.

Retrieval answers “where,” not “is it right”

Here’s the distinction that gets lost. RAG is very good at provenance. It can tell you where a number came from. It is not good at calculation, because the calculation still runs through the language model, and the language model is a text predictor, not a spreadsheet.

So you get this strange failure mode. The AI retrieves the correct revenue figure. It retrieves the correct cost of goods sold. Then it computes gross margin by predicting what the next tokens probably are, and it lands on a number that looks plausible and is off by enough to matter. Every input was right. The output is wrong. And because the inputs were genuinely retrieved from your data, the answer comes wrapped in citations that make it look trustworthy.

That’s worse than an obvious error. An obvious error gets caught. A confidently cited wrong number gets pasted into a board deck.

Researchers who study this keep hitting the same wall. Embedding text is easy because words carry meaning that vectors can capture. Tabular financial data is hard, because the numbers themselves carry almost no semantic signal. A column of figures is just a column of figures to an embedding model. Recent work on text-and-table retrieval benchmarks found that even when the system surfaced exactly the right rows, models still couldn’t reliably parse the table to derive the trend, and the authors concluded the task needs an actual computational step, not better retrieval. (T²-RAGBench makes this point directly.)

Read that twice. Better retrieval does not fix the math. The retrieval was already correct.

The accuracy ceiling nobody advertises

Naive RAG over financial documents tends to plateau somewhere in the 85 to 92 percent range on numerical question-answering, depending on how you measure. That sounds respectable until you sit with what it means.

In finance, 92 percent accuracy is not a B-plus. One paper studying financial auditing put it sharply: a 99 percent accuracy rate produces zero operational trust if the one percent error is a sign-convention flip on a balance sheet. A margin that should be positive comes back negative. A liability gets read as an asset. The aggregate score looks great. The specific answer is unusable, and you have no way of knowing which answers fell in the bad slice.

The ceiling has structural causes, not solvable-by-prompt causes:

• Numbers don’t embed well. Semantic search retrieves what reads similar, not what computes correctly.
• Multi-step reasoning drifts. Each arithmetic hop through the model is another chance to round, transpose, or hallucinate a step.
• No reconciliation. Retrieval pulls a figure from one source with no check that it agrees with the same figure elsewhere in the ledger.
• Locale and format ambiguity. Is 1,250 one thousand two hundred fifty, or one point two five? The model guesses.

None of those get fixed by indexing more documents or buying a smarter base model. They are properties of asking a text predictor to do bookkeeping.

What actually moves the number

The research that breaks past the ceiling has one thing in common. It stops asking the language model to do the math.

Two moves matter. First, make retrieval aware of metadata and uncertainty, so the system knows not just what it found but how confident it should be. Bayesian, uncertainty-aware retrieval applied to financial QA reduced hallucinations by roughly 28 percent compared with standard methods, while pulling precise figures where conventional retrieval failed. (Bayesian RAG reports a 27.8 percent reduction.) Citation-enforced and fine-grained verification approaches push reliability further by decomposing an answer into atomic claims and checking each one against the source rather than rating the whole response as a blob.

Second, and this is the part most teams skip: route the actual calculation to a deterministic engine. The model decides which figures it needs and what operation to run. A separate, auditable engine runs the operation, on verified inputs, the same way every time. The model never adds the numbers. It orchestrates; it does not arithmetic.

Combine those and the picture changes. Metadata-aware retrieval plus a deterministic calc layer pushes reliability past 93 percent and, more importantly, makes the wrong answers visible instead of hidden. When the engine computes deterministically over reconciled inputs, you can trace any output back through the calculation to the source figure to the ledger. The number isn’t just cited. It’s reproducible.

Why the reconciled model has to come first

There’s a step underneath all of this that retrieval can’t do on its own. Retrieval pulls a number. It does not check that the number agrees with itself across your systems. Your billing platform, your accounting export, and your data warehouse can all report a slightly different revenue figure for the same month, and retrieval will happily hand the AI whichever one it surfaced first.

That’s why a deterministic calc layer only delivers if it runs on a reconciled model, one source of truth that ties out to the ledger before any AI touches it. We’ve made the case for that foundation in Single Source of Truth: Why Reconciled Data Is the Real Unlock for AI in Finance. And it pairs with the more fundamental issue of why AI gets financial math wrong in the first place, which is a property of how these models read numbers at all.

This is the architecture Rexfin runs on. Connect your accounting and financial sources, build one reconciled model, then let AI retrieve figures from it and calculate against a deterministic engine instead of doing the math itself. Retrieval for provenance. Reconciliation for trust. Deterministic computation for arithmetic that ties out. The fuller argument lives in the pillar on the reliability layer for AI in finance.

The takeaway

RAG solved the right problem at the wrong layer. It made AI answers traceable, which finance badly needed. But traceable and correct are not the same thing, and retrieval alone tops out exactly where finance can least afford it. The number being cited doesn’t make the math right. A deterministic engine running on reconciled data does.

If your AI is citing sources but you still can’t trust the figures it computes, the calc layer is the piece you’re missing. Book a demo and we’ll show you the difference between a cited number and a number that ties out.