How to Stress-Test an AI Finance Tool Before You Trust It

A vendor sits across from you and says their AI is “98% accurate on financial data.” Ask them one question: 98% of what, measured against whose answers? In most demos, the honest reply is that the number came from a public benchmark, run on someone else’s filings, scored by a method you cannot see. It tells you almost nothing about how the tool will behave on your closing entries, your intercompany eliminations, your messy chart of accounts.

The published research is not reassuring either. On FinanceBench, a benchmark of over 10,000 question-answer pairs drawn from U.S. public filings, an early test of GPT-4-Turbo paired with a retrieval system incorrectly answered or refused to answer roughly 81% of a 150-question sample. That is a frontier model, with retrieval, on clean SEC documents. Your data is harder.

So stop trusting headline accuracy. Build your own test. A golden dataset is the most useful afternoon a finance team can spend before signing anything, and it is the only evidence that survives an audit committee question later.

What a golden dataset actually is

A golden dataset is a fixed set of questions about your financials where you already know the correct answer, ratified by a human who can defend it. Not a model’s answer. A controller’s answer, tied to the ledger.

Each entry has four parts: the question as a user would actually phrase it, the correct answer with units and sign, the source it traces to (which account, which period, which statement), and the calculation that produces it. “What was Q3 free cash flow?” is a question. “$4.18M, operating cash flow of $6.02M less capex of $1.84M, per the cash flow statement” is a golden answer. The math and the lineage are part of the record, not an afterthought.

You do not need thousands of these. You need the ones that matter and the ones that break things.

What to put in it

Pull from four buckets, deliberately.

Retrieval cases. Plain lookups. “What was total revenue last fiscal year?” These check whether the tool finds the right figure in the right period. They look trivial. They are where stale-data errors hide, because a model will happily quote last year’s number with full confidence.

Calculation cases. Anything requiring math: gross margin, year-over-year growth, days sales outstanding, a three-line EBITDA bridge. This is where most AI finance tools quietly fail. A model can retrieve two correct numbers and still invert the ratio, confuse an absolute change with a percentage change, or flip the sign on a cash outflow. The answer reads plausibly. It is wrong. This is the failure mode we cover in AI Got the Numbers Right and the Math Wrong, and it is the single biggest reason to test calculation separately from retrieval.

Adversarial cases. Questions designed to provoke a confident wrong answer. Ask for a metric you don’t report. Ask about a period that doesn’t exist yet. Ask a question with a false premise baked in (“Why did margin fall in Q2?” when it rose). A trustworthy tool says it cannot answer or corrects the premise. A weak one fabricates. You want to know which you bought.

Multi-source cases. Questions that force the tool to reconcile across systems: a figure that appears in both the GL and a billing platform, a consolidated number spanning two entities. If the tool pulls from one source and ignores the conflict, you will see it here.

Aim for a few dozen to a couple hundred entries. Spread them so calculation and adversarial cases are well represented, because those are the ones that move the pass rate.

Set a threshold, and make it hurt

Pick a number before you run the test, in writing. We use 95% as a starting bar for any figure that reaches a report or a board deck, and lower thresholds buy you very little. At 90%, one in ten answers is wrong, and you cannot predict which one, which means a human has to re-check all of them. The whole point of the tool evaporates.

Then define what “correct” means precisely. Exact match on the figure. Right sign. Right units. Right period. A tool that returns the right magnitude with the wrong sign is not 90% correct on that item; it is wrong. Score it that way. Be stricter on calculation and multi-source cases than on plain retrieval, because those are the ones a controller cannot eyeball.

One more rule worth borrowing from the board-deck verification checklist: an answer without a traceable source counts as a miss, even if the figure happens to be right. A correct number you cannot prove is not an asset on an audit. It is a liability waiting for a question.

Re-test for drift, on a schedule

Here is the part most teams skip. You pass the test in February and assume the tool is safe in July. It may not be. The model behind it gets updated. Your data changes shape. Prompts get tweaked. Accuracy that was 96% can quietly slide to 88% with no announcement, because nothing visibly broke.

Run the golden dataset monthly, or after any model or integration change, whichever comes first. Track the pass rate over time. A two-point drop is a signal worth investigating; a five-point drop is a reason to pause reliance until you understand it. This is the difference between trusting a tool once and governing it continuously.

What the test is really measuring

Strip it back and a golden dataset asks two questions of any AI finance tool. Did it find the right number? Did it do the right math? Most tools that score well on the first fail the second, because the calculation is being done by a language model that predicts what a number should look like rather than computing it.

That is the architectural point underneath all of this. If the math runs inside the LLM, your pass rate will always be a probability, and it will drift. The only durable fix is to take calculation away from the model entirely.

This is how Rexfin is built. We connect your accounting platforms or uploaded statements, reconcile them into one model that ties out to the ledger, and then let AI retrieve figures and run every calculation through a deterministic engine, not the language model. Same inputs, same answer, every time, with a line back to source. Run your golden dataset against a system designed that way and the calculation cases stop being the ones that fail.

Build the dataset before you buy anything. Then book a demo and point your hardest questions straight at us. If the numbers don’t tie out to your ledger, you should walk. We would rather you test us than trust us.