How Rexfin Measures Answer Quality (Instead of Just Claiming It)

Every AI finance vendor says some version of “you can trust our answers.” Almost none of them can tell you how they know that, beyond a demo that went well once. That’s the gap this piece is about: the difference between asserting reliability and measuring it, continuously, in a way that a bad update can’t quietly slip past.

Rexfin’s core promise for its AI answers is simple to state and hard to fake: every numeric claim is either backed by a citation that actually supports it, or the AI refuses to answer. We don’t take that promise on faith internally either. We run it as a graded test, the same way you’d test any other piece of software that has to behave correctly under pressure.

What actually gets scored

The test set is a library of real questions asked against real, seeded filings: the kind a CFO or an analyst would actually ask, split into categories like balance sheet figures, cash flow, ratios, disclosure lookups, and a deliberate category of questions the filing genuinely doesn’t answer. Each question carries a hand-captured correct answer: the right value, the exact page it came from, and, for questions the filing doesn’t cover, confirmation that the figure really is absent, not just hard to find.

Every answer the AI produces gets checked on more than one axis at once. Is the number right? Is it cited, and does that citation actually point to the evidence that supports it? A correct number with no traceable citation counts as a failure, not a pass, because an unsupported number is exactly the failure mode this whole system exists to prevent. And for the questions with no real answer in the filing, did the AI correctly refuse, or did it confidently invent a figure? That last case, a fabricated number on a question that should have been refused, is treated as the single worst outcome the harness can detect, because that’s the number that ends up wrong in a board pack.

Why a blended score would hide the problem

A single “92% accurate” headline number can hide a lot. It’s entirely possible for income-statement lookups to carry a whole test set to a respectable-looking average while cash-flow questions, which require pulling together several figures correctly, quietly perform much worse underneath. So results are broken out by category, not just reported as one number, and the weakest category is called out explicitly rather than averaged away. That matters in practice: a diligence team asking specifically about cash-flow figures gets an answer about cash-flow figures, not a number blended with easier questions.

The harness also checks a layer beyond exact numbers: whether the AI’s surrounding commentary, the narrative sentences around a figure, not just the figure itself, actually follows from the evidence it cited, rather than adding an unsupported claim dressed up in confident language. That’s the failure mode competitors’ “AI insights” features are most exposed to, because a narrative claim doesn’t fail as loudly as a wrong number does.

It runs on every change, not just once

The point of measuring this isn’t a one-time report to put in a deck. It’s wired into the same process that ships changes to the product: a change to retrieval, prompts, or the underlying model has to run against this test set before it merges, and if citation accuracy or refusal correctness drops below an agreed floor, that change doesn’t ship. A rewrite that quietly makes the AI less willing to refuse, trading a safe “I don’t have that” for a confident guess, gets caught here before it reaches anyone’s board pack, not after.

The honest limits

This doesn’t mean the AI never makes a mistake: it means mistakes are measured, tracked by category, and any regression is caught before release rather than discovered by a customer. It also relies on that test set staying grounded in real filings and genuinely absent figures, which is its own discipline. See the canonical atom store for how the underlying facts are kept anchored to source documents in the first place, and calibration and trust tuning for how the same measured, evidence-first approach governs when a check is allowed to block an export.

The takeaway

“Trust our AI” is a sentence. A test suite that scores citation accuracy and refusal behavior on every change, and blocks a release that regresses either, is evidence. If you want to see what happens when an answer’s evidence and verdict get preserved for later, read the answer audit log, or head to the inside-the-rexfin-platform hub for how it all fits together.