Black-Box AI

Black-box AI describes a system whose outputs you can’t trace back to the data or logic that produced them. You get an answer; you don’t get the path that led there, so you have no way to check it before you act on it.

Why opacity is a control failure, not just a UX gap

Finance runs on reproducibility. A figure in a board pack is expected to survive being asked “where did that come from,” and the answer has to be a record, not a description of a process. A black-box system breaks that expectation at the point it matters most: the number is available, the justification is not.

That shows up as a control failure in three places. In audit, a figure nobody can reproduce is a finding regardless of whether it happened to be right. In close, an unexplained variance cannot be signed off, so the team rebuilds the calculation by hand and the automation saves nothing. In board reporting, the CFO personally owns a number whose derivation they cannot inspect.

Explainability theater

Every finance AI vendor now claims to have solved this, usually by adding a reasoning trace or a confidence score. Both are worth having and neither is a control. A reasoning trace is the model’s narration of its own process, which is generated text like everything else and can describe steps the system never took. A confidence score is a probability, and a probability attached to a single reported figure tells a reviewer nothing actionable: an 87 percent confident revenue number is either right or wrong, and the score doesn’t say which.

The useful distinction is explainability vs. verifiability. Explainability shows the path. Verifiability proves the destination.

What a checkable system looks like

Ask what happens when a check fails. In a verifiable system, the figure isn’t rendered at all; in an explainable one, it ships with a caveat. Ask whether arithmetic is computed or generated. Ask whether every displayed number links to a source record, not to a summary of one. And ask whether the lineage survives export, or whether it only exists inside the vendor’s interface.

Rexfin reconciles sources into one governed model, runs calculations deterministically rather than generating them, and cites every figure to the record that produced it, so a reviewer can open the underlying document instead of reading a description of it. See the buyer’s checklist for spotting a black-box forecast.