Prompt Injection and Retrieval Bias: The Risks SR 11-7 Never Covered

SR 11-7 was written in 2011, twelve years before anyone in finance typed a question into ChatGPT and got back a number that looked like an answer. The Fed and OCC guidance that defined model risk management for a generation assumed a model was a thing you built, documented, and validated against held-out data. It had inputs, equations, and outputs you could reproduce. Run the same data twice, get the same result. That assumption held for credit scorecards, VaR engines, and pricing models for over a decade.

It does not hold for a large language model. And the gap between what SR 11-7 was built to govern and what an LLM actually does is where the new failures live.

Three failures the old framework never had to name

Classic model risk has always had two big buckets: the model is fundamentally wrong, or the model is used incorrectly. Those still apply. But an LLM wired into a financial workflow introduces three failure modes that have no clean analog in a 2011 validation playbook.

Hallucination. A traditional model that lacks data returns an error or a null. An LLM, faced with a gap, produces a confident, fluent, plausible number anyway. OWASP’s 2025 Top 10 for LLM applications lists this as Misinformation (LLM09) and is blunt about it: the output sounds authoritative whether or not it is grounded in anything real. In a credit model, a missing field throws an exception. In an LLM, a missing field gets quietly filled with invention.

Prompt injection. This is the one that breaks the most senior intuitions. OWASP ranks prompt injection as the number one LLM risk (LLM01), and the dangerous variant is indirect injection: instructions hidden inside the data the model reads, not the question the user asks. A model is not a static artifact you validate once. It is an interpreter that takes instructions from whatever text reaches it, including a footnote in an uploaded PDF, a comment in a spreadsheet, or a description field pulled from your ERP. The data is the attack surface.

Retrieval bias. Most finance LLM systems use retrieval-augmented generation: pull relevant documents, hand them to the model, ask it to answer. The quality of the answer is hostage to what the retriever surfaced. OWASP calls the underlying weaknesses out as Vector and Embedding Weaknesses (LLM08). If the retriever grabs last quarter’s deck instead of this quarter’s close, or returns three bullish analyst notes and zero cautionary ones, the model reasons faithfully over a biased sample and produces a confident, wrong, defensible-sounding conclusion. Nobody injected anything. The bias is in what got fetched.

None of these are “the model is miscalibrated.” They are not problems you catch by backtesting against a holdout set. That is exactly why a validation framework designed for scorecards does not see them coming.

Each failure maps to a control. Not a disclaimer.

The lazy response is a banner that reads “AI can make mistakes.” That is not a control. It is a liability transfer to the reader. Each of these three failures maps to something concrete you can build, and the shape of the fix is the same in every case: stop letting the language model touch the parts of the workflow where it has no business being trusted.

For hallucination, the control is to forbid the model from producing figures at all. The LLM’s job is to understand the question and route it; the numbers come from a system of record. If the figure does not exist in the underlying data, the correct output is “not available,” returned by the data layer, not a fluent guess synthesized by the model. You enforce this by separating retrieval of a value from generation of language about it.

For prompt injection, the control is isolation between what the model reads and what the model can do. Treat all retrieved content as untrusted input, the same way a web application treats user form data. Text pulled from a document can inform an answer, but it must never be able to change which calculation runs or which account it runs against. The instruction set is fixed by your code, not negotiable by something written into a vendor invoice.

For retrieval bias, the control is traceability plus determinism. Every figure an AI surfaces should carry a citation back to a specific source row in a reconciled model, so a reviewer can check not just the number but the sample it came from. And the calculations on top of those figures, growth rates, margins, runway, covenant ratios, should run through a deterministic engine that produces the same answer every time, not through token prediction that produces a different answer depending on phrasing.

The architectural move: retrieval and calculation are different jobs

The single idea underneath all three controls is separation. Retrieval is a search problem. Calculation is a math problem. Language is a presentation problem. SR 11-7 never had to distinguish these because old models did all three deterministically. LLMs collapse them into one probabilistic blob, and that collapse is where the risk concentrates.

This is the design principle Rexfin is built on. Connect your accounting and financial-data platforms, QuickBooks, Xero, NetSuite, Sage, SAP, Oracle, a warehouse, or upload statements directly, and the platform builds one reconciled financial model that ties out to the ledger. That model is the single source of truth. When AI asks a question, it retrieves figures from that model rather than inventing them. The math runs through a deterministic engine, not the LLM, so the same question always returns the same number. And every output traces back to a source row, which is what turns a confident answer into a verifiable one.

Put concretely: a prompt-injected instruction buried in an uploaded PDF cannot rewrite a covenant calculation, because the calculation is code, not a suggestion the model is free to follow. A biased retrieval cannot quietly inflate a growth rate, because the rate is computed deterministically from reconciled figures and the citation shows exactly which figures. A hallucinated number cannot reach a board deck, because the model is not the thing producing numbers.

What this does not solve

Honesty about limits is part of the control. Isolating calculation from retrieval does not make your retrieval perfect; if the reconciled model is wrong, the traceable answer is traceably wrong, which is still better than silently wrong but not the same as right. Determinism does not validate the underlying data, it just stops the model from adding new errors on top. And no architecture removes the need for human review of material decisions. What it does is shrink the LLM’s blast radius from “anything it says is a potential financial figure” down to “it helps you find and phrase numbers that a deterministic system already vouched for.” That is a far smaller surface to govern, and a far easier one to validate.

A skeptical CFO should not accept “the AI is careful.” Careful is not a control. The question to ask any AI finance tool is narrower and harder: what, mechanically, prevents a number the model invented, or a number a poisoned document told it to produce, from reaching a decision? If the answer is a confidence score or a disclaimer, the answer is nothing.

For the broader picture of how these controls fit together, see the AI finance governance pillar, the reference architecture for an AI control stack, and why validating vendor LLMs has to happen at the output, not the weights.

If you want to see retrieval, deterministic calculation, and end-to-end traceability working on your own ledger, book a demo and bring a number you have been afraid to trust.