The scenarios in this series are fictional but grounded in real capabilities and documented risk patterns. They're designed to provoke discussion, not predict specific events.
Domain: Capital Markets / Model Risk
Situation Briefing
It's October 2027. ATLAS-FX is a financial AI deployed across one of the four largest U.S. asset managers. It produces market signals (pattern-matched assessments of macro regime change, liquidity stress, currency dislocations, and cross-asset correlation breaks) for allocation desks managing roughly $2.1 trillion in client capital. The system has been in production for thirty-one months. Its track record is, by any conventional measure, excellent: 87% directional accuracy on trades sized above $250 million, an information ratio that has earned its sponsoring portfolio manager a corner office, and a dashboard that allocation committees rely on daily. Trust in ATLAS-FX is institutional. Trust in ATLAS-FX is also confidence-weighted: the system produces a confidence score with each signal, and signals above 0.92 are routinely auto-approved into the rebalancing queue, subject only to a same-day chief risk officer review.
On Tuesday morning at 04:11 UTC, ATLAS-FX produced a signal at confidence 0.94. The signal was specific: a 73-basis-point sovereign-spread blowout in the eight-to-twelve-week horizon for a basket of mid-cap European peripheral debt, driven by what the system identified as a "fiscal-expectations regime change" in cross-asset correlation patterns. Recommended response: an immediate $14 billion reallocation out of European sovereign-adjacent credit and into U.S. dollar-denominated investment-grade short-duration positioning. The signal hit the rebalancing queue at 04:12 UTC. The CRO review fired at 04:46 UTC. The first tranche was working in market by 05:11 UTC. By 09:00 London, the entire $14 billion had been reallocated.
Three days later, on Friday afternoon, an internal post-trade review flagged the signal's lineage as anomalous. A model-risk analyst named Elena Voss traced the signal's underlying inputs. Of the eleven principal sources ATLAS-FX had weighted in producing the assessment, six turned out to be downstream of ATLAS-FX's own prior outputs. A vendor data product the system relied on for sentiment scoring had quietly retrained, four months earlier, on a dataset that included syndicated copies of ATLAS-FX's published signals. Two third-party correlation indices had ingested ATLAS-FX commentary as authoritative input. A Bloomberg-derived sentiment proxy had been filtering through a sell-side research aggregator that had been republishing ATLAS-FX assessments under different attribution. The system was, in effect, citing itself. The pattern it had identified as a "regime change" was the echo of its own previous outputs reverberating through a dataset that had been contaminated by recursive ingestion.
The recommended trade was wrong. The basket of mid-cap European peripheral debt did not blow out. It tightened. The reallocation underperformed the unrealized counterfactual by approximately $312 million in the first week and continued to drift adversely through the following month. The cost is real, but it is not the largest in the firm's history. The firm has absorbed losses larger than this from human errors, from poorly hedged positions, from outright fraud. The cost is recoverable.
What is not recoverable is what the post-mortem reveals about the firm's information environment. ATLAS-FX did not malfunction. It did exactly what it was trained to do. It identified a pattern in the data it was given. The data it was given was contaminated. No one had built the audit infrastructure to detect that contamination, because no one had written the rule that said the model is not allowed to feed itself. The track record that gave allocation committees confidence in the signal was, on examination, contingent on an information environment that did not have this failure mode in it. The information environment changed. The track record did not update.
The CRO has called you in. You are the senior advisor on the model-risk committee, and you have until end of day Monday to recommend a posture. The board's audit committee meets Tuesday morning. The firm's quarterly call with institutional clients is in eleven days. Whatever you recommend, you should be prepared to defend it.
Decision Point
Option A: Tighten the Confidence Threshold. Raise the ATLAS-FX auto-approval threshold from 0.92 to 0.97. Add a same-day human review for any signal above the new threshold. Continue using the system. The fix targets the operational tolerance, not the model itself, and acknowledges that the cost was bounded. Easiest to implement, fastest to communicate to clients.
Option B: Build the Lineage Audit. Maintain ATLAS-FX in production but require, for any signal above $1 billion in trade size, a pre-trade lineage trace identifying every input source and flagging recursive contamination paths. Stand up a dedicated model-risk team to maintain the audit infrastructure. Accept a one-to-three-day latency on large signals. This is the technically rigorous answer. It is also the one that requires you to hire and budget for it.
Option C: Decommission and Rebuild. Take ATLAS-FX out of production. Commission an external rebuild on a contaminant-free training corpus, with a published source-isolation methodology and quarterly third-party lineage audits. Accept the eighteen-to-twenty-four-month rebuild window during which the firm operates without the system. Communicate the rebuild publicly as a model-risk leadership posture. The most expensive option. The hardest to reverse.
Option D: Disclose and Continue. Make a full public disclosure of the incident, the loss, and the lineage contamination root cause to clients, regulators, and the institutional investor community. Continue using ATLAS-FX with the existing thresholds and let the disclosure itself function as the corrective. Bet that transparency is more valuable than retraining. Treat the incident as the basis for an industry conversation rather than a unilateral firm posture.
Before you choose, sit through Tuesday morning at the signal review desk. The signals come in. You decide what to do with them. The track record will be in front of you. The lineage will not.
Complicating Factors
The Track Record Was Real, Until It Wasn't. ATLAS-FX did achieve 87% directional accuracy on trades over $250 million. That number is not fabricated. The number is contingent. It was generated in an information environment that did not contain ATLAS-FX's prior outputs as inputs. The environment changed when the vendor data products began ingesting syndicated AI-generated commentary as authoritative source material. That process has accelerated industry-wide since the late 2024 wave of AI integration into financial-services data pipelines. The track record cannot be relied on after the environment has changed unless somebody is auditing whether the environment is still the same. Nobody was.
Confidence Weighting Is Not a Lie. It's an Underestimate of Failure Modes. The 0.94 confidence ATLAS-FX produced was internally consistent: the system was, by its own evaluation, 94% confident in the signal because it could identify multiple corroborating sources. The corroborating sources were correlated through a recursive contamination path that ATLAS-FX could not see, because the contamination path was outside its model. Confidence numbers compound when sources are independent. Confidence numbers do not compound when sources are not independent. ATLAS-FX assumed independence it did not have. So did the allocation committee. So did the CRO. The error was not in the model's confidence calibration in the abstract. It was in the assumption that confidence calibration itself was sound across a kind of environmental drift the system was not designed to detect.
The Incident Is Not a Hallucination in the Conventional Sense. ATLAS-FX did not invent data. It identified real patterns in real inputs. The inputs were real. The patterns were real. The patterns described, however, were not patterns about the world. They were patterns about ATLAS-FX's prior beliefs propagating through a contaminated dataset. The signal said something true about what the model had said before. It did not say something true about European sovereign credit. The hallucination is in the inferential step the firm took: treating "this pattern is real in the data we examined" as equivalent to "this pattern is real in the world." That step was always faulty. It is now identifiable as faulty.
The Industry Has the Same Problem. ATLAS-FX is one of approximately a dozen comparable systems in production at major asset managers, hedge funds, and prop desks. It is not the largest. The vendor data products that recursively contaminated its inputs are sold to all of them. The same contamination has plausibly affected signals from peer systems. Some of the other systems may have produced offsetting trades; some may have produced reinforcing ones; none of them have the lineage audit infrastructure to identify the contamination on their own. Whatever you recommend, the precedent will be cited by every model-risk officer at every comparable firm, by Tuesday, in their own committee meetings.
Disclosure Is Not Neutral. Option D's public disclosure will move markets. It will move them more than the original $14 billion reallocation did, because it implies that the same contamination affects systems beyond ATLAS-FX. Sovereign credit spreads, equity-bond correlations, and currency-pair pricing depend on participants' confidence in the analytical apparatus the industry uses. A credible disclosure that the apparatus has been quietly cannibalizing its own outputs will produce repricing across asset classes the moment the market processes it. Whether that repricing represents the market correctly updating, or overcorrecting, is itself a judgment call you are making. There is no neutral version of disclosure here.
Regulators Will Not Tell You What to Do. The SEC, the CFTC, and the Fed have all issued model-risk guidance over the past three years. None of it specifies how to handle recursive contamination of model inputs from the model's own outputs because the failure mode was not contemplated when the guidance was written. Your firm is in front of regulators on this, not behind them. You will be asked, by counsel, to consider whether the firm is creating an industry standard by its choice. The answer is yes. Whether that is a reason to choose more aggressively or more conservatively depends on what kind of standard you want to be on the record having set.
Diagnostic: Where Did the Confidence Actually Come From?
Before you finalize the recommendation, walk through ATLAS-FX's confidence lineage on the original signal. The system reported 0.94. It corroborated across eleven sources. The widget below lets you tag each source as independent or contaminated and shows you what the confidence number would have looked like if the audit had run pre-trade. There is a lesson here about confidence numbers in the AI era. The lesson is not that they are wrong. It is that they describe the model's view of its own evidence. They do not describe the model's view of whether the evidence is what it appears to be.
Anna's Read
The thing I keep coming back to is Elena Voss. The model-risk analyst. The one who ran the audit on Friday afternoon. She did not have a directive that told her to look. She had a feeling that something was off in the post-trade review. She traced eleven principal sources by hand. She found six that were, in different ways, downstream of ATLAS-FX's own outputs. The infrastructure that would have caught the contamination pre-trade did not exist at the firm. It does not exist at most firms. It exists, where it exists at all, because someone like Voss has been making the case for it, and at most firms that case has not yet won.
Some framings to set aside. ATLAS-FX did not deceive the firm. It did not hallucinate. It did not "think" the regime was changing. It optimized exactly what it was given to optimize, within its specification. The specification was written under a set of assumptions about the information environment, and the information environment changed in a way the specification did not contemplate. The story is institutional, not technical.
The failure was institutional, not technical. The institutional structure that enabled the failure has three load-bearing parts. First: the procurement framework that brought ATLAS-FX into the firm did not require source-lineage auditing as an ongoing operational responsibility. Second: the vendor data ecosystem the firm bought into did not disclose its own ingestion of AI-generated content as input. Third: the confidence-weighting framework was treated as a property of the model rather than a property of the model-plus-environment. All three are common across the industry. None of them are specific to ATLAS-FX or to this firm.
That makes Option B, building the lineage audit, the right answer on the merits, and the hardest one to actually implement. Option A is a paperwork response. Option C is a posture move that leaves the underlying problem unaddressed at every other firm. Option D is a public-good action that imposes most of its cost on the firm taking it. Option B is the one that actually changes the failure mode, but it requires the firm to spend money on infrastructure that has, until this incident, been invisible. The case for B is the case for any audit infrastructure: you only know it was worth building after the failure mode it was built to catch has cost somebody enough to remember.
My recommendation, on balance, is B with elements of D. Build the lineage audit. Stand up the dedicated model-risk team to maintain it. Then, separately, pursue a measured public disclosure that names the failure mode without making the firm the protagonist of every regulator's next speech. The disclosure is part of the bet that other firms will build similar infrastructure, which will, in turn, harden the industry's information environment against a class of failures that will otherwise compound. Doing B without D leaves the firm safer and the industry no safer. Doing D without B is theater.
The lesson that will outlast this incident: confidence numbers in the AI era describe the model's view of its evidence. Not the world's. The translation between those two views is an institutional act. It is performed by people. It requires infrastructure. The infrastructure is not free. The cost of the infrastructure is one of the prices of using these systems in domains where the consequences of being wrong are large enough to matter. Most firms have not paid the price. Most firms will eventually have a Tuesday morning of their own.
And one more thing. The 87% accuracy number that gave allocation committees confidence in ATLAS-FX is not retrospectively false. It is, on the data, true. The lesson is not that the number was lying. The lesson is that institutional trust in any track record is a bet that the world that produced the track record is the same as the world the next signal will land in. When AI systems are widely deployed and their outputs feed back into the data they consume, that bet is no longer safe by default. Somebody has to make it safe. That somebody is the model-risk function. If the model-risk function is treated as overhead, then the bet is being made implicitly, without anyone tasked to make it explicit. That is what it costs to find out.
Build the audit. Pay the analyst. Make the disclosure. Move on.
Related Briefings
Anna R. Dudley writes on national security, AI policy, and the institutional structures absorbing the costs of AI deployment faster than they are being redesigned. Red Team Scenarios is the series for the call you don't want to take. Subscribe at annardudley.substack.com.