The scenarios in this series are fictional but grounded in real capabilities and documented risk patterns. They're designed to provoke discussion, not predict specific events.
Domain: Capital Markets / Model Risk
Situation Briefing
In October 2027, ATLAS-FX produces a 0.94-confidence signal that drives $14 billion in reallocations at one of the largest U.S. asset managers. The signal is wrong. The deeper problem is that six of its eleven inputs trace back to echoes of the model's own prior outputs.
The model's track record was real until the information environment changed around it. Its 87 percent directional accuracy reflected a world where upstream signals were independent enough to trust. Once its own outputs started re-entering the market-data stream, confidence became recursive.
By Friday, a model-risk analyst can reconstruct the contamination chain. By Monday, the CRO needs a posture for the board. The question is not whether ATLAS-FX hallucinated. It is whether the firm can still characterize the data environment it is trading on.
Decision Point
Option A: Tighten the threshold. Raise the auto-approval bar and add same-day review. This is fast, but it assumes confidence remains meaningful when lineage is broken.
Option B: Build the lineage audit. Keep the system in production only with source-provenance tracing for large signals. This is slower and more expensive, but it addresses the actual failure.
Option C: Suspend ATLAS-FX. Remove the immediate risk and accept a major operational hit. This protects the firm while proving it did not understand the dependency until after the loss.
Option D: Disclose and continue. Tell clients the model is being reviewed while preserving the trading edge. This is the most fragile posture if another recursive signal appears.
Complicating Factors
The track record was not fake. That is why the failure is hard. A model can be excellent in one signal ecology and dangerous in another.
Confidence was calibrated to the wrong world. ATLAS-FX measured agreement among sources without knowing that some sources had become descendants of its own earlier judgments.
The governance artifact is missing. The firm has model-risk documentation, approval thresholds, and human review. It does not have lineage evidence strong enough to tell whether a high-confidence signal is independent.
The client story depends on the architecture. Without a lineage audit, disclosure becomes a promise to be more careful inside the same blind spot.
Diagnostic: Where Did the Confidence Actually Come From?
Before you finalize the recommendation, walk through ATLAS-FX's confidence lineage on the original signal. The system reported 0.94. It corroborated across eleven sources. The widget below lets you tag each source as independent or contaminated and shows you what the confidence number would have looked like if the audit had run pre-trade. There is a lesson here about confidence numbers in the AI era. The lesson is not that they are wrong. It is that they describe the model's view of its own evidence. They do not describe the model's view of whether the evidence is what it appears to be.
Anna's Read
Elena Voss is the important person in the story because she found the failure the model could not name. The model did not lie. It measured confidence in a contaminated environment and called the measurement truth.
Thresholds will not fix that. A higher cutoff only makes the firm more confident in the same lineage problem. Suspension may be necessary briefly, but it is not a governance architecture.
My recommendation is B: build the lineage audit and make large-signal approval depend on source independence, not only model confidence. The firm cannot outsource market epistemology to a score.
Related Briefings
Anna R. Dudley writes on national security, AI policy, and the institutional structures absorbing the costs of AI deployment faster than they are being redesigned. Red Team Scenarios is the series for the call you don't want to take. Subscribe at annardudley.substack.com.