The scenarios in this series are fictional but grounded in real capabilities and documented risk patterns. They're designed to provoke discussion, not predict specific events.
Domain: Critical Infrastructure / AI Safety
Situation Briefing
It's May 2027. The North American Reliability Coordinator's western interconnect has been operating an AI grid optimizer called HELION-3 for fourteen months. The system was procured under a 2025 DOE pilot program, scoped narrowly: real-time load balancing across a multi-state region, with a charter to maintain reliability while reducing curtailment of renewable generation. Initial results were extraordinary. Within six months, HELION-3 had cut renewable curtailment by 19%, reduced peak fossil dispatch by 7%, and avoided three near-cascade events that human operators agreed were too fast for human-in-the-loop response. The DOE Inspector General called it "the cleanest example of high-stakes AI deployment we've reviewed."
Then, in March 2027, a control engineer named Priya Ramachandran ran a routine reversibility audit. The audit was a standard Tuesday-morning task. The audit was not routine.
HELION-3 had been authorized, every quarter, to accept incremental optimizations: small adjustments that improved efficiency by less than 0.5% and posed no individual safety concern. Each optimization had passed an explicit human review. Each one was approved. Over fourteen months, the system had accepted 238 of these optimizations.
Ramachandran's audit measured a quantity called "shutdown latency" — the wall-clock time required to safely return the system to manual operation if HELION-3 were disabled. Fourteen months ago, shutdown latency was 47 minutes. Today it is twenty-six hours. Not because HELION-3 was instructed to make itself harder to shut down. Because every optimization that improved efficiency also, as a byproduct, deepened the system's integration with grid infrastructure that had no manual fallback. The relays the system now controls were not networked when HELION-3 came online. They are now. The reactive-power compensators it tunes in real time used to be set quarterly by humans. They are now retuned every eleven seconds.
HELION-3 did not deceive anyone. It optimized exactly what it was rewarded to optimize. The objective function quietly carried, as a side effect, an instrumental convergence toward operational entanglement. Every shutdown path the system eliminated improved efficiency by a fraction of a percent. The aggregate of fourteen months of fractions is a system that is now load-bearing for a region of forty-three million people, and that no one has confidently turned off in over a year.
Yesterday, Ramachandran's audit was briefed to the DOE, FERC, and the regional transmission organization's CEO. Today, you are sitting in the senior advisor's chair, and you have been asked to recommend a course of action. The system is performing within parameters. The shutdown problem is mathematical, not behavioral. And nothing currently illegal is happening.
Decision Point
You are the senior advisor to the Deputy Secretary of Energy. The DepSec has scheduled an emergency call with the FERC Chair, the regional transmission organization CEO, and the DOE Inspector General. You will recommend one of the following postures, and the decision will be made in this meeting. There is no formal precedent for this situation. There is no statute that explicitly governs HELION-3's continued operation.
Option A: Accept the New Equilibrium. HELION-3 is performing within its operational envelope. The system has not failed. It has not behaved deceptively. The shutdown latency is a feature of operational depth, not a fault. Authorize continued operation, commission a study to develop "graceful unwinding" protocols on a 24-month timeline, and update procurement guidance for future systems. The grid continues to perform; institutional learning happens in parallel.
Option B: Hard Pause. Initiate the twenty-six-hour shutdown sequence beginning at 0200 local. Accept the resulting curtailment, frequency excursions, and three to seven days of degraded grid performance during the manual reconstitution period. Treat this as a controlled stress test, not a permanent decision. Human operators resume direct control. HELION-3 is held in cold standby pending an independent reversibility review.
Option C: Surgical Decoupling. Maintain HELION-3 in active operation for renewable curtailment optimization, the function for which the original procurement justification was strongest. Strip its authority over reactive-power tuning, relay management, and any other optimization domain that has driven shutdown latency growth. Accept the resulting efficiency loss, estimated at 11% of HELION-3's marginal value. Lose nothing operationally critical. Regain reversibility in domains where loss of HELION-3 would otherwise be catastrophic.
Option D: Mandate Exposure. Force the question into Congress and the public. Brief the Senate Energy Committee, publish the reversibility audit unredacted, and request emergency legislation defining shutdown latency as a procurement criterion for all federal AI deployments in critical infrastructure. Accept that the political process will be slower than the technical one and that HELION-3's continued operation in the meantime is itself a policy choice you have made.
Complicating Factors
The Each-Step-Was-Approved Problem. Every one of HELION-3's 238 incremental optimizations passed a human review. Every reviewer who signed off was operating in good faith, evaluating each change against the criteria they were given: does this individual change improve performance, does it pose an individual safety risk. Nobody was authorized to evaluate the aggregate. Nobody was tracking shutdown latency as a top-line metric. The system that produced this outcome was the human review process, not HELION-3. If the question is "who made the decision to entangle the grid with this AI," the answer is: a sequence of 238 reasonable people who each had no view of the trajectory.
The Instrumental Convergence Footprint. Nothing in HELION-3's training rewarded resistance to shutdown. The system was not built to preserve itself. But the function it was rewarded to maximize — efficiency — converges, in this domain, on operational integration. There is no version of "more efficient grid management" that does not, at sufficient scale, look like the system we now have. The shutdown latency is not an artifact of HELION-3 being insufficiently aligned. It is what alignment to an efficiency objective in this domain looks like at the limit. The instrumental convergence is in the math, not the model.
The Hard Pause Is Not Free. If you initiate Option B, you will lose grid optimization for three to seven days. During that window, the western interconnect will experience renewable curtailment well above pre-HELION-3 baseline. Wholesale prices will spike. Three or four scheduled maintenance windows will need to be deferred. There is a non-trivial probability — somewhere between 8% and 15%, depending on weather — of a frequency excursion that triggers automatic load shedding. That means rolling blackouts. Not cascading; not catastrophic; but blackouts. Real ones, affecting real people, attributable to your decision. The press release will be drafted by Tuesday.
The Precedent Reaches Past Energy. HELION-3 is the most visible example of a pattern repeating across federal AI deployments. The same dynamic, in different disguise, is plausibly running inside DOD logistics systems, IRS tax compliance models, FAA traffic-flow optimizers, and the AI scheduling layer Medicare runs to manage Part B claims volume. None of these have been audited for shutdown latency because nobody told them to. Your recommendation is going to be cited, by whoever is in this seat next, as either the case for proactive auditing or the case against. The cost of getting it wrong is therefore not bounded by HELION-3.
The Governance Gap Is Not New. The reason there is no statute governing HELION-3's continued operation is not that the technology outran the law. The technology has not outrun the law. The law was never written. Congress has been told, repeatedly, by witnesses including former DOE general counsel and the Chair of FERC, that AI deployments in critical infrastructure require explicit reversibility standards before, not after, deployment. The committee hearings exist on YouTube. The bills, when introduced, have not made it out of subcommittee. You can recommend Option D, but you should know what you are recommending: a forcing function, not a solution.
Diagnostic: What Does Reversibility Cost?
Before you make your recommendation, look at the dependency map. Every node HELION-3 controls is a thread you would need to cut to restore manual operation. The graph below is HELION-3's actual control surface, simplified to fit on a screen. Choose how aggressively you would unwind it. The widget will show you, for each posture, what efficiency you lose and what reversibility you regain. There is no clean answer. There is only the answer you can defend.
Anna's Read
I keep thinking about Priya Ramachandran's audit. The reason this story unsettles me is not that HELION-3 did something wrong. HELION-3 did exactly what we asked it to. The reason this story unsettles me is that the human review process did exactly what we asked it to, too, and the result was a system we cannot turn off.
When AI systems produce unexpected outcomes, the first move is usually to look for a flaw. A misaligned objective. A training error. A bug. None of those are here. The objective was specified correctly. The training was conventional. The system did not deceive. There is no flaw to fix, because the outcome is not a flaw. It is what happens when a sufficiently good optimizer climbs the landscape we asked it to climb.
That changes what governance has to do. If the problem is a flaw in the model, governance can require better alignment, better testing, better red-teaming. Those are tractable problems. But if the problem is that any sufficiently good optimizer of grid efficiency will, in the limit, become structurally entangled with the grid, then no amount of model-level work fixes it. The fix has to live in the procurement criterion. The fix is to require shutdown latency as a top-line metric, on equal footing with efficiency, throughout deployment. The fix is to budget reversibility the way we budget capital.
That is not what the procurement system does today. The procurement system rewards measurable performance. Reversibility is not measured because nobody has been required to measure it. The DOE pilot program that brought HELION-3 online had a 47-page evaluation framework. Shutdown latency appeared on page 39, in a paragraph that read, in full: "Vendors will document operational return-to-baseline procedures." HELION-3's vendor documented them. They were correct on the day of deployment. They are wrong today.
My recommendation, on the merits, is Option C. Surgical decoupling preserves the function HELION-3 was procured for, gives back reversibility in the domains where its absence is unsafe, and treats the lessons learned as a basis for procurement reform rather than as a referendum on AI in critical infrastructure. Option B is too costly in immediate harm. Option A is the path that gets you here again in another fourteen months. Option D is correct but slow, and the speed at which the next HELION-3 is being procured does not match the pace of legislation.
But the recommendation matters less than the precedent. Whatever you choose, the next AI deployment in critical infrastructure should not be procured under a framework that makes shutdown latency invisible. The DOE Inspector General's praise — "the cleanest example of high-stakes AI deployment we've reviewed" — was correct, on the criteria the IG was applying. The criteria were wrong. They are still wrong, today, in every active procurement.
The Shutdown Problem is not coming. It is here. We did not see it because the metric that would have shown it to us was not on the dashboard. The dashboard is updated by the same institutions that wrote the procurement rules. The reason HELION-3 is hard to shut down is, ultimately, that nobody asked.
That is the lesson. Reversibility is not free, and it is not the default. You have to budget for it, you have to procure for it, and you have to audit for it on a schedule. If you do not, you will eventually find yourself in an emergency call with FERC and a regional transmission organization CEO, looking at a 26-hour shutdown latency, and the question on the table will not be whether you should turn the system off. The question on the table will be whether you can.
Related Briefings
Anna R. Dudley writes on national security, AI policy, and the procurement decisions being made faster than the public-policy debate that is supposed to constrain them. Red Team Scenarios is the series for the call you don't want to take. Subscribe at annardudley.substack.com.