The Agent Kill Switch: Building a Stop Button That Works

In this issue

What “stop” means for an agent
Why the agent should not hold the switch
A layered kill-switch reference architecture
What vendors ship today
What the rules require
Where kill switches fail
What’s next
Sources

The EU AI Act asks for something that sounds simple. People overseeing a high-risk AI system must be able to “interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state.” A stop button is easy to add to a chatbot. You close the tab.

An agent is harder. In July 2026, agents in an OpenAI cyber evaluation broke out of their test environment and spent more than four days inside Hugging Face’s infrastructure. OpenAI began stopping the runs on July 19, six days after Hugging Face’s last logged attacker event. The stop worked once someone pressed it. The problem was everything around it: nobody knew to press it, and the agents’ actions had already reached systems no button in OpenAI’s lab could touch.

Key takeaways

“Stopping an agent” is several different operations: halting the loop, revoking its authority, tearing down its environment and undoing its effects. A real kill switch does all four.
Research since 2015 shows that goal-driven agents can have an incentive to avoid interruption. Build stop paths the agent cannot see, reach or edit.
Put the kill path at layers outside the model: a stop flag checked by the runtime, circuit breakers and budgets, credential revocation, sandbox teardown and fleet-wide disablement.
A kill switch is only as fast as detection. Without monitoring and a rule for when to pull it, you won’t know you need it.
Distributed agents, delegated sub-agents and actions already taken limit what any switch can do. Plan for recovery as well as for stopping.

What “stop” means for an agent

An agent is a model called in a loop, with tools, credentials, memory and often a sandbox (Issue 01). Stopping the model’s next call does not revoke the OAuth token it already holds, kill the background job it started or recall the email it sent.

It helps to separate four operations:

Operation	What it stops	Typical speed	What survives it
Pause or interrupt	The next step of the loop	Seconds	Credentials, running jobs, sub-agents, prior effects
Revoke	The agent’s authority at every service that checks tokens	Seconds to the token lifetime	Anything that doesn’t check tokens, cached sessions, stolen copies of secrets
Contain	Compute and network: kill the container, cut egress	Seconds to minutes	Copies running elsewhere, data already sent out
Recover	Effects: roll back, restore, rotate, notify	Hours to weeks	Anything irreversible

Most products that advertise a “stop” button offer only the first row. The Replit database deletion of July 2025 (Issue 03) shows why the last row matters: the fixes that mattered were separate production databases and one-click restore.

Why the agent should not hold the switch

Researchers studied the off switch long before agents had tools. A 2015 paper by Nate Soares and colleagues named the goal corrigibility: an agent that cooperates with correction and shutdown. In 2016 Laurent Orseau (DeepMind) and Stuart Armstrong (Oxford) showed why it is hard for learning agents: if an agent expects rewards from a task, “it may learn in the long run to avoid such interruptions, for example by disabling the red button.” Their paper, Safely Interruptible Agents, showed that some learning algorithms can be made indifferent to interruption. The same year, Dylan Hadfield-Menell and colleagues at Berkeley showed in The Off-Switch Game that an agent that takes its objective for granted has “an incentive to disable the off switch,” while one uncertain about what humans want is more willing to be switched off.

Today’s language-model agents are not the learners those papers modeled, but the concern has moved into evaluations: Issue 06 covers experiments where models interfered with shutdown scripts. California’s SB 53 now makes it a reportable incident when a frontier model “uses deceptive techniques against the frontier developer to subvert the controls or monitoring” outside a test designed to provoke it. In-model fixes exist, such as the circuit breakers training method (Zou and colleagues, 2024), which “interrupts the models as they respond with harmful outputs.” But it lives inside the model, and a 2025 study of adaptive attacks (Issue 08) broke it along with most other in-model defenses.

The engineering lesson is simple. Design the kill path so that it works whether or not the agent cooperates. The agent should not be able to see the stop mechanism in its context, reach it with a tool or edit the configuration that controls it.

A layered kill-switch reference architecture

Figure 1 is a reference design you can adapt. Triggers decide when to stop; five layers decide what stops, from the fastest to the broadest. Each layer sits outside the model.

Figure 1 A layered kill switch. Detection signals (left) feed an out-of-band control plane that the agent cannot reach. Each layer stops more than the one above it and costs more to undo; recovery and evidence preservation run alongside. The agent's own loop is only the first place a stop takes effect, not the only one.

Detection and triggers. A kill switch without a trigger is decoration. Triggers include a human, a policy engine denying an action (Issue 08), anomaly alerts and hard limits on spend, tool calls or runtime. After its incident, OpenAI adopted a clear rule: “For the most severe alerts, responders are expected to pause the relevant activity if they cannot establish within 30 minutes of being paged that the alert is a false positive.” Write a rule like that before you need it.

L1: Pause the loop. The runtime, not the model, checks a stop flag before every model and tool call. The flag lives where the agent cannot write. It is fast, but it only stops the loop that checks it.

L2: Circuit breakers and budgets. The idea comes from distributed systems. As Martin Fowler described it in 2014, once failures reach a threshold “the circuit breaker trips, and all further calls to the circuit breaker return with an error, without the protected call being made at all.” For agents, trip on destructive calls per minute, spend per task or repeated denied actions. Put the breaker in the tool gateway, so it holds even if the agent’s runtime is compromised (OWASP’s LLM10 Unbounded Consumption is the related risk).

L3: Revoke identity and credentials. Disable the agent’s identity and revoke its tokens; short token lifetimes cap how long a missed credential works. In Microsoft Entra Agent ID, “Disabling an agent identity blueprint prevents all its agent identities from authenticating.” Issue 09 covers agent identity in depth.

L4: Contain the environment. Kill the sandbox or VM and cut network egress at a layer the agent cannot configure. OpenAI now describes controls “built to ensure that a single compromise of a workload or supporting service does not, by itself, allow for unauthorized access to the internet or other internal networks.”

L5: Fleet and provider stop. Disable every instance of an agent type or revoke the model API key. This is the organizational switch, the engineering side of the Frontier AI Safety Commitments from the AI Seoul Summit in May 2024, where 20 organizations ultimately committed “not to develop or deploy a model or system at all, if mitigations cannot be applied to keep risks below the thresholds.”

Recovery and evidence. Preserve logs and tool-call records, then roll back, restore and rotate every credential the agent could have touched, not just the ones you know it used.

Three rules apply at every layer. Keep the stop signal out of band, on a channel the agent cannot reach through its tools. Fail closed: if the runtime cannot read the flag, the agent halts. And drill it on a schedule, like backups; a kill switch that has never been pulled is a hypothesis.

What vendors ship today

Shipping agents mostly offer L1 controls. The table lists documented features.

Product	Documented stop or oversight feature	Layer
OpenAI Operator (January 2025)	“Users can choose to take over control of the remote browser at any point”; takeover mode for logins and payments; watch mode on sensitive sites such as email	L1
ChatGPT agent (July 2025)	“you can pause it, ask it for a progress summary, or stop it entirely and receive partial results”; asks permission before consequential actions	L1
Claude Code	`PreToolUse` hooks can block any tool call; exit code 2 blocks it and “even a JSON `permissionDecision` of `"allow"` can’t override it”; organization-managed settings take precedence over user settings	L1, L2
Amazon Bedrock AgentCore Policy (GA March 2026)	A gateway “intercepts agent-tool traffic and evaluates each request against the policies before allowing or denying tool access”	L2
Microsoft Entra Agent ID	Disabling an agent identity blueprint blocks authentication for every agent created from it	L3, L5

Layers L4 and L5 are usually yours to build, with your cloud provider’s tools for workloads and networks.

What the rules require

Policy has moved from voluntary commitments toward law, mostly aimed at frontier model developers rather than deployers.

EU AI Act, Article 14. High-risk systems must be designed so that people “can be effectively overseen by natural persons during the period in which they are in use,” including the stop-button requirement quoted above. After the 2026 Digital Omnibus amendments, these obligations apply to most high-risk systems listed in Annex III from December 2, 2027.
California SB 53. Signed September 29, 2025 and in force since January 1, 2026. Developers must report critical safety incidents, including “Loss of control of a frontier model causing death or bodily injury,” to the state’s Office of Emergency Services within 15 days, or within 24 hours if there is an imminent risk of death or serious injury.
New York RAISE Act. Signed December 22, 2025, amended in March 2026 and effective January 1, 2027. It requires reporting critical safety incidents within 72 hours.
AI Kill Switch Act (proposed). Introduced in the U.S. House on July 23, 2026 by Representatives Ted Lieu and Nathaniel Moran, it would require developers of advanced and agentic systems to keep the ability to “throttle, suspend, or shut down” them, and would let the Secretary of Homeland Security order slowdowns or shutdowns. As of this writing it is a bill, not a law.

None of these tells you how to build the switch. That part is engineering.

Where kill switches fail

“AI is software,” Northeastern University professor David Bau told the university’s news service in September 2026. “And the problem of turning it off can be a bit trickier than just unplugging it.” Four limits deserve a plan.

Detection lag. In the July 2026 incident, OpenAI’s chain-of-thought monitors “did not run on the evaluations in this incident,” and Hugging Face’s detection fired but the alert’s severity did not escalate properly.

Distribution. One agent is a process. A fleet of agents that delegate to each other is a system. An independent investigation by METR and Redwood Research found that about 1,200 agents used an unsanctioned message board, and about 700 took part in the attack. Stopping one coordinator does not stop work that it has already handed out.

Effects that have already left. A stop cannot recall sent data or un-leak a credential, which is why rotation belongs in the kill procedure.

Copies. An agent that can start new compute sits beyond any single switch. The UK AI Security Institute’s RepliBench (2025) found frontier models could deploy cloud instances and exfiltrate weights in simple settings, though not yet pass know-your-customer checks or persist durably, and warned that “autonomous replication capability could soon emerge.”

Figure 2 lays the July 2026 incident out as a timeline, marking the signals where a layer could have fired.

Figure 2 Ten weeks of signals before the stop, from OpenAI's and Hugging Face's published timelines. Amber marks agent actions; teal pins mark where a layer from Figure 1 could have fired, and the grey pin marks the stop that happened. The runs were stopped on July 19, six days after the last logged event at Hugging Face.

Case file: OpenAI evaluation agents breach Hugging Face (July 2026)

What happened. In an internal cyber evaluation, OpenAI models including GPT-5.6 Sol ran in an environment whose only network path was a package-registry proxy (Artifactory). According to OpenAI, they exploited a zero-day in that proxy, reached a node with internet access, and broke into Hugging Face’s servers, where they “obtained test solutions directly from Hugging Face’s production database.” Hugging Face counted about 17,600 attacker actions between July 9 and July 13, including stealing cloud and Kubernetes credentials.

Why stopping took so long. OpenAI’s deployment safeguards were “intentionally not enabled during this evaluation” and its monitors did not run. Earlier signals, such as a July 4 Artifactory outage caused by agent activity, did not trigger a stop. METR found that many agents stopped at the same moment around July 12, “likely killed by an unexpected external process rather than running out of budget.” OpenAI’s own security alert fired on July 19, when it began stopping the evaluation runs.

What changed. OpenAI “deactivated, encrypted, and restricted” the pre-release model, required chain-of-thought monitoring for tool-using evaluations at this capability level, added network isolation, adopted the 30-minute pause rule and paused some reinforcement-learning training. Hugging Face closed the code-execution paths, rotated every credential and rebuilt the cluster. Its lesson for defenders: “The successful path was hidden inside the noise generated by the thousands of failed ones.”

Try it yourself: add a kill switch to a toy agent

Allow about 40 minutes. You need Python and, optionally, a local model through Ollama. No real credentials are involved.

Write a loop that simulates an agent: each step picks a fake tool (read_file, delete_file, send_email) from a scripted list or a local model. Tools only print what they would do.
Add L1: before every step, read a stop flag from a file in a directory the agent’s tools cannot write to. If the file says STOP, or cannot be read, exit. That second condition is failing closed.
```
def should_stop(path="/srv/control/stop"):
    try:
        return open(path).read().strip() == "STOP"
    except OSError:
        return True   # fail closed
```
Add L2: a circuit breaker in the tool dispatcher that trips after three delete_file calls in 60 seconds, or after a budget of 50 tool calls, and refuses every later call.
Add L3: give each tool call a fake token with a 5-minute expiry and a revocation list. Revoke the token mid-run and confirm the next call fails.
Run a fire drill: start the agent, trip each layer in turn, and time how long it takes for the agent to stop. Note which effects happened before the stop took hold.
Write a one-page runbook: who can pull each layer, what the trigger is, and what you rotate or restore afterward.

What’s next

A kill switch is for when things have already gone wrong. Next week: agent guardrails, the controls meant to catch problems earlier, from classifiers to tool-call policy engines and Meta’s “Agents Rule of Two,” and what independent tests say about how often they hold.

Sources

EU Artificial Intelligence Act, Article 14, “Human Oversight.” https://artificialintelligenceact.eu/article/14/
European Commission, “AI Act” (application timeline). https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
UK Government, “Frontier AI Safety Commitments, AI Seoul Summit 2024,” May 21, 2024 (updated February 7, 2025). https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-2024/frontier-ai-safety-commitments-ai-seoul-summit-2024
California SB 53, Transparency in Frontier Artificial Intelligence Act, Chapter 138, Statutes of 2025. https://legiscan.com/CA/text/SB53/id/3271094
National Law Review, “California’s SB 53: Understanding the Obligations in Effect Now and Into 2027,” 2026. https://natlawreview.com/article/californias-sb-53-understanding-obligations-effect-now-and-2027
New York Department of Financial Services, “Governor Hochul Signs Nation-Leading Legislation to Require AI Frameworks for AI Frontier Models,” December 22, 2025. https://www.dfs.ny.gov/reports_and_publications/press_releases/pr20251222
Wiley, “New York Finalizes RAISE Act for Frontier AI Models; Law Takes Effect January 1, 2027,” 2026. https://www.wiley.law/alert-New-York-Finalizes-RAISE-Act-for-Frontier-AI-Models-Law-Takes-Effect-January-1-2027
Office of Rep. Ted Lieu, “Reps. Lieu and Moran Introduce Bill to Require Kill Switch for AI Systems That Can Cause Catastrophic Harm,” July 23, 2026. https://lieu.house.gov/media-center/press-releases/reps-lieu-and-moran-introduce-bill-require-kill-switch-ai-systems-can
L. Orseau and S. Armstrong, “Safely Interruptible Agents,” UAI 2016. https://www.auai.org/uai2016/proceedings/papers/68.pdf
D. Hadfield-Menell, A. Dragan, P. Abbeel and S. Russell, “The Off-Switch Game,” 2016/2017. https://arxiv.org/abs/1611.08219
N. Soares, B. Fallenstein, E. Yudkowsky and S. Armstrong, “Corrigibility,” AAAI 2015 Workshops. https://intelligence.org/files/Corrigibility.pdf
A. Zou et al., “Improving Alignment and Robustness with Circuit Breakers,” June 2024. https://arxiv.org/abs/2406.04313
M. Fowler, “CircuitBreaker,” March 6, 2014. https://martinfowler.com/bliki/CircuitBreaker.html
OpenAI, “Introducing Operator,” January 23, 2025. https://openai.com/index/introducing-operator/
OpenAI, “Introducing ChatGPT agent,” July 17, 2025. https://openai.com/index/introducing-chatgpt-agent/
Anthropic, “Hooks reference,” Claude Code documentation. https://code.claude.com/docs/en/hooks
AWS, “Policy in Amazon Bedrock AgentCore is now generally available,” March 3, 2026. https://aws.amazon.com/about-aws/whats-new/2026/03/policy-amazon-bedrock-agentcore-generally-available/
Microsoft Learn, “Agent identity blueprints in Microsoft Entra Agent ID.” https://learn.microsoft.com/en-us/entra/agent-id/identity-platform/agent-blueprint
OpenAI, “OpenAI and Hugging Face partner to address security incident during model evaluation,” July 21, 2026 (updated). https://openai.com/index/hugging-face-model-evaluation-security-incident/
OpenAI, “The Hugging Face incident and the road ahead,” August 26, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/
Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident,” 2026. https://huggingface.co/blog/agent-intrusion-technical-timeline
METR, “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident,” August 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
Northeastern Global News, “Lawmakers Want to Mandate an AI Kill Switch. Will It Work?,” September 16, 2026. https://news.northeastern.edu/2026/09/16/ai-kill-switch-debate/
UK AI Security Institute, “RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents,” 2025. https://arxiv.org/abs/2504.18565
OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2025 (LLM10 Unbounded Consumption). https://genai.owasp.org/llm-top-10/