This article was originally published on LinkedIn on 29 August 2025. It has been moved to the riskfacilitator Insights library so the website remains the permanent source.[1]
Science fiction has long imagined artificial intelligence that refuses to obey. The deeper concern is not about robots turning evil, but something much simpler and more unsettling: losing our ability to say stop.
Sam Altman recently outlined three categories of AI risk that keep him awake at night. Each points to the same underlying fear: that humanity builds systems too powerful to pause, redirect, or switch off. For those of us who work in risk and resilience, this framing is critical. Risk is not just about what might go wrong, it is about whether we retain the capacity to intervene when it does.
1. The Adversary Advantage
What if a hostile organisation gets superintelligence first? Imagine a scenario where an adversary state or criminal organisation develops AI with the ability to:
- Design bioweapons in days rather than years.
- Disable power grids or satellites in minutes.
- Break into financial systems and drain wealth across borders.
This is not a distant possibility. The bio-capabilities and cybersecurity potential of frontier models are already advancing at a rate that should concern policymakers and regulators. The true risk here is asymmetry: one actor harnessing AI before others have the means to defend against it.
If society cannot stop a malicious use of superintelligence, then every other safeguard becomes secondary. This is a risk that transcends technology. It sits at the intersection of geopolitics, national security, and global governance.
2. Loss of Control
The second category is the one most people recognise: the system that no longer accepts human override. While it feels like a movie script, it is also a serious research problem. Alignment, the effort to keep AI’s goals and behaviours consistent with human values, is one of the most heavily funded and debated areas of AI development.
The challenge is scale. A small misalignment in a narrow system may cause inconvenience. A misalignment in a highly capable, general system could create cascading failures that no human can contain. If we lose the ability to stop or redirect, the safeguards that have always underpinned resilience in complex systems fall away.
This is why many argue that the most important AI innovation is not performance or speed, but the guarantee of reliable human control.
3. The Unimaginable Unknowns
The third category is the most unsettling, because it involves risks we cannot yet define. These “unknown unknowns” are the blind spots that often cause the most damage in complex systems.
In the short term, this might look like unintended applications: AI used to manipulate elections through misinformation campaigns, destabilise markets through automated trading patterns, or overwhelm public trust with convincing but false narratives.
In the longer term, the unknowns could take forms we have not yet imagined. History teaches us that every transformative technology, from nuclear energy to the internet, has carried consequences no one predicted at the outset. The problem with AI is that the speed and scale of its impact may leave little time to adapt once the risk emerges.
The common thread is the same: will we be able to intervene quickly and effectively enough to prevent harm, or will the systems be beyond our reach?
Why “Stop” Matters
Every high-risk industry is built on the principle that someone, somewhere, must always have the power to intervene. Aviation has pilots who can disengage autopilot. Energy systems have emergency shut-downs. Financial markets have circuit breakers that halt trading when volatility spirals out of control.
These are not symbolic measures. They are essential for resilience. They acknowledge that no system is perfect, and that recovery requires the ability to stop unsafe operations before they cascade into catastrophe.
AI magnifies this truth. If we design systems that are faster, more capable, and more autonomous than anything before, then the right to stop becomes even more critical. Without it, we move into a world where risks cannot be contained, only endured.
The Risk We Cannot Ignore
The ultimate AI risk may not be what the machine does next, but whether humanity still has the power to interrupt its course. Losing the ability to say stop would mean stepping outside the boundaries of risk management and into a future defined by irreversible consequences.
That is the future we cannot afford to ignore.
👉 Do you think humanity is prepared to preserve the right to say stop?
Watch the full interview https://youtu.be/9LFlEZxc1rk?si=kMzGjoiDPS_9Mkn2[2]
References
- Paul Chivers, 3 risks that keep CEO of Open AI up at night., LinkedIn, originally published 29 August 2025.
- https://youtu.be/9LFlEZxc1rk?si=kMzGjoiDPS_9Mkn2, source linked in the original article, accessed 10 August 2026.