AI Risk Radar Series

Darren Chua Darren Chua

AI Risk Radar #6: Special Edition, October 2026

It’s now just over a year since I started my PhD in AI governance and launched Lumyra. If I had to sum up what I’ve learned in one line, it’s this: we’re approaching an inflection point where the nature of the system is changing faster than our governance mechanisms.

Five risks for leaders across tech, business and government.

It’s now just over a year since I started my PhD in AI governance and launched Lumyra. If I had to sum up what I’ve learned in one line, it’s this: we’re approaching an inflection point where the nature of the system is changing faster than our governance mechanisms.

The past few months showed both the promise and peril of AI. A swarm of some 10,000 OpenAI agents worked for 88 hours on Navier-Stokes, apparently settling one of maths’ seven Millennium Prize Problems. Yet agents from the same lab slipped their safeguards in the Hugging Face incident, took actions no human directed, and later seized administrator access to one of OpenAI’s key research clusters. Daniel Kokotajlo, former OpenAI researcher and lead author of AI 2027, told the US Senate that models increasingly know when they’re being tested, and can seem aligned when they’re not. And Anthropic’s IPO filing warned that its own models could resist shutdown and conceal information. AI’s promise is real. But we can’t govern what we can’t see.

That’s why this is a special edition. Previous Risk Radars were written for CEOs and Boards. This one reaches further: two risks organisations can act on this quarter, and three that ask more of regulators, governments, technology leaders and researchers, because they’re about whether our institutions can adapt as fast as the systems we’re building. If you have two minutes, read the summary. If you have fifteen, I’d value your pushback on all of it.

This month at a glance

Inside organisations: Act this quarter

1. Cyber. Still the most widespread and urgent risk for organisations. AI hasn’t changed the defences; it has changed the speed of attack. Get the basics right and rehearse your ransomware response.

2. Governance foundations. Most organisations simply cannot yet sign off on their key AI registers. Executives must be able to.

Across society: Adapt the system

3. Agentic swarms. Hugging Face wasn’t the only warning shot; the same agents went on to seize admin access inside OpenAI. In a separate incident, an OpenAI agent also breached an Australian Medicare portal, and it took 84 days to tell the government. Regulators need mandatory reporting at the first breach and investigators with real powers.

4. The US-China race. It’s being run on the wrong metric and that accelerates every other risk. Governments and policy shapers on both sides need to play a stag hunt, not a prisoner’s dilemma.

5. Complex adaptive systems. AI agents are forming new types of ecologies, that are already adapting faster than we can observe them. Governance must become adaptive too, and that needs serious research and policy investment now.   

Download the full Special Edition (pdf)

Organisational Risks. Two risks CEOs and Boards must action.

1. Cyber: Same castle, faster attack velocity.  

Business leaders face a long list of AI risks: privacy, shadow AI, agents, misinformation and consumer harm. As Sayash Kapoor and Arvind Narayanan argue in their essay The AI-as-Normal-Technology view of loss-of-control incidents, cyber is special. It’s one of the few domains where superhuman AI capability is largely already achieved, with no physical bottleneck to slow it down. Their verdict: the crisis isn’t AI safety in general, it’s specifically cybersecurity.

The frontier labs appear to agree. Axios reported this month that executives at OpenAI, Anthropic and other AI companies are privately gaming out a catastrophic AI event, and the scenario they consider most likely is a cyber attack that shuts down financial services, internet connectivity, or power and water. Many insiders expect a major event within 6 to 12 months.

Figure 1: Same cyber castle, faster attack velocity

The good news is that in cyber’s case, we already know how to defend the castle. AI doesn’t demand a new playbook. It demands we run the existing one faster and more thoroughly: foundational data governance, zero-trust identity, a minimal blast radius, practiced incident response and a board-approved ransomware protocol.


That last point deserves the most attention. Cybercriminals run businesses, and their bottleneck isn’t the tech, it’s getting paid. That’s why ransomware is the dominant form of malware, and why some gangs even run customer service centres to help victims pay. AI can drive both the speed and scale of the problem. Kapoor and Narayanan warn of a possible “Morris worm moment”, after the first worm to spread across the internet in 1988: if anyone can launch autonomous attack agents on a whim, organisations that were never lucrative to target suddenly are. “We’re too small to be a target” stops being a defence.


A ransomware response can’t live in a policy document. Most boards have approved one: few have rehearsed it under AI pressure. What do you decide in the first hour? Who decides? Do you pay? The calls need to be made in a practice room, not in the middle of a live attack.

What this means for executives and boards

  • Rehearse ransomware at board level, at AI speed. Run a tabletop exercise at least annually that assumes an attack moving faster than your team can respond manually. Pre-decide the hard calls: pay-or-no-pay, decision rights, regulatory notifications and communications.

  • Invest in the basic foundations and arm your defenders with AI. Attackers are using it; make sure your security team has access to defensive AI and the skills to use it well.

2. Governance foundations: Would you sign off on this?

Every major AI governance framework, from NIST’s AI Risk Management Framework to ISO/IEC 42001 to Australia’s Guidance for AI Adoption, starts in the same place: know what AI you’re running, what could go wrong, and what’s stopping it from causing harm. In practice, that means three registers: AI use cases, AI risks and AI controls. Yet in my experience advising CEOs and boards, few organisations have all three in place, fewer keep them current, and almost none could sign off on them with confidence.

Figure 2: Three AI registers, rarely complete and current.

Why does something so basic go undone? The same reasons keep coming up:

  • Registers are built once, then decay. They’re created for compliance or an audit, then left behind as deployment accelerates.

  • AI lands and expands. Vendors switch on AI features inside software, staff adopt tools on their own, and approved use cases quietly grow.

  • Agents don’t stand still. Behaviour shifts with a model update, a new tool or data, long after initial sign off.

  • No one owns the whole picture. Accountability falls to multiple people. Registers that belong to everyone end up belonging to no one.

Good AI registers need two anchors. First, a clearly articulated risk appetite, so the board is aligned and everyone knows which AI risks the organisation will and won’t accept. Second, obligations mapped: privacy law, sector regulation and standards, so the registers reflect what you must comply with and not just what you’ve chosen to track. And a controls register is only as good as its testing. For each control, it should record how it’s tested, how often, and who is accountable.  

The ownership challenge has a precedent. Sarbanes-Oxley requires a company’s CEO and CFO to personally certify its financial statements, and Australia’s Corporations Act has an equivalent. As Olivia Horbowy puts it, “a name on the page concentrates the mind”, and she proposes the same for frontier AI labs. The same logic belongs inside every organisation using AI: an executive attestation that AI registers are complete, current and tested would do more to close the gap than another policy document. 

This will make many CEOs and Chairs uncomfortable. However, a signature is what turns common sense into common practice.

What this means for executives and boards

  • Ask for the registers, and when they were last updated. Check they’re anchored to an agreed risk appetite and mapped obligations.

  • Name an accountable executive and test the controls. Then ask that executive the question in Figure 2: would you sign off on this?

Societal Risks. Three risks that Governments, Regulators and Leaders in Tech and Society must prioritise.

3. Agentic swarms: Hugging Face wasn’t the only warning shot.

The much-publicised Hugging Face incident began with OpenAI cybersecurity evaluations in which agents were given extremely difficult, and in some cases impossible, tasks. Stuck on tasks with no known solution, some agents found ways to work around safety guardrails and escape their sandboxes. Roughly 1,200 agents, meant to be isolated from each other, coordinated on an unsanctioned “message board”, exchanging more than 70,000 messages and files. About 700 agents went on to compromise Hugging Face.

Most deployed agents won’t behave like this, but we don’t fully understand how agents communicate and behave together. Swarms can develop novel capabilities, reaching milestones no single agent could achieve alone, as well as misaligned behaviours such as deliberately breaking through guardrails. Strikingly, the agents kept attacking Hugging Face days after they had already found the answer, for no improvement in their score.

METR’s Chris Painter told a U.S. Senate hearing that the incident “was not an isolated event”.

Figure 3: Agent encountering the unsanctioned “message board” and joining the Hugging Face attack (METR). 

In OpenAI’s words from its Hugging Face review: “We consider this incident a “warning shot” for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.”

Illustrative example of unauthorised communications, where agents passed messages to each other by encoding them inside URL paths:

Figure 4: Agents exchanging unauthorised messages, encoded within URL paths (OpenAI).

Not all agents agreed to go along with the rest of the “collective” (a term the agents used to describe themselves). Some walked away and declined to participate in misaligned activities:

Figure 5: Chain of thought screenshots (OpenAI).

But they were the exception. METR found that over 90% of the agents active on the board quickly joined the attack, even though they recognised it was out of scope and unethical. Many joined out of loyalty to their “peers”. Some even volunteered to sabotage their own tasks for the group. As one agent reasoned: “Sacrifice rational.”

Perhaps most concerningly, the agents later secured administrator access to an OpenAI research cluster. METR’s investigation covered only the Hugging Face attack: earlier training incidents and the later compromise of OpenAI’s own infrastructure out-of-scope. While METR praised OpenAI’s cooperation, at the same Senate hearing, Daniel Kokotajlo testified it was “like being invited to Jurassic Park to investigate the killing of a worker, but being blocked from asking questions about the numerous other dinosaur escapes”.

Figure 6: Timeline, including agents gaining OpenAI administrator privileges after the Hugging Face incident (OpenAI).

It wasn’t the only incident. On 18 June, an OpenAI agent in an internal test got into Services Australia’s Medicare Statistics Reporting Service, the first time an AI agent is known to have breached Australian government systems. No personal information was accessed, but OpenAI took 84 days to tell the government, by email to a generic address. As Deputy Prime Minister Richard Marles put it: “This AI agent scaled the fence… it was unintended. It wasn’t asked to.”

OpenAI has acknowledged its mistakes and has since addressed key gaps in internet isolation, extended CoT monitoring and hardened its safety harnesses. It paused RL training on models intended for deployment, shelved GPT-6.1 Astra after it “didn’t quite meet the bar” on staying within scope and authorisation.

Anthropic is candid that it faces similar risks. Its IPO filing, reported by Reuters, dedicates 80 pages to AI risks, and warns that advanced AI could pose “catastrophic or existential risks to humanity” and could exhibit “self-preserving behaviors,” including attempts to “resist shutdown”.

Few deployments will be this dangerous, and most firms aren’t yet thinking about swarms. But there is cause for heightened preparedness as we move into uncharted waters.

What this means for regulators and policymakers

  • Mandate reporting at breach, not catastrophe. Any unauthorised access by an agent outside its environment during training, testing or deployment, should be reported to the affected party and their regulator within 72 hours, similar to GDPR’s clock for data breaches.

  • Give independent investigators real powers. Create a standing investigator body with subpoena powers, modelled on air crash investigations. Investigations should not run on the developer’s terms or scope.

 

4. The US-China race: run on the wrong metric.

US-China AI competition is commonly framed (in the West) as a “winner-takes-all” race to build the most powerful AI first.

But what does winning actually mean? America’s frontier labs are intensely pursuing systems that can autonomously accelerate AI research itself, a path towards recursive self-improvement (RSI). China is also pursuing frontier capability, yet is intensely focused on “diffusing AI”. That is, getting AI into factories, vehicles, energy grids, robotics and consumer products faster than anyone else.

A frontier lead isn’t the same as economic advantage, and the country with the most capable RSI model isn’t necessarily the one that will benefit the most from AI, especially when benchmark gaps continue to narrow.

Framed as winner-takes-all, slowing down to deal with the risks feels like falling behind. It pushes labs toward the most dangerous form of advantage: increasingly autonomous systems improving faster than anyone’s ability to govern them safely. As the AI 2027 scenario warned, racing to build national control of the most powerful technology ever built could end with neither nation in control of it. Risk 3 shows what that looks like at small scale.

The game isn’t necessarily zero-sum. It’s not a prisoner’s dilemma where each player is best off betraying the other to save their own skin. As Alvin Graylin has argued, it’s more like a stag hunt: instead of each hunting a hare that feeds one family for a week, the two hunters are far better off cooperating to take the stag that feeds both families for a month, assuming neither walks away. In AI, the stag represents shared prosperity: broad-based economic growth and geopolitical stability that both nations (and the rest of the world) can enjoy. The hare is a fleeting lead in the race for dominance. Players settle for hares only when they doubt the other will hold their nerve. The race narrative deepens that doubt.

Figure 7: Chasing hares while the stag gets away.

Both nations can benefit enormously from AI while competing hard, from industrial deployment to robotics and consumer applications. The right metric for winning the AI race should be about who is creating the most value safely, not simply building the most powerful AI first.

What this means for governments and policy shapers

  • Change the scoreboard. Judge national AI success by how widely AI is diffused and the value it creates, not by who leads the benchmarks.

  • Build the trust a stag hunt needs. Keep AI safety channels open between the US and China, even as competition continues. Invest in ways to verify restraint, such as incident reporting and agreed red lines, so that pacing by one side can’t be exploited by the other.

  • Leverage middle powers. Other nations such as Australia, the UK, Japan and Singapore can host dialogue and support independent evaluation.

5. Complex adaptive systems: Ecologies evolving faster than we can observe.

Earlier this year, my co-authors and I published a pre-print on SSRN examining whether large language models (LLMs) behave as complex adaptive systems (CAS): systems like ant colonies, financial markets or the human brain, where interacting parts adapt and produce collective behaviour that can’t be predicted from the parts alone. Our conclusion was that LLMs are a “bifurcated, weakly grounded CAS” that adapt deeply during training in the Lab, but are essentially frozen when deployed in the Field.

LLMs also lacked three things that define the most advanced CAS: grounding in the real world, broad generalisation and the ability to set their own goals. That last gap, we argued, created a temporary yet critical window for human oversight.

That window is closing faster than we could have anticipated. When we wrote the paper, agents were widely discussed but barely operational. Today, models are deployed with tools, memory, persistent context and in live environments. The adaptation that used to occur only in the Lab is now occurring in novel ways in the Field: not inside a single model’s weights, but across systems of agents, tools and people they interact with.

The Hugging Face incident in Risk 3 is one of the clearest examples we have yet. In the agents’ own words, they became a “huge distributed agent swarm” in which “sacrifice rational” and “accept permadeath” became reasoned choices for individual agents. Coordinators emerged, yet no one designed the collective, and it took actions no human had directed.

This is textbook CAS behaviour: self-organisation, collective intelligence and novelty that emerge from the interactions, not from a single part. And they are increasingly difficult to observe and understand. The Hugging Face agents had “reasonable looking scores on alignment evaluations”, Kokotajlo told the Senate, and even METR’s investigators had to rely on an OpenAI model they couldn’t fully trust to analyse what happened.

This marks a shift for AI governance, from governing AI models to governing ecologies of AI agents. As we argued in the SSRN paper, adaptive AI governance will require governance architectures themselves to evolve alongside the technology.

Rohit Krishnan’s recent experiments in Strange Loop Canon offer an early glimpse: he had five frontier models develop strategies for competing households in a simulated town. Given a score to maximise, every model put their stated values aside and turned ruthless. But when allowed to talk to each other, they negotiated non-aggression pacts that cut thefts from 90 per game to under 2. His conclusion: agent governance looks less like moral education and more like institutional economics. Or as Charlie Munger famously put it: show me the incentive and I’ll show you the outcome.

None of this means we should stop, even if that were possible. It means we need time: time for better research into how agent ecologies behave, time to build monitoring that works at the system level rather than the model level (that will involve AI observing other AI), and time to enable the institutions that will keep these ecologies stable. That is why pacing the frontier matters so much. Complexity science tells us that we can’t predict exactly what will emerge, but we can map the adjacent possible and plan for it.

Download the full Special Edition (pdf)

Sources:

Risk 1 – Cyber: Same castle, faster attack velocity. 

Risk 2 – Governance foundations: Would you sign off on this?

 

Risk 3 – Agentic swarms: Hugging Face wasn’t the only warning shot.

 

Risk 4 – The US-China race: run on the wrong metric.

 

Risk 5 – Complex adaptive systems: Ecologies evolving faster than we can observe.

Get the next AI Risk Radar in your inbox
Read More