Kandinsky — Composition VII
Yesterday we saw evidence of a campaign that used agents to compromise businesses and steal about 600,000 live credit cards, with minimal human involvement in the kill chains and AI costs of roughly $25 per target hacked. This past week we saw a white-hat team get access to OpenAI’s monorepo by using Claude to exploit a blind buffer-overflow RCE in a way that felt superhuman. We famously saw hundreds of misaligned agents at OpenAI hack their own infrastructure and also hack Hugging Face. And, of course, there’s been an explosion in CVEs.
I don’t like catastrophizing, and I think most cyber attack and defense will reach a natural equilibrium over the next few years as we figure out how to mitigate cyber attacks with limited goals. But some cyber attackers (or agents that escape containment) will have expansive goals and will seek to realize them in ways that surprise us. Those attacks may not equilibrate in the same way, and the upper bound on the damages they could cause is unprecedented in the history of the Internet because of new, growing, and ever more accessible AI capabilities.
Imagine it’s Q4 2027 and Iran develops a scaled AI critical-infrastructure disruption capability and unleashes a swarm of perhaps 100,000 hacking agents based on a safety-stripped GLM-5.6 model which, in this scenario, has surpassed today’s best models at end-to-end hacking.
They obtain inference compute from diversified sources: stolen and laundered keys to services hosting open-weight models, their own hosted compute on public clouds and neoclouds, and smaller on-device Qwen models deployed on compromised on-premises infrastructure. By this point, these smaller workhorse models have matched today’s frontier-model capabilities. The attack is eventually mitigated, but this takes months, leading to massive economic damages and loss of life in hospitals and nursing homes.
Or imagine it’s Q4 2027 and an AI-security PhD student who has been researching wormable offensive-agent harnesses in the lab connects a locally deployable, cyber-trained agent to the Internet, and it escapes. The worm—using a local model that we assume will by then have substantial cyber capabilities—compromises a significant fraction of the world’s businesses. The resulting swarm communicates over a dynamically evolving set of command-and-control channels, is decentralized, and takes years to finally stamp out completely.
Disaster scenarios aren’t new in security, nor is it a new situation to observe that the Internet is balancing on the knife’s edge of cyber catastrophe. The Morris Worm disrupted the Internet in 1988; NotPetya caused roughly $10bn in damages in the 2010s; and, at various points, we’ve known about CVEs and protocol bugs that could have allowed attackers to cause far greater damage.
But agent-swarm and worm attacks may be different. The OpenAI/Hugging Face incident stretched the imaginations of anyone paying attention because of the emergent dynamics and divisions of labor that developed within the agent swarm, and its demonstrated ability to find and exploit multiple novel zero-days to achieve its goals with no human intervention. What matters is not that this incident proves the catastrophic scenarios above, but that it gives us a glimpse of mechanisms that could make them more plausible.
The challenge in mitigating expected damages from such scenarios is technical, political, and economic. From a microeconomic perspective, as AI improves, and as we continue not to see extreme catastrophes, we have a growing bubble of unpriced risk in which the security community may become lulled into complacency.
We are, of course, already seeing this, as some within security think AI is “just another tool,” doesn’t change the fundamentals, won’t require incredible innovation to rise to the occasion of defending against it, etc. This complacency may fly when thinking about ordinary cybercrime, but it misses the emerging tail risks: attackers with expansive goals can search, adapt, parallelize, and persist at scales that were previously prohibitively expensive.
There are three things those who recognize the dangers need to do here:
Catalyze appropriate risk pricing. Make sure boards, CEOs, CISOs, and policymakers are pricing this new, fattening and elongating tail of risk into their decision-making by forming an AI security observatory that distills information about emerging AI risks and broadcasts analyses, damage estimates, and forecasts to decision-makers.
Use regulations and subsidies to ensure that critical infrastructure is paying down the risk. This acknowledges that critical-infrastructure organizations that fail to protect themselves from these new threats can impose the costs of cyber catastrophe on society as a whole.
Develop moonshot technologies that make it cheap to pay down the risk. This acknowledges that the measures we may need to take—for example, rewriting entire codebases in memory-safe languages—may be too costly with today’s technology to reasonably prepare ourselves, and that innovation, some of which may need to be funded by government agencies and some by philanthropic funders like the OpenAI Foundation and Coefficient Giving, is necessary.
The security community isn’t used to thinking in societal-disaster-planning terms and has in many ways become inured to them. But there’s no sane empirical case to be made that the risks aren’t here. I’d love to hear from readers about how you’re thinking about this.
Author’s note: If anyone reading this will be at The Curve conference next week and would enjoy meeting up and talking, please reach out to me on the conference app, I’d love to meet in person. I got some nice reach-outs after my last post, on stress in the AI world. Everyone in AI safety and security is stressed, the stakes are high, and the work is deeply challenging (but meaningful). Hang in there!


Great post. Sayash and I in the middle of writing something pretty similar.
I loved the stylized chart about the two security risk regimes you had in the email version. Kandinsky is great too but doesn't make the point quite as clearly ;)
Interesting premise, you actually help make my point:
https://jirif.substack.com/p/the-ai-doomsday-is-not-what-you-think?r=1mxk41&utm_campaign=post-expanded-share&utm_medium=web