Forecasting Swarm Size
Michaelmas 2026 project
Research question
Paolo Bova
Efforts to quantify loss-of-control risks depend on the “size” of AI deployments that can act outside of developer control. Recent incidents might be a good reference class for thinking about the size such swarms could reach in the future. To make use of these incidents as a reference class, we need to propose a measure for what we mean by size. I propose we work with the following definition: the volume of covert project-work a rogue AI deployment can complete before developers detect it and mount resistance, measured in human-expert-days.
I currently fit this quantity to exactly two data points: roughly one covert project-day in early 2026 (models covertly cheating their graders during internal testing) and eight to nine covert project-days by summer 2026 (a multi-agent breach spanning several days of coordinated work), giving a cubic growth curve against a capability index. What data, beyond these two points, would let us forecast how swarm size grows through 2027 with real statistical grounding?
Project outline
The project has three parts. First, we will survey existing benchmarks and evaluations that could proxy covert task-completion time. We will design an estimator for swarm size based on this proxy. Second, we will estimate the swarm size of all summer incidents. We will also break down what the key factors were contributing to these scores and test for a trend over time. Third, we will establish the relevance of this measure of swarm size to the likelihood of rogue deployments succeeding at acquiring resources and inflicting mass harm. For ‘acquiring resources’ this will involve estimates of the affordances a rogue deployment may find relatively undefended under scenarios with or without widespread AI deployment. For ‘inflicting mass harm’ this will involve estimating the difficulty of different steps required to launch cyber or kinetic attacks on critical national infrastructure, or in the production and deployment of novel weapons such as pathogens.
Impact case
Swarm size feeds directly into my model's headline catastrophe-probability estimates, which aim to inform risk thresholds for loss-of-control risk. Swarm size is currently one of the least confident inputs in the model. Tightening it, in either direction, would materially change estimates of loss of control risk, and speak directly to how AI labs should price loss-of-control risk. This work also contributes to answering how much runway current monitoring gives a rogue deployment before detection, and whether that runway is closing or widening.
Ideal candidate
I am looking for participant(s) who would enjoy thinking about how to make this definition of swarm size more concrete. Comfort with statistical modelling is handy because we are fitting data to a statistical model. I think candidates will likely fit this criteria if they are studying a degree with a quantitative component such as but not limited to economics, maths, physics, or computer science. Prior exposure to AI safety or governance is preferred. Coding experience is not required.