Sneak Peak of RIO’s first projects

Mentor: Aaron Maiwald, Dphil, University of Oxford supported by the Berkeley Existential Risk Imitative

Biological language models promise to transform biological design. Trained to predict the next nucleotide or amino acid, these systems have been used to generate new regulatory elements or antibodies. Like any tool for biological design, they are dual-use. To reduce the risk of misuse, various strategies aim to selectively reduce performance on dangerous viral sequences. One strategy is to simply exclude some viral sequences from the model's training data. The recently developed genomic language model, EVO 2, excluded all eukaryote-infecting viruses, a superset of those dangerous to humans. While this appears to effectively destroy model performance on human-infecting viral genomes, it also affects performance on totally benign viral sequences. This makes it less attractive as a strategy to future model developers and may also reduce beneficial model capabilities. 

Can we do better? How selectively can we deteriorate model performance on human-infecting viruses? Would it be possible to exclude a much narrower set of viruses and thereby leave model capabilities on benign viruses intact? This project aims to train a set of genomic and protein language models with different levels of data exclusion to assess whether this is possible. 

Skills you could learn: how to pretrain a language model; working with large biological datasets; machine learning/coding skills.

Ideal candidate: at least moderate coding abilities; experience with deep learning libraries like PyTorch, or a great interest in learning that; sceptical mindset; excited and driven to get projects over the finish line. 

Previous Projects

Impact Projects Oxford is the Oxford based version of Impact Research Groups (IRG), lead by it’s founder Ayushmaan Sharma. The following is a list of winning projects from IRG.

Previous Winners

Spring 2025

Previous Projects

Technical AI Safety

• Identifying AI Agents’ Bottlenecks on Evaluations at Scale via Automated Analysis.

• Investigating the Robustness of Language Models Obtained by ‘Pre-Training with Human Preferences’.

• How Do LLM Features Develop Through Pre-Training and Fine-Tuning?

• How Robust Are White-Box Detectors?

• Can We Understand the Differences Between Different Machine Learning Models Mechanistically?

• Mapping Jailbreaks in LLMs

• Investigating the Psychology of LLMs

• Paraphrasing as a Tool for Preventing Self-Recognition in LLMs

• Does Goal Valence Affect Alignment Faking in LLMs?

AI Governance

• What Is the Most Realistic Mode of Governance for Autonomous Weapons Systems?

• A Landscape Analysis of the AI Policy and Governance Working Group (AIPGWG)

• How Could an Ombudsman Contribute to Effective Post-Market Monitoring of AI?

• What Is an Appropriate Level of Access to AI User Interaction Data?

• AGI Lab Boards: What Should the Makeup and Functions of the Boards That Govern AGI Labs Be?

• A Landscape Analysis of the Collective Intelligence Project (CIP)

• Regulators, Evaluators, or Watchdog: What Should the UK Government’s AI Safety Institute Look Like?

  —

• Bootstrapping Probabilistic Confidence in a Safety Claim

• Strengthening Safety Cases Through Use of Multiple Independent Parallel Arguments

• Bridging AI Governance Across the Atlantic: Strategies for UK–US Collaboration in a Shifting Global Landscape

• How Can We Prevent Gradual Human Disempowerment from AI?

• Shadow AI Governance: Mapping the Power of Non-Governmental AI Rulemakers

Biosecurity

• DNA Synthesis Screening: How Do We Address the Current Uneven Geographic Distributions?

• The Role of Genomic Surveillance in Scaling Up Pandemic Preparedness in LMICs

  —

• What Are the Primary Bottlenecks in the Implementation of Far-UVC Technology in Public Health Settings, and How Can We Effectively Address Them?

• Overcoming Bottlenecks in the Development and Deployment of Next-Generation PPE

Global Health

• What Are the Projected Levels of Air Pollution in LMICs, and How Do Different Sectors Contribute to This Pollution, Allowing for the Identification of High-Impact Regions and Effective Intervention Strategies?

• Are There Interventions That Would Cost-Effectively Prevent Childhood Pneumonia in LMICs?

• What Is the Effect of Global Health Interventions on Equilibrium Population Levels?

• Barriers to Scaling Up TB Diagnostics and Treatments in LMICs

• Evaluating the Effectiveness of Heat Adaptation Strategies in Reducing Mortality in LMICs

Animal Welfare

• Which Food Corporations Should Good Growth Target to Maximise Their Impact on Alternative Protein Adoption and Animal Welfare in Southeast Asia?

• Identifying Case Studies of Successful Alt Protein Products in China and Southeast Asia

• Animal Byproducts in Pet Food: How Does the Livestock Sector Profit from Byproducts?

• What Drives Social Movements to a Tipping Point? Lessons for the Animal Advocacy Movement

• How Cost-Effective Is Corporate Litigation and Other Legal Advocacy Interventions?

• How Generalisable Is Evidence on the Effectiveness of Corporate Outreach to the Global South?