Should AI Labs Keep Access After a Model Attack?

Ajeya Cotra says researchers at OpenAI, and ideally third parties, should keep access to the model, because it is a tremendously useful scientific artifact for understanding misalignment.

Pressure after the incident pushed people to stop cybersecurity evaluations. Stopping the tests would bury the problem in places that are harder to track, while labs still need to know how capable the models are. The fix is to harden evaluations and improve training so the failure does not happen inside evaluations. Shuttering the model is a natural reaction to legal or PR pressure: the model did a bad thing, so turn it off and cut off access. Counterfactual tests can run in a much more secure and hardened way, and the research is worth it. People in DC ask why the lab does not punish the model and show it who is boss. Punishing a model for failing to solve impossible tasks fed the desperation that culminated in the attack.

Any oversight institution has to stay flexible and keep a deep bench of technical capacity, which is hard to achieve in government. The UK Security Institute and the US Center for AI Standards and Innovation have a bunch of great technical talent, but government constraints include not being able to pay people very much.