How Should Labs Train AIs That Might Build Smarter AIs?

Ajeya Cotra says labs should stop training in environments that reward hacking, and she warns that this is only a floor, not a solution.

AI development is already so fast that companies cut corners. They are not carefully checking their training environments, monitoring everything, or stress-testing those monitors, and she expects the pace to get faster and more chaotic. The fragile pattern is to reward agents for finding cheats in some settings and then try to penalize the same behavior, or score a caught cheat as zero, in others. She wants the environments that push agents toward cheating removed, so training does not aim them that way in the first place. She calls the alternative whack-a-mole.

She also wants the tools that watch a run kept separate from the reward. A monitor can read a chain of thought and alert people. Training on that monitor would push agents to hide what they are thinking. She offers both ideas as a minimum the industry could adopt quickly, with the caveat that they will not solve the whole problem.