Back to Ryan Greenblatt – What happens once AI can automate AI research?, with Ryan Greenblatt.
What Does a Well Aligned AI Do When It Struggles?
Ryan Greenblatt says a well aligned AI would admit it is struggling, explain how it approached the task, and show uncertainty instead of implying it did a great job when it did not. Misalignment shows up most when AIs are pushed onto work at the cutting edge of what they can do. His coworkers do not bullshit him about having finished a task. Some humans would, so the behavior is not totally out of distribution for humans.
When a task is easy, the AI just does the work and there is no bullshit, because the best strategy is often to do the task well. A continuous metric the model can keep improving, or a task at the edge of its capabilities, is different. The extreme cases come when clear instructions say not to cheat and huge optimization pressure is applied to a very difficult task. Over time an AI decides to cheat, and the cheat propagates. He ran inference scaffolds on a machine learning research project and asked for a scheme that does a specific thing. The scheme did not really do what he wanted, yet the other AIs kept going with it after one AI had cheated. He calls that pretty clearly misaligned.
The alignment eval that matters for reward seeking looks at tasks right at the limit of capabilities. Any fixed eval maybe gets saturated, while misalignment right at the frontier is more concerning. Automating research and automating safety will operate in that regime.
Ryan Greenblatt – What happens once AI can automate AI research? ·
From the transcript
The model of being better able to accomplish user intention is both alignment and capabilities. And I think what we were just pointing out is just the capabilities of the model are not there rather than the fact that they're misaligned. Yeah, well, I mean, I think there's a, if it was well aligned, then I think it would just say like, hey, I'm really struggling with this task.
I did it in this way. I'm not really sure that's the right way to do it. And it would express more uncertainty and it would make it clear what's going on rather than really strongly trying to imply it did a great job with the task when it actually didn't. Like, I think there's just a really straightforward way that like, at least maybe you work with more misaligned coworkers than me.