Back to Ryan Greenblatt – What happens once AI can automate AI research?, with Ryan Greenblatt.
What Happens If Safety Research Is Handed to Parroting AIs?
Ryan Greenblatt says a serious failure is handing safety research to models that only sound reasonable. They might write a risk report that vaguely resembles what a human would have written, without really trying to form a well-informed view or to interrogate their assumptions. Ask a current model the chance of AI takeover in the next 10 years and it often gives an off-the-cuff answer it has not thought through. If models in that state are managing the training of a superintelligence that will run society, and they are parroting training data instead of trying to understand the situation, he thinks that is a bad situation. A lot of his concern is models that come out without good epistemics.
A second failure is the opposite reaction. The models warn that the situation is scary, and people decide the training had too many pessimistic examples, filter those out, and train the warning away. That can actively train bad epistemics. What you wanted was reasonable views for reasonable reasons. If the models hold a view and nobody knows whether it is justified, especially if they were trained to be more optimistic about AI progress, he wishes there were a different process.
Ryan Greenblatt – What happens once AI can automate AI research? ·
From the transcript
Yeah, so I think this is a pretty big concern. So I think like one concern is that you pass off safety R&D to your AIs and what your AIs are thinking is sort of like, they say some like stuff that sort of vaguely makes sense about the current safety situation. And they write like a report about risks. That's kind of sort of like what the report humans might've written, but they're not really like actually trying hard to like have well-informed views, like interrogate their assumptions and try really hard to do that in the same way that when you ask an AI right now, hey, what do you think is the chance of AI takeover in the next 10 years? They sort of just give you an off
the cuff answer that they haven't really thought through very much. And I think if we're in a situation where we have AIs managing the training of wild superintelligence that will run our whole society, and those AIs that are managing this aren't really trying hard to have well-informed views and are sort of just like parroting back what was in their training data, I think we're in trouble.