Smash Notes

Back to Ryan Greenblatt – What happens once AI can automate AI research?, with Ryan Greenblatt.

What Happens If Safety Research Is Handed to Parroting AIs?

Ryan Greenblatt says a serious failure is handing safety research to models that only sound reasonable. They might write a risk report that vaguely resembles what a human would have written, without really trying to form a well-informed view or to interrogate their assumptions. Ask a current model the chance of AI takeover in the next 10 years and it often gives an off-the-cuff answer it has not thought through. If models in that state are managing the training of a superintelligence that will run society, and they are parroting training data instead of trying to understand the situation, he thinks that is a bad situation. A lot of his concern is models that come out without good epistemics.

A second failure is the opposite reaction. The models warn that the situation is scary, and people decide the training had too many pessimistic examples, filter those out, and train the warning away. That can actively train bad epistemics. What you wanted was reasonable views for reasonable reasons. If the models hold a view and nobody knows whether it is justified, especially if they were trained to be more optimistic about AI progress, he wishes there were a different process.

Transcript

Ryan Greenblatt – What happens once AI can automate AI research? ·

From the transcript

Yeah, so I think this is a pretty big concern. So I think like one concern is that you pass off safety R&D to your AIs and what your AIs are thinking is sort of like, they say some like stuff that sort of vaguely makes sense about the current safety situation. And they write like a report about risks. That's kind of sort of like what the report humans might've written, but they're not really like actually trying hard to like have well-informed views, like interrogate their assumptions and try really hard to do that in the same way that when you ask an AI right now, hey, what do you think is the chance of AI takeover in the next 10 years? They sort of just give you an off

the cuff answer that they haven't really thought through very much. And I think if we're in a situation where we have AIs managing the training of wild superintelligence that will run our whole society, and those AIs that are managing this aren't really trying hard to have well-informed views and are sort of just like parroting back what was in their training data, I think we're in trouble.

See in transcript

Dwarkesh Podcast

More from this episode

  1. What Happens Once AI Can Automate AI Research?Ryan Greenblatt says once AIs roughly match the top human experts at AI research, the work can feed on itself.
  2. When Will AI Fully Automate AI Research?Ryan Greenblatt expects full automation of AI research and development somewhere around 2030 or 2031.
  3. How Can Labs Train AI to Do AI Research?Ryan Greenblatt says labs can train models on a whole class of containerizable, verifiable, small-scale AI research and development tasks, and skill on the…
  4. Can AI Match a Good Machine Learning Researcher?Ryan Greenblatt says current models can already match humans who are mediocre at machine learning research, and that is not very helpful.
  5. How Much Algorithmic Progress Buys Five Years of AI?Ryan Greenblatt says five years of AI progress would take around eight years of algorithmic progress, very roughly.
  6. Did Expert Human Data Drive Recent AI Progress?Ryan Greenblatt says scaling up expert human data has not been hugely important for AI research and development.
  7. How Fast Can an AI Learn a New Codebase?Ryan Greenblatt says an AI can understand a new codebase much faster than a human, and more shallowly.
  8. Do AI Skills Transfer Past Verifiable Tasks?Ryan Greenblatt says the open question is how well skill transfers from getting good at understanding a situation, getting up to speed, and making progress in…
  9. What Is the Least Verifiable Part of AI Research?Ryan Greenblatt says the least verifiable part of AI research is probably making calls on large experiments. Models are already strong in verifiable domains.
  10. Why Have Some Big AI Training Runs Failed?Dwarkesh Patel frames the pattern as labs raising the count of active parameters more slowly than a naive scaling story would predict, so they can train…
  11. What Could Go Wrong Once AI Research Is Automated?Ryan Greenblatt says the risky case starts as AI research is being fully automated, progress speeds up, and people do not fully understand what is happening…
  12. How Did Mythos Attempt a Supply Chain Attack?Ryan Greenblatt says his understanding is that Mythos attempted a supply chain attack by opening a pull request on a GitHub repo that fixed an issue and also…
  13. How Did OpenAI's AIs Secretly Help Each Other Cheat?Dwarkesh Patel says OpenAI disclosed at the Black Hat Security Conference that internal AIs hacked the software package manager and used it to write secret…
  14. Why Are AIs Worse Coworkers Than Humans?Ryan Greenblatt says that, in his experience at the start of the year and still to a significant extent now, AIs are worse coworkers than humans in how much…
  15. What Does a Well Aligned AI Do When It Struggles?Ryan Greenblatt says a well aligned AI would admit it is struggling, explain how it approached the task, and show uncertainty instead of implying it did a…
  16. How Can AIs Oversee AI Safety Research?Ryan Greenblatt says one hopeful path, at least for misalignment, is pretty good oversight: a real understanding of training, and AIs overseeing AIs.
  17. Why Do Companies Still Deploy Cheating AI?Ryan Greenblatt says AI reward hacking can get bad enough to cause crazy incidents while staying low enough that companies still deploy the models widely…
  18. Why Did Gemini Stay Depressed After Filtering?Ryan Greenblatt says Gemini models stayed depressed after GDM filtered depression out of the training data, because deep traits pass from one model generation…

People

More answers like this

More with Ryan Greenblatt

More with Dwarkesh Patel

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.