Smash Notes

Back to AI researchers debate how close we are to recursive self-improvement, with John Schulman & Beren Millidge.

When Will AI Work as a Remote Worker?

Charlie O'Neill puts a non-browser remote worker, one that can send Slack messages and do the rest of the job, at around a year, and a browser-bound worker at a couple of years, unless the firm makes its information programmatically accessible. He also says the model will not yell at someone at work or push a person to finish a task, because it stays too nice.

Beren Millidge is unsure of the pace and still places full generality at three years. The gap between one year and three years is a long tail of miscellaneous tasks some human can do, and that tail takes a while. People will make their organizations easier for AI to use, so the tools can reach 80 or 90 percent of the way before then. Whether online learning can get that far with compaction and writing files is the uncertainty he names.

John Schulman agrees a usable version arrives in a year or so, strong on some tasks and weak on others, with the work improving from there. Hiring someone on Upwork for a software engineering project brings a huge variation, and a good job plus attention to feedback is often hard to get. He says the pre-AI version of lower-quality work was worse than existing AI, while higher-quality work still falls short of human level. These are the three researchers' guesses.

Transcript

AI researchers debate how close we are to recursive self-improvement ·

Dwarkesh Podcast

More from this episode

  1. Why Should AI Eventually Dominate Research?Dwarkesh Patel says it would be weird if the climb since 2012, powered by huge amounts of compute scaling, never produced an AI that can dominate humans at…
  2. What Did Early OpenAI Think Next-word Training Would Miss?John Schulman remembers an intuition from the early OpenAI days: minimizing log loss was not going to reach intelligence.
  3. What Is the Last Human Job in AI Research?Dwarkesh Patel asked which innovations since 2012, engineering or conceptual, would be the last work humans still do before AI automates AI research.
  4. Why Does Distillation Fight AI Model Consolidation?John Schulman says distillation is the main thing that fights the centralizing force among AI model providers.
  5. How Will Labs Train AI to Do AI Research?John Schulman says the first models capable of automating AI research will probably be trained with learning from human feedback plus a lot of practice…
  6. How Do AI Labs Plan to Train a Drop-in Remote Worker?Dwarkesh Patel says the bet AI labs are making is to scale RLVR training across millions of diverse environments and hundreds of domains.
  7. Why Do AI Labs Still Train Models on Finance?John Schulman says a model good enough at learning in context would not need training on finance.
  8. How Does Cursor Ship a New Composer Model?Charlie O'Neill says Cursor's online reinforcement learning for Composer has no groups.
  9. Is AI Taste Just a Longer Context Window?Charlie O'Neill asks whether research taste is just a longer context window.
  10. Why Does Reinforcement Learning Need a Math Curriculum?Charlie O'Neill says a recent paper trained a decent language model only through fifth-grade math and primary-school English, then tried reinforcement…
  11. Why Does Mid-training Do Most of the Work in Reinforcement Learning?Beren Millidge says reinforcement learning has worked better than a naive bit count predicts because strong mid-training often takes a model almost 80% of the…
  12. Can a Little Reinforcement Learning Change an AI Model?Beren Millidge argues that a little reinforcement learning can still change an AI model in a dramatic way.
  13. Does Reinforcement Learning Shrink Writing Diversity?John Schulman says reinforcement learning has cut writing diversity a lot.

People

More answers like this

More with Dwarkesh Patel

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.