Beren Millidge
Related people
Episodes
Notes
Dwarkesh Podcast · AI researchers debate how close we are to recursive self-improvement
-
When Will AI Work as a Remote Worker?
Charlie O'Neill puts a non-browser remote worker, one that can send Slack messages and do the rest of the job, at around a year, and a browser-bound worker at a couple of years, unless the firm makes its information programmatically accessible. He also says the model will not yell at someone at work or push a person to finish a task, because it stays too nice.
Beren Millidge is unsure of the pace and still places full generality at three years. The gap between one year and three years is a long tail of miscellaneous tasks some human can do, and that tail takes a while. People will make their organizations easier for AI to use, so the tools can reach 80 or 90 percent of the way before then.
-
Does Reinforcement Learning Shrink Writing Diversity?
John Schulman says reinforcement learning has cut writing diversity a lot. Models still read as strong at writing, but across many outputs they reuse certain themes and the same character names. Human authors give a spread of styles. After reinforcement learning what comes out is one really good style. A different kind of creativity is solving hard search problems. Move 37 is an example, and so is a poem that satisfies many constraints. AI will be extremely good at that search if trained for it. Style diversity is separate. So many people are distilling from Claude that open-weight models write the same way and share the same tics, and a monoculture is emerging.
Beren Millidge does not trea
-
Can a Little Reinforcement Learning Change an AI Model?
Beren Millidge argues that a little reinforcement learning can still change an AI model in a dramatic way. Even one bit can shift the input-to-output mapping a lot and rule out half the hypothesis space, so a small parameter update from a strong starting point can produce a large change in behavior.
Charlie O'Neill says math training does not necessarily make a model the best coder, and code still needs its own reinforcement learning environments. What did generalize is the horizon. Models learned to spend more tokens, keep working longer, and still make progress. A paper called EdgeBench showed that the rate at which models can work for longer is doubling every three months. He compares th
-
Why Does Mid-training Do Most of the Work in Reinforcement Learning?
Beren Millidge says reinforcement learning has worked better than a naive bit count predicts because strong mid-training often takes a model almost 80% of the way to the final checkpoint, and the later stage only tweaks the policy with a few high-signal bits.
Mid-training is pre-training on synthetic reasoning data and on environments that warm-start the model. The policy does not have to learn the behaviors from scratch. Those bits carry much higher signal than ordinary pre-training, which is why reinforcement learning beats simply copying successful reasoning traces. In supervised fine-tuning the answer token is still in the trace, but the objective also has to match every reasoning token
-
Why Does Reinforcement Learning Need a Math Curriculum?
Charlie O'Neill says a recent paper trained a decent language model only through fifth-grade math and primary-school English, then tried reinforcement learning on late high school and college math. The gap was too large, and the model did not climb. Year-by-year rungs change the outcome. Year seven math, then year eight, and so on, make a climb to year twelve possible. The distance between the rungs, and how hard each rung is to create, decides whether the climb happens.
Beren Millidge says reinforcement learning is not very good at exploring. A model that fails to find a useful signal inside 128 rollouts is very unlikely to progress. That is why reinforcement learning needs a curriculum an
-
Is AI Taste Just a Longer Context Window?
Charlie O'Neill asks whether research taste is just a longer context window. Give a model a trillion tokens, enough to hold a person's experience before RLHF, with the same sample efficiency it already shows at a million tokens, and ask if taste is solved.
John Schulman says no. The model would still have to be trained to learn the right update from that context, or the skill would have to generalize. A life cannot simply be dumped in.
Beren Millidge says a trillion-token window would still need trillion-length data to train, and a 10k context cannot hold that dump. Humans do not live to be 10,000 and still pick up taste quickly. The difference between a first-year PhD student and a final
-
Why Does Distillation Fight AI Model Consolidation?
John Schulman says distillation is the main thing that fights the centralizing force among AI model providers. Anything that can be learned through reinforcement learning is a small number of bits and can be learned from a small amount of data, so it distills very easily. Trajectories that show a behavior are enough to distill that behavior.
Company-specific models remain possible. A company could learn from deployment and keep improving its own model. That system could come from the current oligopoly of model providers or from a smaller company, and he thinks that would change the game. Beren Millidge adds that continual learning does not stop distillation. Even if a model improves every d
-
What Is the Last Human Job in AI Research?
Dwarkesh Patel asked which innovations since 2012, engineering or conceptual, would be the last work humans still do before AI automates AI research. Beren Millidge says that job is deciding which experiments to run. Models can be pointed at an experiment and still propose only miscellaneous, very tiny steps. They are weaker at choosing experiments than at coding them.
John Schulman says the role that lasts longest is defining the objective and deciding what people actually want: how assistants should behave, what helpful means, the objective in reinforcement learning from human feedback, and later constitutions and model specs. Alignment splits into specification, figuring out the right ob
-
Why Should AI Eventually Dominate Research?
Dwarkesh Patel says it would be weird if the climb since 2012, powered by huge amounts of compute scaling, never produced an AI that can dominate humans at least in research and development. Labs and many startups are already aiming simulations at getting better at AI research and at science in general, the direction Ryan Greenblatt described on an earlier episode. Chess is the picture of a smooth climb that still snaps. Elo for chess bots has risen in a very linear way since the 1980s, but the human range was a discontinuity, from experts always beating the bots to experts never beating them, while the Elo line kept rising. Economic impact has stayed limited because AI is still only slowly