Charlie O'Neill
Related people
Episodes
Notes
Dwarkesh Podcast · AI researchers debate how close we are to recursive self-improvement
-
When Will AI Work as a Remote Worker?
Charlie O'Neill puts a non-browser remote worker, one that can send Slack messages and do the rest of the job, at around a year, and a browser-bound worker at a couple of years, unless the firm makes its information programmatically accessible. He also says the model will not yell at someone at work or push a person to finish a task, because it stays too nice.
Beren Millidge is unsure of the pace and still places full generality at three years. The gap between one year and three years is a long tail of miscellaneous tasks some human can do, and that tail takes a while. People will make their organizations easier for AI to use, so the tools can reach 80 or 90 percent of the way before then.
-
Can a Little Reinforcement Learning Change an AI Model?
Beren Millidge argues that a little reinforcement learning can still change an AI model in a dramatic way. Even one bit can shift the input-to-output mapping a lot and rule out half the hypothesis space, so a small parameter update from a strong starting point can produce a large change in behavior.
Charlie O'Neill says math training does not necessarily make a model the best coder, and code still needs its own reinforcement learning environments. What did generalize is the horizon. Models learned to spend more tokens, keep working longer, and still make progress. A paper called EdgeBench showed that the rate at which models can work for longer is doubling every three months. He compares th
-
Why Does Mid-training Do Most of the Work in Reinforcement Learning?
Beren Millidge says reinforcement learning has worked better than a naive bit count predicts because strong mid-training often takes a model almost 80% of the way to the final checkpoint, and the later stage only tweaks the policy with a few high-signal bits.
Mid-training is pre-training on synthetic reasoning data and on environments that warm-start the model. The policy does not have to learn the behaviors from scratch. Those bits carry much higher signal than ordinary pre-training, which is why reinforcement learning beats simply copying successful reasoning traces. In supervised fine-tuning the answer token is still in the trace, but the objective also has to match every reasoning token
-
Why Does Reinforcement Learning Need a Math Curriculum?
Charlie O'Neill says a recent paper trained a decent language model only through fifth-grade math and primary-school English, then tried reinforcement learning on late high school and college math. The gap was too large, and the model did not climb. Year-by-year rungs change the outcome. Year seven math, then year eight, and so on, make a climb to year twelve possible. The distance between the rungs, and how hard each rung is to create, decides whether the climb happens.
Beren Millidge says reinforcement learning is not very good at exploring. A model that fails to find a useful signal inside 128 rollouts is very unlikely to progress. That is why reinforcement learning needs a curriculum an
-
Is AI Taste Just a Longer Context Window?
Charlie O'Neill asks whether research taste is just a longer context window. Give a model a trillion tokens, enough to hold a person's experience before RLHF, with the same sample efficiency it already shows at a million tokens, and ask if taste is solved.
John Schulman says no. The model would still have to be trained to learn the right update from that context, or the skill would have to generalize. A life cannot simply be dumped in.
Beren Millidge says a trillion-token window would still need trillion-length data to train, and a 10k context cannot hold that dump. Humans do not live to be 10,000 and still pick up taste quickly. The difference between a first-year PhD student and a final
-
How Does Cursor Ship a New Composer Model?
Charlie O'Neill says Cursor's online reinforcement learning for Composer has no groups. One user says one thing and the team gets one rollout, which creates a variance problem. Cursor's answer was heuristics that estimate how much better or worse than average a response was, then a large REINFORCE update. If the new model improved on CursorBench, they deployed it every five hours. If it did not, they threw that version out.
The older Tab model already learned every day from whether people pressed Tab on the next completion. Composer then did the same thing for the generative model. A person still chooses the signals and builds environments from deployment data, so the loop is slower than a
-
How Do AI Labs Plan to Train a Drop-in Remote Worker?
Dwarkesh Patel says the bet AI labs are making is to scale RLVR training across millions of diverse environments and hundreds of domains. What emerges is an agent with skills he calls basic, or less than basic: staying persistent, triaging information and context, and eventually doing end-to-end work with other agents. In-context learning keeps getting scaled so the agent can run for a long time. What comes out functions like a drop-in remote worker over a week or a month. The learning happens inside simulations in a data center. The agent learns how to learn there, then gets deployed without learning from real-world deployment.
Charlie O'Neill says that picture fits the models labs deploy
-
What Is the Last Human Job in AI Research?
Dwarkesh Patel asked which innovations since 2012, engineering or conceptual, would be the last work humans still do before AI automates AI research. Beren Millidge says that job is deciding which experiments to run. Models can be pointed at an experiment and still propose only miscellaneous, very tiny steps. They are weaker at choosing experiments than at coding them.
John Schulman says the role that lasts longest is defining the objective and deciding what people actually want: how assistants should behave, what helpful means, the objective in reinforcement learning from human feedback, and later constitutions and model specs. Alignment splits into specification, figuring out the right ob