John Schulman
Related people
Episodes
Notes
Dwarkesh Podcast · AI researchers debate how close we are to recursive self-improvement
-
When Will AI Work as a Remote Worker?
Charlie O'Neill puts a non-browser remote worker, one that can send Slack messages and do the rest of the job, at around a year, and a browser-bound worker at a couple of years, unless the firm makes its information programmatically accessible. He also says the model will not yell at someone at work or push a person to finish a task, because it stays too nice.
Beren Millidge is unsure of the pace and still places full generality at three years. The gap between one year and three years is a long tail of miscellaneous tasks some human can do, and that tail takes a while. People will make their organizations easier for AI to use, so the tools can reach 80 or 90 percent of the way before then.
-
Does Reinforcement Learning Shrink Writing Diversity?
John Schulman says reinforcement learning has cut writing diversity a lot. Models still read as strong at writing, but across many outputs they reuse certain themes and the same character names. Human authors give a spread of styles. After reinforcement learning what comes out is one really good style. A different kind of creativity is solving hard search problems. Move 37 is an example, and so is a poem that satisfies many constraints. AI will be extremely good at that search if trained for it. Style diversity is separate. So many people are distilling from Claude that open-weight models write the same way and share the same tics, and a monoculture is emerging.
Beren Millidge does not trea
-
Is AI Taste Just a Longer Context Window?
Charlie O'Neill asks whether research taste is just a longer context window. Give a model a trillion tokens, enough to hold a person's experience before RLHF, with the same sample efficiency it already shows at a million tokens, and ask if taste is solved.
John Schulman says no. The model would still have to be trained to learn the right update from that context, or the skill would have to generalize. A life cannot simply be dumped in.
Beren Millidge says a trillion-token window would still need trillion-length data to train, and a 10k context cannot hold that dump. Humans do not live to be 10,000 and still pick up taste quickly. The difference between a first-year PhD student and a final
-
How Does Cursor Ship a New Composer Model?
Charlie O'Neill says Cursor's online reinforcement learning for Composer has no groups. One user says one thing and the team gets one rollout, which creates a variance problem. Cursor's answer was heuristics that estimate how much better or worse than average a response was, then a large REINFORCE update. If the new model improved on CursorBench, they deployed it every five hours. If it did not, they threw that version out.
The older Tab model already learned every day from whether people pressed Tab on the next completion. Composer then did the same thing for the generative model. A person still chooses the signals and builds environments from deployment data, so the loop is slower than a
-
Why Do AI Labs Still Train Models on Finance?
John Schulman says a model good enough at learning in context would not need training on finance. It could read the books on the fly and work in the appropriate jurisdiction. Labs might still bake those domain skills into the weights with reinforcement learning so the model is more efficient at runtime. In practice, model providers go domain by domain and strengthen the highest-value domains and the most common skills. He counts that coverage as one reason the models have gotten so much better.
Dwarkesh Patel had asked why a lab would bake in PowerPoint if a model can learn on the job the way a person does. Before Schulman answered, Patel offered two candidates: the models are not there yet
-
How Will Labs Train AI to Do AI Research?
John Schulman says the first models capable of automating AI research will probably be trained with learning from human feedback plus a lot of practice environments. Human feedback is there to absorb the researcher's taste. The environments are multi-step research projects. Each iteration patches whatever seems most broken. Researchers using the systems notice consistent weaknesses and patch them by collecting more human feedback or by building another environment.
Dwarkesh Patel had framed a toy version, drawn from a point Ryan Greenblatt made, in which a later model builds smaller models for inner-loop challenges such as beating video games that require continual learning or reaching a ce
-
Why Does Distillation Fight AI Model Consolidation?
John Schulman says distillation is the main thing that fights the centralizing force among AI model providers. Anything that can be learned through reinforcement learning is a small number of bits and can be learned from a small amount of data, so it distills very easily. Trajectories that show a behavior are enough to distill that behavior.
Company-specific models remain possible. A company could learn from deployment and keep improving its own model. That system could come from the current oligopoly of model providers or from a smaller company, and he thinks that would change the game. Beren Millidge adds that continual learning does not stop distillation. Even if a model improves every d
-
What Is the Last Human Job in AI Research?
Dwarkesh Patel asked which innovations since 2012, engineering or conceptual, would be the last work humans still do before AI automates AI research. Beren Millidge says that job is deciding which experiments to run. Models can be pointed at an experiment and still propose only miscellaneous, very tiny steps. They are weaker at choosing experiments than at coding them.
John Schulman says the role that lasts longest is defining the objective and deciding what people actually want: how assistants should behave, what helpful means, the objective in reinforcement learning from human feedback, and later constitutions and model specs. Alignment splits into specification, figuring out the right ob
-
What Did Early OpenAI Think Next-word Training Would Miss?
John Schulman remembers an intuition from the early OpenAI days: minimizing log loss was not going to reach intelligence. The important bits account for such a small fraction of the loss that noise would overwhelm them. Training a language model on next-token prediction was not going to learn the interesting things, so researchers needed better objectives that put more emphasis on what mattered.
Plenty of arguments pointed the same way. People do not learn how to model everything in the environment, and most people cannot create a photorealistic reproduction of a scene they have looked at. A better objective looked necessary. Next-token prediction worked anyway.