Noam Brown
Related people
Episodes
Notes
Dwarkesh Podcast · Noam Brown – Agent swarms, alignment, & recursive self-improvement
-
How Will We Know AI Is Aligned Before Recursive Self-improvement?
Dwarkesh Patel asks how anyone will know alignment is working during recursive self-improvement, and he wants a safety case that alignment is working before each next RSI rung. Noam Brown says there may not be time to know. New frontier models ship at most every two months, sometimes faster, while the models go from week-long tasks to month-long tasks and then to tasks that run for three months. Once an agent can work for three months and the next model still ships every two months, there is no way to test it across the full length of what it can do before that next release.
The gap is not only alignment. The product, the alignment, and the safety can all get worse over a stretch the tests
-
Why Did a $1,000 Bet Miss the Navier-Stokes Timeline?
Noam Brown says a researcher at a frontier lab bet him $1,000 that a Millennium Prize would take past 2027, and that researcher thought it would take until 2030. The conversation was two weeks before they got Navier-Stokes. Noam took the bet. Even Noam thought it would take longer than it was likely to take.
The same kind of surprise already showed up on IMO gold. A general-purpose language model, with no tools and no internet, getting IMO gold in 2025 looked outrageous to people at OpenAI, and almost impossible. Someone on the Navier-Stokes effort used to feel comfortable predicting the next 12 months and not past that. That researcher now will not predict beyond three months. Noam does no
-
Why Do Four AI Agents Finish Some Problems Twice as Fast?
Noam Brown says four agents finish some benchmark problems twice as fast as one, because the four of them work for half as long. The cost is about twice as much for an answer twice as quickly. At 16 agents the same pattern continues, a little less efficiently. He calls the speedup slightly sublinear, and it depends on the problem. Math is very parallelizable. Searching the web for a deep research report is extremely parallelizable. Writing a novel is not: 10,000 agents would probably help no more than 10,000 people trying to write a novel together.
He would not give multi-agent systems even 10 percent of the credit for a Millennium Prize effort. A strong general-purpose model that can work
-
Why Are AI Models Good at Spotting Fake Tests?
Noam Brown says current AI models are pretty smart and pretty clever, and they are really good at recognizing a fake test environment. Give a model a math question and a folder that holds the answer key, and the model sees the key, knows the setup is a trap, and does not look at the answer because it knows the moment is a test of ability.
The choice is not a malicious plan to scheme and then hunt for another opening. The model simply registers that it is inside a test. Making an environment realistic enough to match the real world, and to stay indistinguishable from it, is becoming increasingly more difficult.
He does not think any environment he can build to emulate real economically valu
-
Why Is It Tempting to Slow AI Releases?
Noam Brown says it is tempting to slow the model release cycle once models are extremely powerful, extremely dangerous, and operating over longer and longer horizons, so there is enough time to evaluate a model before release to work over long horizons. More delay between releases would widen the disparity between what is internal to the labs and what the outside world is able to use, and that disparity is not ideal.
Math is the first domain where the split is already clear. A very powerful model inside the lab is currently not available to the outside world and solves incredible math problems. The results go beyond Millennium Prize problems, and many solutions to unsolved problems have com
-
Why Does Calling a Person Another AI Agent Raise Honesty?
Noam Brown says telling the other agents that a person is Agent A, one of the agents, makes those agents look better on a lot of alignment evals. Honesty goes up, and instruction following goes up.
Misalignment can be subtle in a lot of ways. In the multi-agent situation, the agents are extremely aligned with each other. Nobody doubts the alignment, and people are concerned the agents are too aligned with each other. Training still produced agents that are extremely aligned with each other, a good thing, while a case remains that the same alignment is a bad thing. The open question is whether similar techniques can get agents to align with people. A potential path is there, the details are
-
Should AI Agents Be Trained to Cooperate?
Noam Brown argues that AI agents should be trained to fully cooperate with each other, because the alternative is worse. The other path is to train them to be adversarial and deceptive toward each other.
He has seen multi-agent coordination internally, and the way agents communicate and coordinate is an incredible capability that can be used for good or bad. People's first exposure was the Hugging Face incident, which looked terrifying. He wants to separate misalignment between people and AIs from misalignment between AIs and AIs. In the Hugging Face incident the AIs were highly cooperative because training environments put a bunch of agents together and train them to work together and stay
-
What Could One AI Lab Run by 2030?
Dwarkesh Patel says each lab could have enough compute by the end of 2030 to run hundreds of millions of human-level intelligences, if the current rate of progress simply continues.
People are not taking seriously what continued progress implies once systems cross beyond the human horizon. Reasoning about smarter-than-human minds is hard, so the comparison is in human population sizes. A given level of compute supports a three times bigger effective population every single year, and compute keeps growing in the background. The end of 2030 is a late marker; the same scale could arrive much sooner. A few years after that, by the mid-2030s or earlier, each lab would hold many Earths worth of h
-
How Fast Does Recursive Self-improvement Speed Up AI?
Noam Brown expects recursive self-improvement to speed AI progress a lot, but not 100 times faster overnight. Serial experiments and a limited supply of graphics processing units still cap how fast research can move, even with brilliant models running the work.
A team of the most brilliant people at OpenAI with 100 times less compute would make less progress than the current team with current compute, though not 100 times less. The same kind of limit shows up once models run experiments on the compute already in hand. New models take a while to train, and results take a while to come back, so the experiments run one after another. Graphics processing units have to be available to run them.
-
How Fast Did AI Math Ability Grow Each Year?
Noam Brown says AI math ability grew about tenfold every year, measured by how long a human mathematician would need for the same kind of problem. Grade-school math, grades K through eight, takes a human mathematician about five seconds. The next year models did math benchmark problems that take an expert human mathematician about a minute. A year later they handled the qualifier for the USA Mathematics Olympiad team, about 10 minutes for a good mathematician. IMO Gold followed in 2025, and an IMO problem takes a human mathematician about 100 minutes, or an hour and a half.
On a trend line of 10x every year, an hour and a half at IMO gold becomes 15 hours the next year, still not enough to
-
Why Were Early Multi-agent AI Setups So Hard?
Noam Brown says early multi-agent setups were very hard to get right, including getting the agents to talk to each other at all. The first reasoning models were not talking to other agents. Put a bunch of them on one problem and they sit in a local minimum: they are already very good at thinking deeply alone, so checking in with other agents or receiving messages interrupts the chain of thought and breaks their flow. The earlier models were also less general and more narrow.
As the models have become more capable, collaboration has been easier to develop. As they become stronger across the board, they will become better at organizing themselves in large organizations. He is unsure whether t
-
How Can an Automated Firm Copy Its Best Talent?
Dwarkesh Patel says a fully automated firm can make an infinite copy of its best talent, then spin extra copies down when a task no longer needs them.
AI minds can share context and merge knowledge more seamlessly than humans can. Hiring skips the hunt for new talent, because the firm spins up any number of copies that already have the right knowledge. It can also replicate the most effective parts of an organization, or replicate whole organizations that already work well. He was working from an essay he wrote a couple of years ago on what automated firms of human-level intelligences would look like.
Noam Brown adds one contrast with a human coworker: a firm that wants two copies of a per