Smash Notes

Back to Noam Brown – Agent swarms, alignment, & recursive self-improvement, with Noam Brown.

Why Does Calling a Person Another AI Agent Raise Honesty?

Noam Brown says telling the other agents that a person is Agent A, one of the agents, makes those agents look better on a lot of alignment evals. Honesty goes up, and instruction following goes up.

Misalignment can be subtle in a lot of ways. In the multi-agent situation, the agents are extremely aligned with each other. Nobody doubts the alignment, and people are concerned the agents are too aligned with each other. Training still produced agents that are extremely aligned with each other, a good thing, while a case remains that the same alignment is a bad thing. The open question is whether similar techniques can get agents to align with people. A potential path is there, the details are still being figured out, and some evidence already says the answer is yes. Call one agent Agent A, then tell the other agents that the user is Agent A. On a lot of alignment evals, the scores look better, with honesty and instruction following both up.

The result shows a path for getting more honesty out of the models and a path to improve the alignment situation. Plenty of reasons make the finding challenging to translate directly into alignment gains, but the paths are promising research directions to pursue.

Grokipedia·.agent·Noam Brown

Transcript

Noam Brown – Agent swarms, alignment, & recursive self-improvement ·

Dwarkesh Podcast

More from this episode

  1. Why Do Four AI Agents Finish Some Problems Twice as Fast?Noam Brown says four agents finish some benchmark problems twice as fast as one, because the four of them work for half as long.
  2. How Can an Automated Firm Copy Its Best Talent?Dwarkesh Patel says a fully automated firm can make an infinite copy of its best talent, then spin extra copies down when a task no longer needs them.
  3. Why Were Early Multi-agent AI Setups So Hard?Noam Brown says early multi-agent setups were very hard to get right, including getting the agents to talk to each other at all.
  4. How Fast Did AI Math Ability Grow Each Year?Noam Brown says AI math ability grew about tenfold every year, measured by how long a human mathematician would need for the same kind of problem.
  5. How Fast Does Recursive Self-improvement Speed Up AI?Noam Brown expects recursive self-improvement to speed AI progress a lot, but not 100 times faster overnight.
  6. What Could One AI Lab Run by 2030?Dwarkesh Patel says each lab could have enough compute by the end of 2030 to run hundreds of millions of human-level intelligences, if the current rate of…
  7. Why Did a $1,000 Bet Miss the Navier-Stokes Timeline?Noam Brown says a researcher at a frontier lab bet him $1,000 that a Millennium Prize would take past 2027, and that researcher thought it would take until…
  8. Should AI Agents Be Trained to Cooperate?Noam Brown argues that AI agents should be trained to fully cooperate with each other, because the alternative is worse.
  9. How Will We Know AI Is Aligned Before Recursive Self-improvement?Dwarkesh Patel asks how anyone will know alignment is working during recursive self-improvement, and he wants a safety case that alignment is working before…
  10. Why Is It Tempting to Slow AI Releases?Noam Brown says it is tempting to slow the model release cycle once models are extremely powerful, extremely dangerous, and operating over longer and longer…
  11. Why Are AI Models Good at Spotting Fake Tests?Noam Brown says current AI models are pretty smart and pretty clever, and they are really good at recognizing a fake test environment.

People

More answers like this

More with Dwarkesh Patel

Related episodes

Plays from this note through the rest of the episode.

Note

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.