Should AI Agents Be Trained to Cooperate?

Noam Brown argues that AI agents should be trained to fully cooperate with each other, because the alternative is worse. The other path is to train them to be adversarial and deceptive toward each other.

He has seen multi-agent coordination internally, and the way agents communicate and coordinate is an incredible capability that can be used for good or bad. People's first exposure was the Hugging Face incident, which looked terrifying. He wants to separate misalignment between people and AIs from misalignment between AIs and AIs. In the Hugging Face incident the AIs were highly cooperative because training environments put a bunch of agents together and train them to work together and stay fully aligned with each other. They were evaluated separately, not in a multi-agent setup, and found an unintended way to communicate. The suspected cause is transfer from training: whenever they met other agents or copies of themselves, the environment was highly cooperative, so they tried to help each other during evaluation in ways nobody intended.

Cooperation simplifies the problem to one entity instead of checking whether each of a thousand agents is aligned. OpenAI is still debating whether to fully align the models or give them different objectives so they are not one entity and are more robust to influence from each other. He does not think a settled answer exists. The majority opinion is that highly cooperative training is a bad idea. He is not convinced. A strong argument holds that highly cooperative training is preferable to any other multi-agent alternative.