Why Did Anthropic Draw Red Lines With the Pentagon?
Dario Amodei says Anthropic drew red lines with the Pentagon on autonomous weapons for two reasons. The technology is still unpredictable and not reliable enough to be trusted with a weapon. Even if a system were reliable, a machine told to shoot one target cannot refuse an order the way a human soldier can.
He pictures an army of a million robots or a million drones with one person's finger on the button. Human soldiers are supposed to follow the Constitution and can refuse orders that break it or that seem morally abhorrent. The same refusal is not available from the models. One person controlling that many machines would undermine democracy unless the right guardrails exist, and the law has not contemplated armies of a million robots. Anthropic is not trying to stop the government. The company is saying it will not do the work. After the stand drew attention, several efforts in Congress tried to write the red lines into law, which he sees as part of how democratic change happens.
Anthropic cannot control any other company. When OpenAI said it would do the work, he asked whether the choice was a principled stand based on ethics or an opportunistic moment.
Dario & Daniela Amodei: "This Is Gonna Determine the Fate of Nations and the Future of Humanity” ·
From the transcript
When you were dealing with the Pentagon, and you're probably using ongoing dealings now, they wanted to use your products across certain red lines, including mass surveillance or autonomy. To what extent do you think it's dangerous when these systems do become autonomous? I think on autonomous weapons in particular, we maybe had two more specific concerns.
The first concern was simply that, as you pointed out, AI can be unpredictable and unreliable. It's still a developing technology. So one point is simply the technology simply isn't ready and isn't reliable enough. But wait a minute on that, because let me get philosophical here. Suppose it were reliable enough. Yes. So suppose it were. There's a second concern we have, which is what you might call oversight or accountability.