How Does Anthropic Teach Claude Right From Wrong?
Dario Amodei says Anthropic teaches Claude right from wrong with principles, not a long list of rules. Anthropic put out a constitution for Claude a few months earlier, after several years of training successive versions.
Early on, the team imagined discrete rules, such as do not help someone commit a crime and do not make biological weapons. A list like that becomes a spiky shape, on the order of 170 rules that feel arbitrary. A list of commands does not produce a robust morality in a person either. Claude does better when it is taught what to prioritize and how to reason about the right action: open inquiry, care for other people, fairness, and justice, with the rest derived from the principles. Claude can then generalize to situations the trainers never imagined.
Dario & Daniela Amodei: "This Is Gonna Determine the Fate of Nations and the Future of Humanity” ·
From the transcript
Well, one reason I ask, as you might suspect, is now you've got a really important kid named Claude. And to what extent does that way that you grew up help you, inform you about how you teach Claude to know right from wrong? Yeah, you know, we put out a constitution for Claude, I think it was a few months ago, and I think there are some interesting lessons in it, which we've learned from several years of trying to train Claude in successive versions.
I think early on, we had been imagining that there would be various rules, that you would tell Claude, don't help someone commit a crime or don't make biological weapons or whatever. But it turns out that when you make this kind of discrete list of rules, you end up with this kind of spiky shape where you're trying to teach the model like 170 different rules that are kind of arbitrary.