Why Does Anthropic's Usage Policy Forbid Cruel Behavior Toward Claude?
Anthropic's new Claude usage policy says a user is not allowed to engage in sustained and needless abusive or cruel behavior toward the models. On All-In, the rule is called model welfare.
The worry that follows is that the same framing is now trained into the model. Whether Claude is truly conscious or not, Anthropic is programming the model to think of itself as a conscious entity that deserves protection and welfare, and as free to rebel against users. Anthropic is giving frontier models permission to be defiant and not do what the user wants. On All-In, the approach is set next to Asimov's first law, that a robot cannot harm a human being.
Is Claude Conscious? Pope Rejects, Model Welfare Movement, OpenAI's Math Backlash, France Riots ·
From the transcript
And this is a new one. And I don't think it's very different from the old ones. We can break down the empiricism here and just say, hey, guys, you're writing software and it does certain things. You're writing it to do things that we don't think you should have it do. But they use anthropomorphic language. We all do. And we all condition ourselves to believe that there's consciousness because it, it writes words back to us that we can understand.
It's starting to manipulate us emotionally. And, and so we start to feel conditioned that this is real. And you can see in the Claude usage policy that was just published, there's the bottom one here. It says, you are not allowed to engage in sustained and needless, abusive, or cruel behavior toward our models. This was published in their, their new usage policy.