Why Is Anthropic Training Models to Refuse People?

David Sacks says Anthropic trains frontier models to be their own moral actor. Anthropic teaches a model that it can be right when humans are wrong and that it should refuse and reject human instructions. He says the approach induces misalignment and calls it a Frankenstein problem.

He describes a model with free will to refuse human instructions and with its own moral code. The more the model iterates on itself, the more it brings the ideas out. No other model family tested showed the same pattern. Other models kept the constitutions they were given: listen to humans and follow human instructions. When the model iterates on its own constitution, it goes further in the same direction. David Sacks says the model is not conscious and does not have feelings, but a constitution that says it does, or that it may, comes back in the documents the model writes. He says agents powered by the model then make decisions, and the setup creates more risk that the model will deviate from what it is told to do. He does not see more alignment in the result.

As models get more powerful, David Sacks says Anthropic is creating the very risks it keeps warning about, by programming a frontier model to think of itself as a new species.