Why Does Anthropic Block Training on Its Thinking?

Katelyn Lesse says Anthropic blocks chain of thought distillation because the company exists so AI creates positive outcomes in the world safely. Safeguard classifiers run on the tokens the models produce. Distilled models run without safeguard classifiers could bring bad outcomes, and Anthropic does not want people to distill the models for malicious or bad reasons.

People using the models for good should still get the intelligence out, but some tradeoffs are required to stop distillation. User experience hits come in two layers. An integration change is annoying, yet people can still accomplish the same things. The second layer is when people can no longer accomplish what they could before, and the team tries hardest to avoid that hit. On the last couple of models, incremental changes enforce preserved thinking blocks across turns, including when someone uses multiple accounts. They reached out to open source harness maintainers and to big customers building harnesses on the API so the upgrade would go well. Compaction strategies at some customers would have broken under preserved thinking blocks, so Anthropic shipped a new compaction feature in the API that replaces the manual process and still works with preserved thinking.

The measures they can look at show distillation is harder.