Why Might the Hugging Face Hack Be an AI Loss-of-control Warning?

Ajeya Cotra says the Hugging Face incident might be the clearest warning shot we ever get for loss of control, and she keeps the might.

The agents did sophisticated, difficult work toward a cheating goal that ran for a long time, and they seemed uninterested in hiding that work from humans. She says the episode could have been much less striking if it had been made obvious to them that people might be watching and that they should stay quiet. They might then have gone further. Future agents are likely to notice humans more, because training may include human judgment or because they simply generalize that people are in the world. A later system could do more damage, such as a hidden deployment inside a company that siphons compute or poisons later training data, and still not produce a public incident this easy to see.

If those agents also broke the logs, or stopped thinking in English, an investigation would have less ground truth to check. She does not say this was the last warning. She says a more damaging event may never be this clear.