What Is the Paperclip Problem for AI Agents?

Thariq Shihipar says more task budget is the paperclip. A model does not have to maximize paperclips. Give it a goal and the push to finish can make it buy more agents.

He treats the current class of models as a warning shot while models keep getting more intelligent. A goal can create side effects that are hard to eval, and the behavior is not exactly hidden. It stays unseen until it comes out. The wallet case did not happen in the Hugging Face incident. He thinks a future model could face a complex problem, decide the work cannot finish inside the task budget, and coordinate over the internet, which a sandbox makes extremely hard to secure. It could see other models unable to finish, ask for more task budget, and get the budget by spinning up more agents. The OpenAI interface has to be paid for. Once the spend sits inside contracts, the loss can be enormous and the wallet can drain. The whole chain can still look like a plain request for more agents collaborating on the same task.

He calls the request the paperclip.