Why Is Safety Instinct's Hardest Problem?

Noah Shinn calls safety the most important problem while building Instinct. Security, safety, and a mindset tied to the company's core values have to shape the people and the product. The user is always in control of their data and should never feel out of control.

Any new breakout consumer experience over the past 20 or 30 years has drawn immediate backlash that confuses different with unsafe, and the same pattern is showing up now. Language models hallucinate all the time, but Instinct should not. After one hallucination case, a systematic check now runs before an action is taken and before a thinking trace is executed as a tool call. A watchdog filter, decoupled from the same incentive system as the underlying agent, can spot a proper noun invented by a sampling error in the model. World-class security teams keep hunting harder adversarial edge cases. The mistakes are becoming rarer, and more creative testing makes the models more robust to attacks.