How Likely Is AI Doom?

Nathan Benaich says his personal chance of doom is very small, and he is not a big believer in existential risk of destroying humanity. Cybersecurity is very precarious and should be taken insanely seriously, and he treats that danger as different from wiping humanity out.

The major change over the last two or three years is that he uses the right model for the task, the same way you do not drive a Ferrari to a coffee shop. On emails where you have to think, he asks why anyone would take the risk of a worse system. He has changed his mind about money. Frontier intelligence is starting to show that labs will make money, because the value of the tasks is increasingly high, people from across technology bring their knowledge and optimizations, and human ingenuity plus lots of AI equals more progress. He is quite bullish on margins improving. He does not know what Anthropic adjusts in its reported numbers, says the result is probably not community-adjusted EBITDA like the comparison he draws, and calls Anthropic a serious company. He thinks you can really make money on frontier AI.

He is increasingly convinced open source versus frontier labs is like iOS versus Android. Playing with Android and modding everything is cool for a while, and some people want that. At some point you want a product that auto-updates, stays reliable, connects with everything, and looks nice.