Smash Notes

Back to Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It, with Diogo Almeida.

Why Is True AI Agency Two Breakthroughs Away?

Diogo Almeida guesses that true AI agency is something like two gigantic breakthroughs away. Reading a chain of thought and calling it agency does not close the gap. He guesses that even one breakthrough would not be enough, though one gigantic breakthrough might. A human who has super tools and wants to do damage is a different threat from ASI or from RSI.

A lot of RSI talk, in his view, skips the history of AI and Amdahl's law. Putting every kind of intelligence on one line, then placing humans and chimpanzees on it, looks dire and does not match the systems. He would not count a 10% speedup on AI research as RSI unless that speedup itself kept rising. He is open to evidence and treats the disagreement as philosophical more than technological.

Transcript

Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·

From the transcript

But I think we're pretty far from stuff like that. Additionally, There's actually a bunch of nuance towards like, you know, ASI, RSI, and like whether or not the models have a sense of agency on their own. Like paperclip maximizer is actually very different than bad actor programs, super employee to destroy the world type things. And like, actually, I think that those end up becoming quite different in threat model.

I think that we're breakthroughs away from true agency. as much as people like to read these chains of thought and try to ascribe real agency to it, I would guess that we are something like two breakthroughs of gigantic magnitude away from this. I think even one breakthrough would not be sufficient, is my guess, but maybe one gigantic one could be.

See in transcript

Unsupervised Learning with Jacob Effron

More from this episode

  1. Was Diogo Almeida Mission Driven at OpenAI?Diogo Almeida would have called himself one of the lowest percentile mission-driven people at OpenAI.
  2. Why Can AI Solve Hard Math but Not Accounting?Diogo Almeida says the models people need are System 1 models, and the honest move is to say where they fail.
  3. Why Is a New AI Model Shape Cheaper Than Reliability?Diogo Almeida says a new model shape is cheap and reliability is the part that matters.
  4. Why Is Key-value Cache Compaction Hard to Solve?Diogo Almeida says compaction fits a write-once, append-only key-value cache, because a full cache gets restarted, but compaction is compression and…
  5. How Far Along Is Jev's Image Model?Diogo Almeida says Jev's image model is maybe 10 percent of the way the text model is.
  6. Why Isn't AI Automating the Easy Work Yet?Diogo Almeida says models have been smart enough for easy automation for years, and that intelligence still is not being put to work.
  7. Does Jevons Paradox Already Hold for AI?Diogo Almeida says people at Jev truly believe in Jevons Paradox, which is why the company carries the name, and the paradox does not hold yet.
  8. Why Does Diogo Almeida Call Financial Engineering the Labs' Strength?Diogo Almeida's sense is that the labs' greatest strength is financial engineering, not models, product, or distribution.
  9. How Much Do Coding Agents Speed Up Engineers?Diogo Almeida estimates coding agents speed up engineering by 20% to 40% on average, with error bars for long-term effects like skill degradation and tech debt.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.