Why Is True AI Agency Two Breakthroughs Away?
Diogo Almeida guesses that true AI agency is something like two gigantic breakthroughs away. Reading a chain of thought and calling it agency does not close the gap. He guesses that even one breakthrough would not be enough, though one gigantic breakthrough might. A human who has super tools and wants to do damage is a different threat from ASI or from RSI.
A lot of RSI talk, in his view, skips the history of AI and Amdahl's law. Putting every kind of intelligence on one line, then placing humans and chimpanzees on it, looks dire and does not match the systems. He would not count a 10% speedup on AI research as RSI unless that speedup itself kept rising. He is open to evidence and treats the disagreement as philosophical more than technological.
Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·
From the transcript
But I think we're pretty far from stuff like that. Additionally, There's actually a bunch of nuance towards like, you know, ASI, RSI, and like whether or not the models have a sense of agency on their own. Like paperclip maximizer is actually very different than bad actor programs, super employee to destroy the world type things. And like, actually, I think that those end up becoming quite different in threat model.
I think that we're breakthroughs away from true agency. as much as people like to read these chains of thought and try to ascribe real agency to it, I would guess that we are something like two breakthroughs of gigantic magnitude away from this. I think even one breakthrough would not be sufficient, is my guess, but maybe one gigantic one could be.