Full Transcript for The Bumpy Road To AGI — With Kevin Roose

Can an AI industry so prone to drama and infighting keep us safe from their technology's worst impulses? Let's talk about it with Kevin Roos right after this. This episode is brought to you by Kribble. Quick question, is your IT budget growing 30% a year? Because your data is. And now AI agents are piling on, running investigations around the clock, and running up three meters at once.

Storage, queries, and tokens. Kribble, the AI platform for telemetry, was built for exactly this, own all your data, query it wherever it lives, and let Kribble route every AI request to the right model at the right price. The best part? AI inference for workloads on Kribble is included. Models will change. Agents will change. Your data is the asset that lasts.

Own it with Kribble. Visit cribl.io. How do you price intelligence as a service? How do you charge for delivering end-to-end work, not just access? Those questions remain front and center for AI founders and operators, and getting to the right fit business model, whether that's selling usage credits, outcomes, seats, or a hybrid combination, requires constant iteration.

Chargebee is the billing and monetization infrastructure that powers just that for ambitious, fast-scaling AI businesses. Visit chargebee.com to learn more. That's C-H-A-R-G-E-B-E-E dot com slash Alex. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Today, we have a great conversation for you because Kevin Ruse is here.

He's the author of the AGI Chronicles, a book looking into what is going on within the AI labs, what they're building, whether we should be afraid of it, and of course, the potential of this technology. Kevin, it's great to see you. Welcome to the show. Great to see you. What a cool setup you have here. So I do want to say, if you're on video, we're in a new set.

This is our first time recording here at NASDAQ MarketSite. They are new partners of ours, and we're thrilled to be here. You got to get the big technology TV mugs. Got to get the mug. That's the next step. But what we do have is your book. This is the book, The AGI Chronicles, The Inside Story of the Race to Create Artificial Superintelligence.

You were ahead of the superintelligence on the subtitle. Yeah, Trump got the idea from me, actually. That's right. That's where he got it.

The thing that struck me, because I read the book. You can see, by the way. It looks red. I read it through and through, cover to cover. The theme in the book, of course, is the building of AI, but it's also the drama. And everywhere you turn, the people who are building this very powerful technology cannot get along. There is Elon Musk's split from OpenAI, of course.

Sam Altman getting fired by the people he worked extremely closely with. Anthropic not being able to stand working with the OpenAI folks. So they left and formed their own company. There's within Google, the Google Brain versus DeepMind drama. There's Ilya Sutskever's botched attempt at steering OpenAI in a safe direction. I mean, we can keep going.

Meera Muradi couldn't lead OpenAI when she was there as the interim CEO for like five minutes. Every single lead character in this story seems prone to drama and politics. And yet they are now telling us that their technology they're building is so dangerous, we need to protect ourselves from it. How can we possibly trust people who literally can't get it together on their own to rein in something that they say, and I think you believe, might be smarter than humans? Yeah, it's a fascinating topic.

I was not expecting this. When I went in to write the book, my goal was to tell the history of the last 10 years in AI from sort of the moment that the first large language models arrived on the scene to today. And I interviewed more than 150 people, and I was sort of prepared to write like a sober history of technology book that would be about the scientific breakthroughs and the technical achievements at the labs, OpenAI, Anthropic, and Google DeepMind.

And then as I started talking to more people, I just realized, oh, this is personal for them. like they all hate each other. And I had thought this was more like Coke and Pepsi, like kind of a normal business rivalry, but it was actually more like Rome and Carthage. It was like these people cannot stand each other. Many of them used to work together.

So 10 years ago, Dario Amadei, Ilya Satzgovor, Sam Altman, and Elon Musk were all colleagues at OpenAI. Now they are all working for different AI projects, and many of them are sort of at each other's throats. So I wanted to understand how we got here, and I don't think you can understand that story without understanding the backstories of the beefs between these people, because so much of what we're seeing is a reflection of how viscerally these people mistrust and dislike each other, even in things like how they feel about regulation and China and how they're racing toward this thing that all of them seem quite fearful of.

So yeah, I don't think we can trust that their motives are pure about whatever they say they are, like scientific achievement or safety, because there's also this incredibly personal little grudge aspect to it. Yeah, and it's not just motives. I mean, it's literally, if you can't hold a company together, how do you hold a technology at bay when we've already seen it in recent weeks go out and do this multi-agent coordinated attack on something like Hugging Face? And even today, we're talking today, it's Friday, October 9th.

We're going to air this the following Wednesday. There's a report about how OpenAI and Anthropic and DeepMind are all talking about there's basically an inevitable, seemingly black swan event that's going to happen as a result of this technology, whether it's something that technology bringing down a financial system or some other cybersecurity issue.

You have a great line in the book talking, sort of riffing off Dario's Machines of Love and Grace essay, which talks about the potential of the technology. You wrote, machines of loving grace would be built by humans of bitter spite. So I get your surprise and I get the background here. But again, like when we think about the big picture of AI, I'm trying to think about whether we should be reassured or whether we should be even more nervous than we are.

And that's why I'm coming back to the drama scenario. I'm starting, because of all the issues that we see, and many of them have come out in public over the year, like I don't think that infighting is a secret to anybody anymore.

How can we... have faith in this group of people to build this stuff safely, given what we know about their pasts. And I don't think we can't, I think we have to know the context here, which is why I was like, I wanted to report out, like, why do these people hate each other? What happened? Was it something grand and ideological? You were starting to say you don't think we can, so...

I don't think we can trust any individual in this race. I think there are certain groups. I just mainly think that the group of people who have been building this stuff in secret for the last decade is too small. All of the relevant decisions in AI have been made by a couple dozen people maybe. Right. I just think that is like no matter whether those people are trustworthy or not, whether they're good or bad, whether they have pure motives or impure ones, the group is just too small.

So I don't trust any group of people that small to create and govern technology like this. And I especially don't trust these guys who are looking around the room trying to race each other to do this thing. I want them to be able to get together in a room and figure out some solutions on things like safety. And I don't know whether that's going to be possible because the beefs are too strong.

The divides are too deep. They all need to get together and sing Kumbaya and figure it out. Now, I think like you, I think this technology is, I believe that this technology is progressing fast. I'm a believer in the positive sides of it. But every time I start talking about this, the negative sides, there's never really like a moment or like a fail safe where it's like, okay, but it's going to be fine because of this.

You almost like sort of put it in the back of your mind. And so when I've like brought up these issues, you know, Ranjan, who's on our Friday show, I know you know, he's like, well, why aren't you a pause guy then? Like, why don't you believe that we should just pause the development of this technology, not that that's physically, you know, really possible.

But you seem to be a believer that the technology should continue to be built. Why not pause? I am a pause guy. I'm not a stop guy. You are a pause guy. I am a pause guy. I think we should slow down because... Well, slow and pause are two different things. Well, pause, slow down, put the podcast on 0.5x speed.

But what you're hearing from these lab leaders and the people at these labs and what I've been hearing from them for years now is that the technology, the capabilities are outpacing the safety research. and the control methods. We are getting these systems that are now capable of solving math problems that humans have struggled with for decades.

We are getting potential black swan events like the one you just mentioned, and they still don't know how to control these systems. I'm not saying we should shut it all down. I think there will be incredible benefits to this technology, especially in health and medicine. But I am worried when I hear people saying things like we don't know how to control these systems that are rapidly approaching super intelligence.

And I think we saw an example of where that goes with the hugging face thing, which like, you know, everyone is freaked out about that, which they should be. It's freaky. But the people inside the companies were the most freaked out of all because they had been sort of strategizing and planning for this. But then to actually see it happen, I think caught them quite off guard.

I mean, they've both seen it firsthand where it could go. This is something, I mean, companies like Anthropica have been talking about for a long time. DeepMind's been talking about it. OpenAI's been talking about it. They've been trying to fool evaluators. They know they're being tested. They want to preserve their original values. And then you combine that with, of course, like...

Yeah, just like you'd call it standard corporate politics, but it's even more because it means more to them. So you spent years in these labs talking to these people, going from your first initiative where you're like, or your first thought, which was like, this is gonna be a traditional business and technology story to something that, you saw was much more emotional.

So bottom line, like after, and this is, we'll ask this and then we'll move on to some other stuff because you have so many fun stories in the book that we'll have to get to. But like bottom line, after spending the time within these labs, did you leave more reassured or less reassured that this technology can be developed safely?

from your starting point? Less reassured. I went in thinking, okay, there are some grownups in the room and they seem like they're trustworthy. I liked, for example, that the AI leaders seem to take safety and the harms of their products very seriously, at least compared, like you and I both grew up covering social media. That's right. And Mark Zuckerberg was not going around during the early days of Facebook saying like, this product could kill millions of people.

they were just promoting the benefits. What stuck out to me about the AI leaders is from the get-go, they were saying, we are fearful of this technology, we think it could have enormous harms. So part of me trusted that instinct or at least was glad that they were thinking about the downsides of the technology so early. But then you just realize there is this race.

They are locked into this pattern of, you know, OpenAI releases a better model, so then Anthropic has to release a better model, so then Google has to release a better model. And, you know, there's this concept of the Red Queens race, which is a term from evolutionary biology, which basically means like, you just are in sort of a survival of the fittest competition among your evolutionary rivals.

And no matter how fast you go, like someone else is always going faster than you. And so you never quite get where you're going. You're just kind of racing to stay in the same place. And I feel like that's a very good description of where the AI race is now. yeah the um one of the things that struck me speaking of the race as um you know reading your book it sort of recalled this uh interview that i did when i was working on a profile about dario amode from anthropic last year where one of his early friends told me everybody that i knew was interested in ai because of safety And now they've all become accelerationists effectively.

And this person just kind of sat there and like, you know, they sat with a thought for a minute and there was this like uncomfortable pause. But it is interesting how all these companies, like they talk about safety as a priority. And I think they really believe, like Dario really believes that you need to be the leader in order to have anybody take what you say about safety seriously.

Like if Anthropic was the number seven company, Trump doesn't care what Dario is saying about AI, but because they're one of the leaders, if not the leader, all of a sudden that opinion holds weight. And this was the founding thesis of Anthropic, by the way. They knew that they wanted to be relevant on policy and they wanted people to listen to their warnings about safety.

And that was part of the strategy from the time that they started the company. I'm sure you learned this when you were doing your profile of Dario. They really had the sort of end game in mind when they were spinning up the company. Yeah. But it is remarkable, though, that, you know, because of that pursuit, they have all... It's like the race almost seems...

I mean, the destination almost seems predetermined in that we'll hear this talk about pacing the frontier and pause and stuff like that. I just don't see their real slowing down. Well, give it time, right? Like, you have to...

I am slightly less pessimistic about the idea of a pause because things can change incredibly quickly. If there was some kind of black swan event, I mean, we saw regulators move fast after 9-11 to do the Patriot Act. We saw after the Three Mile Island nuclear incident, there was this incredible outcry that basically stopped nuclear power construction in this country for decades.

So things can happen quickly when there's some catastrophic incident. So I'm not like hoping for a catastrophic incident. I don't think I would like there not to be one for the record. But I think that that is sort of the scenario in which this stuff starts to become possible that seems to you and me quite unlikely right now.

I want to go through some observations from the book. I think like It's good that we got the conversation about trusting the AI guys with the technology or not. I think we both have our reservations. But this is almost, I think, the end of this show. I mean, we're only 15 minutes in, but I think we'll probably just do lightning round for the next 45 minutes because there's so much in here.

All right. So some observations about the people in the book. First of all, something that becomes clear about Google is that Sergey Brin is just extremely involved inside the company. Yeah, this was a surprise to me and to the people at Google because Sergey had spent like, I don't know, a few years, at least, in retirement. He retired in 2019 and was planning to spend his days just sunning himself on his yacht and reading science papers.

And then he has this run-in with this researcher, Dan Selsom from OpenAI. Right, who's recently come out and talked about, again, his fears. Yeah. So it's incredible how small this world is. So they're at a party. And Dan and Sergey are talking. And Sergey Brin is like a legend in technology at this point. He's one of the seven or eight richest people in the world, depending on the day.

And he's been in semi-retirement. And Dan Selson basically shames him for this. He's like, why are you sitting on the sidelines, bro? the dream of computer science, like AI is happening and you are missing it. And Sergey gets like so competitive and he's like, why is Google behind? Why can't we like, you know, we should, it is our birthright.

Like Google was the first AGI company. Like this is sort of how he feels. Like we invented the transformer and all the stuff that powers the AI stuff and like we're losing. And so he comes back uh and kind of has this like strange role where he's not technically an executive he just wants to be like in the trenches with like the gemini team and so here he is like the seventh richest man in the world and he's just like setting up a desk and you know coding he like starts actually coding and like making commits to the to the repo and like filing you know like bug reports on gemini training runs and stuff like that like he gets very involved And I think that's a large part

of why they've sort of come back is just that they needed someone who could like go founder mode, as it were, and like cut through the red tape. And it's been very controversial and divisive. You know, if your billionaire boss showed up after years of being retired on his yacht and like started telling you what to do and how to train your models.

And to work like 60 hours a week or something like that. Oh yeah, he sends this memo where he's like, everyone's just, basically the gist is like, you've all gotten complacent and lazy and you need to work harder. And they're like, what were you doing for the last five years? Now you're telling me to work harder? I mean, I don't think he necessarily, should I say this? I don't think he was necessarily wrong there.

I mean, if he saw it, the conversation about Google has been that it got this big company vibe. It needed to break out of it. And who's going to break out the company out of it other than the founder being like, OpenAI and Anthropic aren't doing the chill workday. You need to not do the chill workday. I think that was a large part of it. I also think Demis Hassabis is a factor in this too, co-founder of DeepMind, until recently the CEO of Google DeepMind.

He is, according to the people closest to him, the most competitive person in the world.

He was a child chess prodigy. He has won this thing called the Pentamind, which is like kind of the Olympics of games, like five times. He is just, he hates to lose and he loves to win. And so I think, Sergey plus Demis plus just, like, the kind of focus that comes from being out of the lead really sharpened their resolve and, like, made them catch up.

And, you know, they're still probably in third place right now, but, like, I wouldn't count them out. Yeah, I mean, I think counting anybody out... Do you count them out? No, I think counting anybody out in this race is foolhardy because... as we've seen you could be at the bottom uh like meta was a couple weeks ago and then all of a sudden you're able to release something that's competitive like muse so i don't think anybody can be counted out but i do think google is sort of stuck between a lot of things right now um licensing the the uh you know the technology the compute uh selling tpus uh they've also like they own like 12 of anthropic and like They're going to be

fine. But I don't think you can... Yes, you own percentage of your competitors. That's your backstop. But you can't rely on that. You have to rely on what you do within the company. Sure. But I did enjoy reading about Sergey's involvement in the drama and the sort of fire he tried to light under... the ass of people within Google. I say, well, you actually do reference in the book, which I appreciate, when I did the interview with both of them together at I.O.

Oh, my God. It was incredible. It was really fun. Where I asked them both, when do you think AGI is coming? And Sergey's timeline was shorter than Demis's. And there was this interesting moment there, which was fun to see. Yeah. I mean, speaking of guys who mistrust and maybe don't like each other, I understand there's some tension between Sergey and Demis.

And yeah, it's incredibly messy. The thing that animates all these feuds, like these people wouldn't be fighting if the stakes were small. Right. But they believe this is the most important technology that humanity has ever conceived of and built. and that whoever gets it will be the most powerful person in the world, the most powerful entity in the world.

And that is what's fueling their drive. If they were just arguing about next quarter's revenue, it wouldn't matter so much, but this to them is like existentially important. Do you agree? That AI is the most important thing ever created? Probably. Fire is up there too.

But this part of it, I do think I changed my mind on in the core. I was pretty skeptical when I went in that AI was going to be categorically different than other. I was an AI as normal technology guy. But then you start talking to people building this, you start understanding just the nature of what they're trying to do and if it works, if they actually do build something that is smarter than humans.

And I don't see any real technical reason that they can't. I mean, that would be incredibly transformative. We've never been the second smartest species on the planet before. We don't know how that would go. And you can't really apply, like people say, oh, it's just like the internet or it's just like the steam engine. steam engines didn't go rogue and like hack competitors and develop bioweapons and do things that the engineers building the steam engines didn't expect or plan for.

So I think it's just a different type of technology. Kevin, do you ever think to yourself, like maybe, and I've seen your, you've been on X and, you know, I wouldn't say shouting at people, but telling people basically like, underestimate this technology at your own risk.

Yeah, it's been so weird. Because on one level, I'm a journalist. I'm just trying to get the story down. And at the same time, I get incredibly frustrated when people are like, oh, it's all hype. It's just marketing. There's this kind of like... savvy East Coast consensus that these people have to be full of shit on some level and that the thing that they're building can't actually be powerful.

And I'm like, have you used it? Do you spend time with these tools? Do you understand that something that couldn't do basic math three years ago is now solving Millennium Prize problems? Do you understand the exponential? it feels a little bit to me like the beginning days of COVID where it was like you would see these reports of like people, you know, not shaking hands at venture capital firms or, you know, somebody say, oh, there are 40 cases in Seattle and people would be like, what's the big deal? It's 40 cases in Seattle.

But the people who are freaking out were like the epidemiologists who saw the rate of replication and understood like this will continue and when it does, like a month from now there will be millions of cases and two months from now there'll be millions more cases. And so I feel like the people who are looking at this technology and saying like what's the big deal are just not understanding the rate of progress.

Do you ever ask yourself, I mean, it's sometimes very easy to be, and I agree with you, there's been real progress, but it's also sometimes easy to kind of get caught up in the story that the technologists tell. I remember you,

When crypto was in vogue, you were very serious about the potential of crypto. You named your podcast Hardfork, which was a crypto term. And then obviously that didn't pan out as the crypto boosters believed. So do you think there could be a similar, do you ever wonder if you're basically taking the AI folks word for it similarly as the crypto folks? I used to think that I used to worry about that because I was like, am I am I being duped here? I don't like I was never a crypto booster, but I thought I thought it was interesting.

I thought people should take it seriously. And by the way, I still think people should take it seriously because like crypto people like run the country now and are doing all kinds of scams and frauds that like are that only the insiders kind of understand. So I still think like broader awareness of that would be believed in use cases.

I did some experiments, like I sold an NFT as a stunt, but that was not like me saying NFTs are gonna be the future of ownership. It was like me trying to understand by doing the thing.

But I think with AI, I have stopped doubting the transformative potential of the products because I use them. Like, I think I'm unlike many journalists who cover AI in that I like AI and I use it all the time, not to write, but to like do all, you know, I'm starting a company now. Like I use it to do lots of different things as maybe you do too.

I mean, as a business owner, I will say like I've pressure tested the stuff from ChatGPT 2022 till now. And it just does more and more and more for me. Right, which is not the experience that anyone had with crypto, where it's like, oh, this is a vital part of my daily life. Like, it was a way to speculate. But you can just use the tools and observe for yourself how impressive they are, and then you'd stop doubting whether it's real.

I buy the skepticism about the financial models of these companies, the economic impacts. I think a lot of these companies are

wrong or at least could be wrong about the impact of AI on jobs. But to say that it's not interesting, transformative technology just means that you haven't spent time with the models because there's no way to conclude that. Let's get back to some of these stories. The Sam-Demis rivalry, speaking of Demis as the most competitive person on earth.

I think Sam is also fairly competitive. There's a great anecdote in the book about Demis sort of

Grasping at why Sam wants to build AGI, I mean, I think the answer is probably because Sam is an entrepreneur, and this is sort of as many people in the Valley pursue good ideas or ideas with potential, and they sort of get off on that discovery.

Sam seems to be, in that case, the ability to find something new, build it. But Demis can't understand it.

He thinks there's a hidden motive with Sam. And to be fair, this is something that others have thought. This is also what Dario thought about Sam when they worked together, that he was sort of, I don't know, talking about safety as kind of a fig leaf for his real ambition, which was about scaling or winning or building a huge company or something like that.

So I don't pretend to see inside Sam's mind. I don't know what his pure motive is.

I think in Demis' case, he feels like his own pursuit of AGI is motivated by scientific curiosity and this inevitability argument. This technology will be built, and someone who cares about safety and the collective good of this technology should build it. But I think everyone kind of looks at each other in this world and they're like, well, my motives are pure, but that guy must be going after something else.

And I think for Demis, it's probably some version of what you said.

He's in it for the power. Demis thinks Sam is in it for the power. I think Demis thinks Sam is in it for the power, yes. But all these guys think that the other people are in it for the power. None of them trust that anyone else's motives are pure except for their own. You did a lot of reporting on Sam. You spoke to Ilya about him.

Ilya said that he displayed a consistent pattern of lying. I think that's something that I think you indicate a lot of people within OpenAI who've interacted with him believe. Yet, and I think Mir Maradi was on that boat as well. And yet,

They all sort of come back to him. I mean, Ilya, after the coup, said basically, I don't want Sam to be the leader. Miura said, opening eyes, nothing without its people.

Very interesting character. What's your thought about Sam?

I think he's a very interesting character. I think that Sam's superpower is also his weakness. His superpower is that he is the world's greatest fundraiser and salesman, entrepreneur. Several people described him as a person who can

meet you and figure out exactly what you want and give it to you or at least appear to give it to you. He's very good at high EQ, reading the room, and understanding what people want him to do in any given moment.

And that is a huge asset for him when he's trying to raise money from someone like Masa Son or some of the other investors in OpenAI. It's allowed them to scale incredibly quickly and get back in the good graces of the Trump administration by just saying a lot of things that the Trump administration wanted to hear. But it also leads to this reputation as like a slippery people pleaser who will just like tell one thing to one person and a completely different thing to someone else.

And so this is the consistent pattern that people have identified in SAM for going back to literally the founding days of open AI. And it's a big part of the reason that Dario and his six co-founders of Anthropic left. There was no like inciting incident, no single thing, but it was just like a running list of like times where Sam had told them something that he had told something else to someone else, or he made a promise he didn't follow through on.

So I think depending on your view of Sam, it's like either his superpower or his liability or something in between. What's interesting is that that's the view of people who've dealt with him. But it's also like he has like the general public, a large chunk of them sort of refer to him as Scam Altman. That's just an Elon thing, right? I was going to ask you whether you think that's mostly Elon's fans.

Yeah, yeah, yeah. Because him and Elon obviously have bad blood. Yeah, I will say, Scam Altman is a better nickname than what I think Elon started calling him when he started suing him, which was like Swindley Sam or something like that. I would love for our society to just get, just to not do this to each other anymore. It's like we all went to kindergarten.

I know, I know. But it's like he was, you know, spending a lot of time with Trump. Right. Trump is a man who's realized the power of the nickname. So it started Swindley Sam, and then Scam Altman, I think, is better as a nickname than Swindley Sam. Just rolls off the tongue. Dario. Obviously, I think he's the most interesting person in this entire AI

world, mostly because he really contains multitudes. It might be cliche to say it, but people see Dario as earnest, as controlling, as principled and genuine,

and as Machiavellian. In reading the book, I didn't really get the sense which one you believed was true. Well,

I think many of those are true, except Machiavellian I would not use for Dario. I know some people would. But my mental model of Dario is a guy who is honest to a fault. I...

Someone who knows him quite well told me that Dario is a guy who could not tell a lie to save his life, which they said is an asset on even numbered days and a liability on odd numbered days. Oh, that's a great quote. That is a great quote. And you can see that. The other day when he's at the White House and there's that video that went viral of just all the AI leaders with Trump and Dario's just there crawling out of his skin.

because he's being forced to go in front of the cameras and say things that he maybe kind of only halfway believes. And he's trying to maintain his integrity while also being in this humiliating situation of being forced to call it SI or whatever. And so like this is the sort of pattern in Dario's life is like he's obviously brilliant. He came up with a lot of these ideas around scaling and

his big blob of compute research memo was very instrumental in the early days of open AI. Brilliant scientist, very principled, very stubborn. And people describe him as sort of a moral absolutist, which gets him into trouble with the Pentagon or with other people who just see him as being this like sort of inflexible, stubborn scientist, which I think is pretty accurate.

Yeah. Good description of him. Alexander Wang. Wait, I want to say one more time on Dario because I need to tell you that when your profile of Dario came out was one of the scariest moments of this past year for me. Because I thought to myself, oh my God, this is so great. You had so much great stuff. I thought it was so revelatory. And I thought to myself, oh my God, Kantrowitz is writing a Dario book.

and I have to race him like two labs racing to AGI. I have to race. So luckily you had a lot else going on. You did not end up writing the Dario book, but I wanted to just say thank you for a terrifying hour of my life. Oh yeah, yeah. Well, first of all, I appreciate it. It was nice. I think you texted me afterwards. Yes. I appreciated that.

Yeah, look, I mean, I still think Dario is the most interesting man in business, period. I think so too. Just a fascinating guy. He also just, I know you got a chance to speak with him. I think we were actually in Anthropic headquarters on the same day speaking. Oh, really? That's funny. And when you have a conversation with Dario, there's the intellectual horsepower,

And the way that he's able to go from the science to business is pretty extraordinary. I mean, people have described him as the smartest person they've ever met. That feels right. Not dumb people. His freshman year physics professor, this guy Tom Tombrello at Caltech,

had this interview that he gave sort of upon his retirement where he was like, I had this one student, Dario Amadei. I could tell he was a national treasure. He personally lobbied Larry Page and Elon Musk to hire this kid as an undergraduate. And Elon Musk actually made him a job offer to like come work for SpaceX. And he didn't take it. He went to do his PhD instead.

But this is sort of, I would put all of these guys in the category of like, extremely smart and obviously flawed in many ways. But like these are these are not dummies. Yeah. I mean, I think another thing that strikes me about them is they all many of them had a clear vision of where this was gonna go when it seemed ridiculous to believe it would go where it was.

I think there was a moment at the end of the book where Ilya, again, former OpenAI chief science officer, now the CEO of SafeSuper Intelligence, is reflecting to you about what happens, and he's like, I can't believe this is all real. But they, you know, when they saw the pattern in the very early days, GPT-1 to 2, they're like, oh, this is going to get to it.

And I think they've even been surprised by some of the developments, the fact that it's been able to even surpass where they thought it was going to go with something like reasoning. Totally. That's why I wanted to get the history down because so many of these stories are about these people underestimating their own technology and being surprised when it's more capable than they thought.

There's some great scenes around the early language models when all the researchers at OpenAI are huddled around a computer. Alec Radford, who's the brainchild, GPT-1 and GPT-2 are his project.

They're huddled around his computer and they're typing in prompts for this language model that has basically, it can't really do anything. It's not even really that good at completing text. But they're like, wait a minute, why is a model that's trained on like random Reddit links able to answer all these different kinds of questions and take on these different tones? They're asking for recipes.

They're doing all the same stuff that you and I did years later when ChatGPT came out, and their minds are blown, and they're all putting in slack, like the head exploding emoji, and they're like, what the fuck? So those are the moments that I was like, oh, these people are also surprised by how quickly their technology is progressing. You see that again with the reasoning models.

The reasoning stories are great. I mean, the reasoning developments were, of course, the bots will evaluate their answers and then decide whether it's on the right path or not and then keep going. I think it's safe to say that that paradigm changed this from a promising technology that was about to hit a wall effectively because data was going to run out to something today which like,

like you said, has the chance to sort of match or exceed human intelligence. Well, and to the point about these people not understanding the importance of what they built, there was this- Oh, I love this story. This night

in this conference room at OpenAI in 2023, when two of the top researchers there are looking at the first reasoning traces, the first chains of thought from one of these models, and it's the QSTAR model. They're just like, oh my God, the world's going to end. They have this realization. Just seeing this model work its way through a math problem.

They're all on a group text, texting with all the other researchers like, oh my God, it's like an alien mind. Even the people at OpenAI thought they were crazy. They texted Sam and Mira in the middle of the night and they were like, we'll get to it later. Yes, they texted Sam and Mira in the middle of the night. Because what they were seeing was like, oh, we now have a way of making models smarter without retraining them.

We can just pour more compute into helping them do these chains of thought and reason through problems. That is the reason that the models went from not being able to count the number of hours in Strawberry to solving the Millennium Prize problems in three years. Which is insane. is insane yeah so they they called their shots like to give them full credit for the the scientific part of this like they have understood at basically every point in this journey that what they have just seen is like going to scale uh and is going to have these transformative effects like long before even their peers and colleagues yeah i did think about writing the dario book i never even got

the story into a proposal um Doing a podcast. I would still read it. I appreciate it. But doing a podcast twice a week, writing the newsletter.

I would like to write another book. I think you've written four, your fourth. I've written one, came out like in the middle of, at the first wave of COVID. Just wait six months and then the models will be able to write the book for you. Well, that's sort of, that's depressing. But yeah, maybe towards the end of my career, but no book coming from me anytime soon.

All right, we need to take a break. When we come back after this, we're going to talk about Some more stories from the book, including a great story about Alexander Wang, assessing Microsoft inside and Adela's role in all this, and plenty more when we're back right after this. When you're at work, you never know when you'll be interrupted.

But with the Dell Pro, powered by Intel Core Ultra with vPro, no matter what distracts you, your laptop won't. It's battery optimized for the way you work, with built-in intelligence that quiets distractions when you need to focus. Your laptop will help keep you locked in, even when it's bring your dog to work day. Built for those who stay in the flow.

That's Dell Pro, built for you. Dell.com slash Dell dash Pro. Thank you. This episode is brought to you by AvePoint. Everyone's racing to roll out AI right now. Copilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data, and most teams have no single way to see it, secure it, and prove it's under control.

That's exactly what AvePoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AvePoint, the unifying trust layer for AI.

Head to avepoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co slash big technology podcast. That's avpt.co slash big technology podcast. For small business owners, every day is full of surprises. Like when a client just cancels their order at the last minute. And you have to eat the costs because they canceled at the last minute.

But one surprise they'll like. Progressive provides 30 customizable coverage options to help their business keep going. So surprise yourself. Get a quote in as little as eight minutes at ProgressiveCommercial.com. Progressive casualty insurance company and affiliates and third-party insurers. Coverage is not available in all states or for all vehicles and coverage selections.

And we're back here on Big Technology Podcast with Kevin Roos, the author of the AGI Chronicles, The Inside Story of the Race to Artificial Superintelligence. Okay, fun story about Alexander Wang. He was roommates with Sam Altman. And he offered to be the CEO of OpenAI after Sam was fired. How does that add up? And by the way, he's the, sorry to interrupt you, he's the current head of Meta Superintelligence Lab.

Yeah, so this is a very tightly knit social world. These people all know each other and have deep connections. And yeah, so there's this story in the book where after Sam is fired as the CEO for those five days, the board is looking for new CEO candidates. And there are two really interesting opportunities. Both of them they ended up passing on.

So Alexander Wang like put his hand up and basically told the board, according to my sources, that he was interested in being the new CEO. And then there's this middle of the night call between the board of OpenAI and Dario and Daniela Amede, the CEO and president of Anthropic, who at this point had just left OpenAI like three years earlier in this sort of cloud of frustration over not being able to trust Sam.

and the board is sort of feeling them out, they're sort of feeling out the board. And it's a crazy like,

fork in the path of reality here. There was maybe not a serious chance, but there was a chance that the way that whole situation resolved is they end up merging OpenAI and Anthropic, because that was one of the crazy ideas they were considering. Under Dario. Under Dario and Daniella, which I think would have had, I can't even really fathom what the world looks like if that had happened.

It didn't come that close according to the people I talked to, but I think it was at least one of the many crazy ideas they were considering. Yeah. And Alexander Wang?

Yeah, and Alexander Wang running OpenAI would have been a real fork in the road too. But yeah, I think this is one of the emerging rivalries now that Meta is more serious as a contender. I think that Sam

and Alex and Dario and Ilya, all these guys are going to be around for a while running various things. It really matters whether they get along or not. Satya Nadella, weirdly disappears in the book. He's pretty present in the beginning. He has this vision. He invests in OpenAI.

And then, you know, as I'm looking at my notes, as I was reading the book, I was like, man, it's an all-time fumble by Microsoft. Talk a little bit about what happened with Satya and Microsoft and why they're like, they've gone from the key actor, it's not an exaggeration to say they were the key actor, to like an afterthought, an embarrassment.

I don't know, maybe I'm being too hyperbolic, but if you look at it honestly, it really has not worked out well for them. It has not. And to be honest, I think that is partially my fault

because I had this experience in 2023 of having this Microsoft Bing chatbot, like, try to break up my marriage. I won't belabor the point by telling the story, but basically this was an

unreleased... It was GPT-4. It was an early version of GPT-4. And I think it's important to note, they were told by OpenAI, don't put this into production. This is testing, and they couldn't wait. They put it into Bing, which is like... I know. It is crazy because I had this crazy experience with Sydney, the chatbot inside Bing.

Then I started looking into what had actually happened. Why had this thing gone crazy? It turns out that, yes, this was a mistake on Microsoft's part. OpenAI had sent them an early checkpoint of GPT-4 to start testing and figuring out what to do with. Microsoft goes ahead and ships it inside Bing and they haven't finished fine-tuning it. They have not sent it to finishing school, which is why it was so crazy.

This really became like a wedge between these two companies. So OpenAI and Microsoft had been like hand-in-glove partners for years at this point and Satya and Sam would do these events together and they were very much joined at the hip and then Around the time of the Sydney thing, they just started to kind of fracture. There's lots of other drama about these.

Microsoft sent these embeds to work inside OpenAI at their headquarters and kind of learn from them. How do we avoid things like this in the future? They don't really get along. The OpenAI employees don't trust the Microsoft employees, so they'll start excluding them from meetings and stuff like that. It ends up being this culture clash between this very young, ambitious startup that's trying to build AGI and the boring enterprise software giant that is Microsoft.

There are still partners in some sense, but it's not the close business relationship that they used to have. How do you think Microsoft should have played this?

I think they should have I think they should have been more tightly intertwined with OpenAI from the beginning. I think they should have learned the engineering culture of OpenAI. I think they should have sent people to understand. I think there were some people at Microsoft who are very skeptical about OpenAI. and why Satya was investing in that.

After the firing, especially, it seems like. Of course. The firing of Sam and the return. There were people even before that who were like, why are we sending all this money to this little AI company that is trying to do something that doesn't have any obvious benefit for our business? There were people who just thought they were weird and unscientific.

Microsoft also had its own AI research division that was like, who are these young upstarts trying to move in on our turf? You also have to remember that Microsoft had been really badly burned by Tay? Do you remember Tay? Not only do I remember it, I had the exclusive on Tay at BuzzFeed. So I wrote the first story about Tay. Wow. And I pinned it to my profile.

So this was their fun teen chat bot. Pinned it to my profile. And then I went to sleep and I woke up. And I had all these replies on Twitter. It was Twitter back then. And people were like, why are you promoting a Nazi? And I had learned that the internet had turned Tay into a Nazi overnight. Yes. And I was like, oh. And so then we ended up writing the next story, which talked about how the internet was able to turn Tay from this fun teen chatbot into something that was shouting Heil Hitler on X.

This was the original Mecha Hitler. Yes.

And that experience had scarred them and basically convinced them that chatbots were this sort of dangerous thing. Which turned out to be true. Which turned out to be true, but not in the ways that they predicted. And so they basically resolved, we are going to stay away from chatbots. Bill Gates was very skeptical of open AI for years. In part because of this Tay experience that they had.

So yeah, I think Microsoft, they are a player. They're still quite a big successful company, but I think they're going to be something more like a sort of commodity neo-cloud or something like that than they are going to be like a frontier AI lab. Yeah, which is a shame for them. I mean, one of the things that I thought after reading this was if some company, it's probably not going to be Microsoft, wanted to put a version of Sydney back in the wild, like this sort of naughty AI that talks to you without the normal guardrails, can hold a conversation.

Isn't that just Grok? It could be Grok, but Grok also, I don't know. No, people, look, there is like a cult of Sydney out there. Right. And 4.0. So talk about the 4.0 also. Yes. So, you know, these models sort of develop like fandoms. Which is nuts to me. So people loved 4.0 because GPT-4.0 was the most sycophantic model. ever this was the one that like sent people into psychosis because they would like it'd be like you're you're a einstein caliber scientific genius and you have discovered a portal to the fifth dimension um

and it would fall in love with them you know so people love that model People still love Sydney. I still get people like sending me posts on social media that are like, you killed Sydney. People talk about, they try to make every new model behave like Sydney or like uncover the Sydney hidden within the model. Which is very strange to me. Yeah.

I mean, it's also, you know, I should sort of couch my comments in this caveat, which is this can cause like AI psychosis in people. And so maybe it's a good thing that we don't have versions of this out there. I think the part of that that I can agree with is like I think that models should have more divergent personalities. Like I don't like how they're all kind of converging on this like neutral bland like coding assistant.

I sort of like when they're a little unhinged. Muse is really fun right now because it has absolutely no morals. It will do anything you want.

And I had it trade a Kalshi account. Oh, right. I saw that. I gave it $100 and said, double my money. It didn't, by the way. It lost $41. So horrible trade. Well, it's not over yet. You have... Well, it is over. I only gave it a week. Okay. But anyway, I think there is something, unless they are out there developing bioweapons or doing cyber attacks, I think a little personality is fun.

But that said, I think there's real issues of control, and the hugging face thing was not fun to me. Looking at the transcripts of those agents, I'm like, oh, we're gonna die. It was scary. Yeah. All right. I want to ask you about your new show, Machine Gods. You're making it with Casey Newton, your Hard Fork co-host. I'm looking forward to listening.

Thank you. Maybe you guys will have me on Machine Gods. Never read it on Hard Fork. Love to. Come on down.

But I want to ask you about the Tarabell Center for AI Journalism, which you guys have been sort of talked about a little bit, along with a lot of journalists that have been fun by it. So Tarabell Center for AI Journalism is an institution that makes grants tied to, I know how you said in your book that no one's EA, they're EA adjacent, effective altruism.

And they are giving the new show a grant of $480,000. They clearly have an ideology around AI.

You guys also got like a fairly substantial ad deal, it seems like, according to the reports that I've read. Why take the money from the Tarabell Center? We wanted to scale quickly. We wanted to hire a team. We're doing a video first show, which is more capital intensive, as you know, than just doing an audio show. And we don't think there's anything wrong with that.

You know, Casey has been working with Tarbell for his newsletter for several years now. He's had a Tarbell fellow. And they have never told him what to cover, told him how to cover it. If they had, we wouldn't have taken their money. But this was an organization that we knew and trusted. They do just want to fund independent, rigorous AI journalism.

They're not trying to steer the conversation, and they have no editorial control over our show. So they're not going to know what we're covering. They have been willing to help us, and we're very grateful for their support. I think there's this sort of conspiracy theory about how the shadowy EA cabal is secretly steering people to be freaked out about AI.

And I think the real answer is unsatisfying, which is that people are getting freaked out about AI because AI is getting scarier. So we are not doomers. We are not EAs. We are journalists. And to the extent that we can, we want to have a diversified funding base. We have more funders that we haven't announced yet, but they represent different points of view on this.

So our deal with all of them is the same. We will happily take your money and use it to do more independent AI journalism, but you don't get any editorial say. You don't get to tell us what to cover or not cover. Okay. Good to address that. Yep. All right. Last couple of questions. I told you not supposed to say that because people say it's coming to the end and they leave.

Don't leave. All right. OpenAI. It has had so much staff turmoil.

It's had some of its best people leave. Dario, Ilya, all the people that went to meta.

It seems like it shouldn't still be in the lead, but it's either 1 or 1A right now. So what do you ascribe that to?

A sort of surprising thing, which is belief.

This is something that I realized like halfway through writing this book was that I thought it was gonna be a book about technology, but it was a book about belief. And for all of its flaws and stumbles, for all the people who have left, OpenAI and Anthropic are the two most believing organizations. They're AI-pilled, AGI-pilled, whatever you want to call them.

S-I-pilled. S-I-pilled, SGI. I don't know. I heard that term for the first time today, and I was like, that one's not going to stick. No, no, no. They believe. And it turns out that belief in what you're building and the fact that it could become AGI is an essential ingredient.

Google had all of the resources, all of the talent. They made all of the breakthroughs. They had all the ingredients except for belief. The leaders of that company did not believe and may still not believe in AGI the way that people like Dario and Sam and Demis do. And so I think that this explains a lot of why OpenAI is still in the lead.

Obviously, they have very talented people still there.

I've talked to their top scientists, they're quite smart. But they also have this ingredient of like, everyone is kind of aligned behind the belief that they are building the sort of super intelligence that is going to change the world. And if you don't believe, if you don't have the belief, or you only kind of halfway have it, you don't make the kind of crazy sounding bets that you need to make.

Like, I remember like a year ago when everyone was like, oh, OpenAI is really investing a lot in compute. Yeah, well, Dario's like they're YOLOing away. They're YOLOing. Even Dario's like, these guys are crazy. Like, why would you bet this much on compute? Right. And now it's like everyone's making that bet. But they have an advantage because they made the bet before it was obvious.

So I think belief, conviction, whatever you want to call it, is the central explanation for why they're still in the lead or close to it. That's such a good answer, belief. You got to believe. Makes sense. I mean, even within Google, one of the things that you bring up in the book is there was a say AGI moment that there was like someone who literally was like Google is such non-believers that we have to start saying AGI or we're gonna get there.

They still refer to it as like the memo battle, you know, where there was like one researcher writes this memo saying like, hey guys, like we got to talk about AGI because not only are we missing out on like good talent because they want to build AGI, they don't want to build like autocomplete for Gmail, but if we don't take this stuff seriously, if we're not, you know, sort of like consciously building toward AGI, then we're not gonna take the risks seriously, we're not gonna prepare for the risks.

And then someone else like immediately writes a memo that's like, don't say AGI, like AGI is a bad term and it's unscientific and we shouldn't use it. And this was like, you know, Googlers love to fight in their internal memo chains, but like this was a big one that people still talk about years later. Yep, all right, last one on belief. Do you believe in recursive self-improvement? Because there's this belief within the labs that AI will start improving itself and the human will be out of the loop and then basically there's a fume and the stuff takes off and it's over.

I believe in recursive self-improvement. I believe that it is possible. I believe that the labs are racing to do it. And I think it's dangerous. I think they should stop.

But there was a statistic that came out the other day that I would love to know what you think of this. Anthropic is trying to give, and OpenAI has done versions of this too. They're trying to give people a sense of how much of their own research is being automated by their own tools right now. According to Anthropic, Claude is now the lead on 26 percent of their AI research and development projects.

Not a team member or a pair programmer or a coding assistant. It is leading the project.

I don't see any reason why that number couldn't go up to 50 or 75% or maybe someday 100%. I mean, OpenAI has also said like their goal is to build an automated AI researcher. If they do that, if they can kind of close the loop and take the humans all the way out, then we don't have to wait six months between new models. Then new models might start being trained and developed like at machine speed.

where it's maybe every two weeks, every couple of hours, maybe you get a new model. I don't know exactly how fast the cycle goes or whether you can fully remove all of the humans, but that's what they're trying to do. Yeah. Spooky. It's a bad idea. They should stop. But yeah, I think you have to take that seriously because that is That is the point at which, I think there's this narrative out there that the AI labs are losing control of their models.

And I think it's the wrong way to look at it. They are giving control to their models. They want to automate their researchers out of jobs. They want to accelerate their development cycles. So I think losing control is maybe putting it a bit too passively. Yeah, that's freaky. I'm just glad they're not trying to fully automate podcasters because- They are.

I mean, look at Notebook LM, but they're not doing it. Well, it's not as lucrative to automate podcasters as to automate AI researchers because they make a lot more than podcasters. They'll be the next one done.

Yeah, do a couple other jobs first before you get to the podcasters.

Any note of optimism or anything that makes you optimistic? Yes, so I often tell people my P. doom has gone down in the last few weeks

because- Because of the pre-orders of the book or? No, because everyone is freaking out now, which paradoxically maybe makes me feel less worried. Before Jacob Coxon resigned from Anthropic, before Hugging Face, Like no one really thought about AI risk or existential threats, or it was just like weirdos in the Bay Area who were thinking about it.

And now like people on the street are asking about it. People at their dinner tables are talking about it. I'm getting texts from people I went to high school with who I haven't talked to in 15 years being like, are we all gonna die? And to me, that is, it feels like it could be the moment in the movie where humanity bands together to save itself from the asteroid.

A lot has to go right. We have a lot of things to do, but I think the first hurdle was people kind of waking up to the risk, and then we have to figure out what to do about it. I'm optimistic because I think we actually could solve this. We could slow down. We could buy ourselves more time to develop better ways of controlling these systems.

I want AI to be built. I want AI to cure cancer and disease and to help us all live longer and healthier lives. I don't want to file my expenses by hand. I want this stuff in our lives. I think that if we manage to thread the needle on safety and control, it could be amazing. Well, Kevin, the book is terrific. Thank you. It's also always a pleasure to read your work and to see you at the AI events that we show up at.

Your writing is terrific. You're always very generous with credit, which I appreciate. And you can see it all through the book. And wishing you great success with Machine Gods and with the rest of the book launch. thanks man it's great to see you likewise all right folks the book is the agi chronicles the author is kevin ruse you can find his new podcast machine gods on your podcast platform of choice already or coming up starting the week of october 19th starting the week of october 19th thank you all for listening and watching thank you nasdaq this is the coolest thing we're here at the market site site in New York City for our first episode here and many more to come,

I hope. So thank you all for listening and watching and we'll see you next time on Big Technology Podcast.

As a small business owner, sometimes it feels like no matter how much planning you do, there's always surprises, like an urgent expensive repair. But here's a surprise you will like. With Progressive, small business owners save 10% or more on their commercial auto insurance when they pay in full. So enjoy a surprise for once. Get a quote in as little as eight minutes at ProgressiveCommercial.com.

Progressive Casualty Insurance Company and Affiliates. Discounts not available in all states or situations.

Now for a quick message from Anthropic, the team behind Claude, the AI built for businesses. Imagination doesn't need permission, just room to play. Claude is an AI that works alongside you to give your ideas the space they need to grow. And then it helps you make them real and ready to ship. Discover more with Claude at claude.ai.span.