Full Transcript for Claude now leads 26% of Anthropic’s R&D, Meta drops an 816B open model & Gemini 4 Argon

AI engineers are sleeping on the personal agents renaissance explosions happening in the real world. It's kind of crazy. Americans say that they much prefer living near a nuclear power plant than a data center, which is quite wild. Antropic is also going to do an explosive IPO. They reported an insane quarter, $16 .8 billion, about 45% from the second quarter increase.

Alex Volkov, We've been doing optimizations for Gemma. We have been able to achieve over 100 ,000 tokens per second with the E2B, so our smallest model. Antricopathy says, 6 billion people have barely touched an LLM. Antricopathy says, I'm very pro -Rumor, but you know, I'm also a psycho, so you don't have to be. Welcome to Thursday, October 15.

I'm live from New York. My name is Alex Volkov on AI Avengers with CoreWeave. And with me, Wolf from Rainbow Wolf, also AI Evangelist, also from CoreWeave. And Peter Gostev, evals and model behavior at ARENA. And I just spent three days at AI Engineer here in New York. I can't wait to tell you guys all about this. And while I was there, it was an incredible week just blowing up.

So we had to try and catch up. Meta ship Muse Wonder, which was rumored by them as this big watermelon model. And it smells big. It definitely smells like a big model. And you can use it right now in Muse, and we're going to talk about this. It's number two on ARENA, which puts Meta Square in the Frontier Lab race. As we told you before multiple times, do not, under any circumstances, do not bet against Zuck.

We told you. We told you this two years ago. We told you this after Lama 4 dropped. It was a disaster. And we told you this when he started hiring people with soup. And so now we're going to feel it. Peter, I think you tested it out. I think you have some thoughts. I would love to hear from you. Yeah, maybe to differentiate, we do have a few different leaderboards.

And they are second on the taxed arena. And they're actually 12th on the agent arena. Interesting. Yeah. And it's kind of interesting to interpret. I think we do have this kind of divergence now going on. So for example, Argon from Google as well did really well on taxed. So it's actually first. And you can roughly interpret it as how much people like to chat to the model.

So it's interesting that there is a divergence between what people want, who people want to talk to versus agent capabilities and coding and all of that. And it looks like it's different. So yeah, Argon is fourth on agent arena and Muse Wonder Max. So it's not the full version. It's on, it's ranked 12th. Very interesting. We also have to talk about the whole thing where I saw the timeline react negatively because Muse Wonder wasn't able to recreate the 3D scenes.

And people were like, Oh no, this is a bad model because it can't compete with Astro on like the 3GS thing. We have to address this because I don't know if my assistant every day needs 3GS. So I don't know if this is like personal thing that I need my assistant to do. Okay. Uh, what else? Okay. So later today, Omar Sanseviro from Google DeepMind joins us to talk about it, about Argon specifically.

We also have thanks to Peter. We have Nathan Denyash from the state of AI report joining us in an hour to talk about the state of AI. Uh, they've been running this for a decade, Peter. Is that correct? Is it like a crazy thing? I'm really excited to hear from him. Yeah. Yeah. No, he's a, he's a great guy. Yeah. He's a, he's been doing this for ages.

Yeah. I actually haven't checked how long, but yeah, I've been reading it for many years. I think it's, I think it's a decade, but Slopus prepared the news while I was running at the engineer. So hopefully Slopus did a good job. Uh, good job Slopus. Um, and then what else? Oh, Antropic had an investor day. So before we only had rumors and leaks, whatever.

Antropic is about to go IPO. We have some numbers and a specific thing for kind of the only sponsor for this podcast core. We've with our agent lens product, uh, Antropic cloud code specifically now supports open telemetry. So you literally can start running cloud code and then see your results in, um, in forge agent lens. Which is super cool.

I think I've seen some examples already. Uh, so we're definitely going to talk about this at least a little bit. And then, uh, so after Nathan Benyash, we'll talk about state of the art, but I think since I was a little bit of out of the loop, I do have a whole prep thing here. Would love to hear from you guys about the highlight of the week.

I think we'll start with Peter. Well, my highlight was, uh, I was at, uh, NVIDIA event yesterday. Oh, you met Jensen, right? Yeah. Yeah. Tell us. Tell us. Yeah. He was lashing on the show. Yeah. They had a kind of, it was very confusing afternoon. During the day, during the afternoon, they were building, they had this kind of builders event.

So it was all like people you meet at AI engineer conference kind of thing. So that was really fun. Um, and yeah, met a few people from Twitter, however, before they had like, uh, it was an office in London of a company that does like humanoid robots, uh, which I had no idea. Like it was central London, you know, so that was awesome. Um, yeah.

And they were kind of, it was all about open claw kind of thing. So they were kind of showing how they also launched the laptop, right? The, the one with the kind of spark, um, or, or other, did it start shipping or something? I think they announced that a few months ago. Um, so yeah, I think a few cool things like that. Yeah. And then in the evening, it was very different.

It was in a fancy, uh, kind of, uh, reception lounge, uh, of, uh, anyway, there's a weird converted church building in London and those AI minister and all of the fancy people from, I don't know, universities and industry and a very different, very different vibe. That's so cool, man. It's kind of like besides the AI engineer loan that we both went to and we recorded from this is the first time that you're saying something about like somewhere else outside of the States that gets me FOMO.

I'm like, Oh, that sounds super cool. I wish I would have been there. Uh, that's your highlight. Obviously, uh, the folks are welcome to check out Peter's timeline for this. Uh, you posted, uh, very well from there. Um, and you can tell us more about this and then Wolfram, what's your highlight? What's your one highlight from this week? Okay.

There's so much stuff, but if I have to pick one, just one that I'm currently most excited about, it's an open weights release, uh, of fish audio, fish audio drama three. Yeah. That is currently I mean, there's so much I could name multiple things, but since we're going to cover Gemini for Argon and you are our Google guy with the pixel phone and whatever, and you didn't choose Argon.

I think that I'm very interested to hear what you think about Argon since this wasn't your highlight. Uh, I can say something about, okay. Yeah, we'll definitely, we'll definitely get to that. Um, highlight of the week for me. Um, okay. This is a highlight and a hot take. I've spent the last three days among the top AI engineers in New York and also this engineer conference was adjacent

to obviously wall street. So this was a finance and the engineers together. So there's a lot of finance bros, a lot of big trading companies, a lot of banks. The highlight for me is I realized something that I can kind of Wolfram and I talked about on the show, Peter, maybe with you as well. Um, AI engineers are sleeping on the personal agents, Renaissance explosions happening in the real world.

It's kind of crazy. Uh, so folks who are listening to Thursday, I, I know you guys are engineers. Hopefully we do a good job to tell you about the stuff that's happening outside of our bubble as well. It is kind of crazy. It feels to me that because everybody's building their own harness, or they're so used to cloud code doing stuff for them.

They're trying to force this behavior into tools that don't need to do this tools. Meanwhile, the world is changing for normies because of Grog, Bot and Muse and to some extent bots or dots to some extent dots. Uh, and, and this is like the summary for me because I'm like a very personal assistant pilled or personal agent pilled. And I see that I talked to folks and they don't have one running around 24 seven doing stuff for them.

So this is my highlight of this, not the AI news, but this is news to me that even the engineers that we talk to every week for the past three years, that love to stay at the forefront of what's happening are missing a huge, huge thing that's happening. So hopefully we're here to tell you about this huge new thing and we're here to make sure that you guys don't miss it because I think it is huge.

Because most of the normies that I talked to did hear about some of these personal agents. Wolfram, go ahead. What really excited me this week. It's another thing. It's not an AI release or something, but it is something someone I very much respect said because Linus Torvalds really said that not using AI to create code is like, it's just a tool.

So if you don't use it and you are a smart person, you should be using it. He said, if anyone is submitting as an example, it's, it's as crazy to say you can't use AI for doing something. It's like saying you can't use a compiler to create code because that is also a machine tool that you are using and you are not using assembly, assembler to create the code.

And that was an example. And it's great to see the mindset change. I mean, he already, already had an open mindset and he says, as long as you understand the code, I don't care how you create it. That is the important thing. And I think more people should be thinking like that. It doesn't matter what tool has been used, but did you, what you ever, whatever you created, do you understand it and does it work? That is the important thing, not how you create it.

Do you understand the part is interesting. We should definitely talk about this because I definitely slopped. I turned my slop cannon towards some projects that Peter, I think you can resonate. I have no idea what's going on under the hood, as long as I see it and it works and it's dope. But I do agree. There's some stuff, especially in critical infrastructure.

So the folks that open the eye, let me scratch that the folks that they engineer from banks, for example, there's a lot of stuff that needs to be human readable. There's a lot of stuff that they need to understand everything. Somebody taught me the concept of toxic access where the loan book, the master loan book record or whatever, something inside the bank that all the money lives in for everybody.

If there's people with access, they can change some stuff that's like toxic access or something like that. So that obviously people need to understand what's going on there and be accountable as well. Let me run to TLDR, which is a section of Thursday, where we tell you briefly about the most interesting, exciting parts of what happened, curated to our interests.

I think I'll stop saying this soon, but we used to be able to cover all the news. I no longer with fair conscience can say this is all, everything that happened in the world of AI. But here is the most important thing that happened in the world of AI. Let's go.

All right. Here's everything on Thursday that we're covering on October 15th in about five minutes. Let's get into it. So we're live from New York after AI engineer, where I think the highlight was Capital One Max Canut Alexander made the best case I've heard when code quality still matters. This is probably the MVP talk, and I would recommend folks to tune into the live show and wait for the release.

Swix also came on stage and he's on conference and talked about how to kill your SaaS hackathon. There was a hackathon this weekend and 72 clones of big sasses spent 100 billion tokens over this weekend to build the clones. And then Anthropic shows dynamic workflows as well. Let's see what else we have with the TLDR.

In big companies and APIs, Meta's first big and heavy and frontier -ish model, or maybe frontier model. No, it's actually a frontier model. Meta Muse Wonder is number two on Arena, and this is the best version. And it is now available in different tools. So it's available in Metacode, and Muse only for subscribers. Some builders are split about this.

Also, there's two versions there. There's Muse Wonder and Muse Wonderful.

Did you see what I did there? Muse Wonder and Muse Wonderful? That's, I think, what they did. Because everybody calls their models Max on the Max reasoning. They call it Full, and it's wonderful, and it's pretty cool. Gemini 4 finally launched. Welcome, Gemini 4. Again, as a reminder, we'll have the folks from Gemini, Omar Sansevro and Genevieve joining us in about an hour to talk about Gemini 4 Argon.

It's out for everyone. It's number one on Arena and Visual Evals. And we can't wait to talk to these folks and hear all about the use cases about Gemini. And then we talked about Entropic opening, open telemetry traces and talking about their IPO. And Agent SDK still runs on your cloud subscription. Wolfram, hopefully you caught that. There was an update and now there is an Agent SDK.

And if you guys remember the whole, maybe Entropic will block you from the subscription service if you use it for like Hermes Agent or whatever. Now we're back at the place where Agent SDK supposedly runs on your cloud subscription. Entropic also showed investors a 16 .8 billion dollar quarter ahead of its reported November IPO. And there's a rumor about Fable 5 .5 showing up in Azure, but it's unconfirmed.

Just a rumor. Folks from TypeSafe, this was just this morning. This is like nearly breaking news. I think it's in the breaking news category. Jav 1 .14, which can now see images. So image Jav, we heard that it's coming and image Jav is now here. So it's now in preview.

And I think it's one, I really want to try it. I didn't have time running around, but like image Jav is something I waited for a long time. Microsoft shipped its infant ship, Maya 300 and its own decision model, Microsoft decision one. So decision models are taking over. And I'm very, very interested to see how decision one stacks up with image Jav.

Open source LLM in open source LLM for open decision models was released almost one from 0 .5 B to 27 B. They also answer with bounding boxes. So it's not only Jav. Jav answers with bounding boxes. So do these models.

I haven't, I haven't had the time to test them. MeCPM returns with 4 .7 and runs a 35 billion vision model on iPhone in about three gigs. Liquid shout out to our friends from Liquid AI. There is the biggest liquid foundation model, 24 billion parameters built for local agents. And we constantly talk about running agents locally. And then Quinn Geyer, RL framework for long horizon agents came out.

I think it's a paper. In this week's buzz, we told you about Forge, Corvi Forge, the rebrand for our Agente Cloud offerings on SaaS. Forge model distillation is now in preview. And I think that's super cool. And also Agent Lens in preview. And Agent Lens is just hot this week because now you can connect cloud via open telemetry and see the most, you know, the most useful and not useful traces from your cloud run directly in Agent Lens.

I think that's super cool. Apparently, XAI dropped Grok Voice 3. Do you guys here see this speech to speech leaderboard? It tops the speech to speech leaderboards. And apparently, all of the Starlink support calls are now used by Grok Voice 3. And Wolfram, you also mentioned Phish Audio, correct? Drama Voice. Can you just summarize something for us? Yeah, it's a new model, a speech model that you can direct, where you can give instructions for how the voice should be sounding.

I think this is best if we, when we talk about it, we show some examples. Yeah, we may replace it. It's open source as well, so you can run it locally. And it's also at the top of what is possible with voice right now. Oh, that's super cool.

All right. Let's go to industry. Industries, where we talk about stuff that are not releases, but I think it's important from this week. Andre Karpathy says, six billion people have barely touched NLLM. He did some back of the napkin math. And don't we love a good like Karpathy tweet that says the industry ablaze? And it feels like he's been tweeting less since he's joined on Tropic.

So he literally had this like funnel of saying six billion people barely touched NLLM and AI. And the gap, he says, is getting wider. And then he kind of does calculate. So hopefully everybody who tunes into Thursday AI folks, you're on the top of Andre Karpathy's funnel. But he does have this thing where like only a handful of people touched the models at the Frontier Lab.

And last week, State of the AI report came out. Nathan Benyash, the author, the guy who's been running this for a while, is going to join us. It was at 10 a .m. Pacific, which is going to be in an hour to talk about it. And I'm excited. Peter, you connected us to Nathan. I'm very, very excited. He's been running this State of the AI report for a while now.

Alex, I think, did you mention Nemetron? I don't think so, right? No. It just launched about half an hour ago. Ooh. No, half an hour ago. Yeah. Nemetron 3 .5 Super VL, 120 billion total parameters and 12 billion active parameters. No, I do not. It looks like they're focusing on more. It's multimodal. So the idea is that it's good at video understanding, image understanding, and long running agents.

And roughly where they're kind of pitching it at DeepSeq, V4 flash, including small, that kind of level. But that's meant to be quite fast. And they say it's performance at that sort of level. So, yeah. Just in your rates. We haven't tested it. It just literally came out. Yeah. Yeah. Yeah. Perfect. Shout out to our friends from Nemetron, Tim Green, and our friend Chris Alexi, who works on that team and always joins us and talks about it.

So shout out to NVIDIA. There's also rumors that NVIDIA, you guys see this, NVIDIA is in talks to buy Reflection. I think it was Reflection. Well, the lab behind Beam that we talked about, Beam last week. I'm pretty sure it's Reflection. There's rumors that Jensen's collecting the infinity stones of open source. So hug and face, Reflection also rumored, also friends of the pod.

The Israeli company rumored that NVIDIA is like talking about purchasing them as well. All right. I think it's time for us to go to Frontier Labs and talk about Metamuse Wonder.

All righty, folks. Let's talk about... Do not bet against Mark Zuckerberg. And now Alex Wang. So in addition to having great meme taste, Meta is now back at the Frontier. Meta's Frontier model is number two on Arena and just 12 points behind Argon, which we're going to talk about. But some builders are calling these benchmarks. And I think several things can be true.

Meta Super Intelligence Labs released News Wonder and News Wonder Full. Wonderful is kind of the over the top model that we don't get in the API. It only exists in meta subscription tiers. Alex Wang called it their largest and most capable model yet. I joked with somebody yesterday that all I need to say every week on the show is that this new model launched, numbers are bigger and they claim it's their best yet.

And it does... We're kind of there, right? We're kind of there where it's like the new iPhone 18 moment for every lab and this is the best iPhone we've released yet. Both Meta models, though, they do smell like a big model to me. So both and one million token, um, News Wonder via API, it's four bucks in and $20 out per million tokens, basically Opus 5 .5 pricing.

But if you see the emails charts, they actually show that they beat Opus on multiple interesting benchmarks. And those benchmarks are not your standard benchmarks. Besides the SWE bench one, I'm pretty sure that those benchmarks are a little bit different. Um, News Wonder in parentheses full is the better model. That's the one that kind of they say beat Opus and beat Astra and beat Fable only for News Code subscribers and, uh, 50 or $200 a month.

This is the tier that you need to pay either $50 a month or $200 a month to get it on News Code. No open waits for it either. But here's the news that excites us. Uh, Muse Spark, the model that was there before News Wonder, they told us about open source and Muse, Muse Spark is basically the new llama. They're going to not yet, but they're going to open source Muse Spark.

Uh, I hope this is the latest version 1 .3, but I'm very, very excited. Uh, Meta says the Meta Wonder models are too big and maybe even too risky for open source. That's an interesting call out. Um, right. Let's just dive in there. Uh, Peter, I know you guys tested News Wonder. Um, would love to hear from you. Yeah. So you, you're right to say that it was second on the text arena, which means that people like talking to it.

It is, I think, 12th on the agent arena and agent arena is when all people like you and I go on arena and use it in various kinds of ways, which could include coding, could include white color work. So that doesn't, that's not amazing. I must say 12th is not, it's not that great. Um, which is kind of interesting. Like people say, or is it benchmarks or, uh, like Alex Wang is, uh, fighting back and saying, oh, you know, who needs 3d and all of that.

But you know, 3d aside, I have tried also building websites with it and I must say it was kind of awful at websites. I don't know. Maybe it was my experience. Maybe I did something wrong and I didn't spend like many hours with it, but I think that's kind of an important as well. I've never seen a model that's really bad, but it's good at 3d and it's good at websites, but it's fundamentally bad.

Like I think, I think it's kind of unreasonable to say, well, it's much as opposed, but actually it's really bad at half of the things that people use it for. So it's kind of, I don't know. I don't want to be unfair to them. It's clearly a better model, but equally, I think it's still behind on some things like front end, for sure. 3d aside front end, it seems like really quite bad actually.

This is different here. I will say while 3d is not a use case that I expect my assistants to kind of use, and it's obvious that Mera is fine tuning Muse Wonder for the music system, right? Like it's very obvious to us why they have the model and the way they're going to put this capability in people's hands is via their personal agent or AI assistant, whatever you want to call this category.

And so they're optimizing for that thing. We should probably mention the two innovation type things that they're adding there. They're not evaluated, which is dynamic memory and dynamic contact stuff. Um, however, Peter, to your point, while asking my assistant to create 3d castles and 3d series, or whatever Nissan created and they showed us last week, a virtual copy of Toronto with a hundred thousand agents running, that's not what I'm going to ask my assistant to do.

Building websites is kind of what I think many people would like their agents to help them with building websites, one -off websites for like a use case or like more, more stuff as well. So I think that this is definitely in the realm of something that people will ask their assistant to do. Wolfram, do you have any thoughts on Mera Wander, Mera Muse Wonder? Oh, on one hand, that is exactly this, what I've seen less technical people who are not as much into AI.

They use AI assistants to create websites or update their websites. So that is very important to be good in this regard. 3d on the other hand, of course, it's a nice demo, but the practical use case for us most of the time is not that high. I tested Wanda and it is excellent in German, which is not true with all the models. So it has the colloquialisms and stuff.

So this is great. Um, yeah, it's, it's a good model. And I think that they have their own agents, uh, that they are offering personal agents means that it can also be used very well in Hermes agent and the other ones. It's available with the API. Yeah. Um, yeah, I think it's great to have more choice here. Yeah. I think a very interesting thing is I haven't seen the Muse code subscription being used in like Hermes agent.

I don't know if that's possible, but I think that'd be cool because now we're going to get there, but cloud SDK looks to be back up. Um, what's always possible is to have your Hermes agent just use another agent. I'm using all the agents like that, that Hermes on my own system is my main and connecting to crockmod and the other ones and just taking control of them all.

Yeah. How did you set it up? Yeah. We had a comment as well about Muse. Yeah. Uh, that's a good idea. Actually. I never thought about trying it as well. How do you, if we have time, how do you, how do you set this? Like what do you use it for? Oh, I've been using this for a long time, even before the agent stuff came up. I was calling it minions.

Amy has her minions and she uses computer use. If there's no API or CLI tool, it's just a skill, basically a minion skill where she can use. We did it in the, in the, when, when the first computer use stuff came out, she was going to the chat with the website and using that as a minion from Claude and stuff like that. And it still works the same way.

You just give it a tool like it can use a CLI tool or it can use a website or crockbot. The app can be controlled with computer use. And it's just another agent for a sub agent for the main agent. So very easy to set up. Okay. I do it all the time in the terminal. Claude calling codex, codex calling cloud. Yeah. The variety gives, gives you a lot for a hundred percent.

I'm just wondering because these are, these are different systems. They're extremely powerful, connected to many different services. So just, I don't know, just the use case, like what are you using it for? I mean, I think it's interesting. Just one use case is a council like GBT pro, which you can't just use with the API and so on. So we are using it that way.

If we are creating a concept for something, it goes through all these. I have a whole council of 12, 13 agents that all work together to make sure that this is, if it's very important to be the best one we can make. Cool. We definitely should talk about use cases though. So we talk about models and numbers, et cetera. But I think as we started to say in the beginning, we're approaching the level of capability where like for most of us, the stuff that we need to do, we can probably already do.

So go ahead. One thing to add as well, I think about the release. And I imagine they probably just don't have enough compute, but it sounds like at least from like, you know, I think they don't. Yeah. Yeah. I don't think so. Because why, why is the full one not available via API? Like, I don't think they're just being stingy, right? Maybe they want to cross sell the subscriptions, but that feels, I don't think that's working that well.

So for example, I think Theo, right? The, the YouTuber, he, uh, he was very complimentary about the muse model, which I was confused by. And then he came out and said, well, actually the, the full one is great. The regular one that we got, he thought was bad. So I, I haven't tested the full one. So my impression is very like meh about the model, but maybe the full one is amazing, but we don't have access to it.

So I, I, I wonder if there's also that kind of mismatch that they think all their model is amazing. Then the reception is like, or is it benchmarks or what, but actually maybe it is amazing, but they, they mean a different model. It's not ideal. This is kind of on the same topic of folks, let's say inside the topic. They're probably not using Opus.

They're probably using like the Opus plus plus whatever model. And so when they talk about the stuff that works for them in the public, I'm not sure that they mean kind of the models that are available to us. So that's eventually very interesting point. And I think Theo also gets early access to like the models that are not available in the API.

Um, you can get muse wonder, not wonderful news wonder within the product. So I got to wonder if any of you guys used it within the product itself. Because I switched immediately. So I, I've been using muse as you guys know, because I've talked about this a lot and I will continue talking about this because I think muse is wonderful. Not the model.

I, when I realized that I can only get access to this new intelligence, essentially what happens is well from, you know all about this, you've been doing this for Amy for a while. Essentially I realized that I can upgrade my assistance intelligence with a click. And that is incredible to me. That is just like, um, this is the scene from the matrix where Neo gets a thing and now he's like, boom, and I know Kung Fu.

So skills is kind of like that, but also like upgraded intelligence kind of like that. And so I upgraded muse, Wolf Fred, my muse to wonder. And they gave it a lot of tasks. Here's use cases time. Let's talk about use cases a little bit. Uh, the podcast studio that I'm in, shout out Virgo and Ezra for helping set this up super quick. Muse found this out.

Muse emailed Ezra a couple of times, including the notes and the requirements I need for the podcast on the way here. Muse got me coffee. I did a meetup yesterday, liquid cooling IRL, AI engineers in sauna. And you guys can see this wristband muse designed and ordered it and shipped to the hotel. And all of this was a news wonder. I was just kept like shoving tasks to it and it was executing them mostly perfectly.

The failure modes of this model, Peter is, it's real hard to benchmark. So we need to talk about how we benchmark those like use cases, but the failure modes of this model were not the logic parts. The failure modes of this model was the website doesn't work. The website is blocking us because cloud site doesn't like agents or the stripe link doesn't quite connect or something.

So the failure were real world failures, including a very funny ones where muse called some businesses and business hung up because they don't want to talk to a personal assistant. It was really funny. So I think, again, I think we're approaching a level of capability where most people do not care. Um, and as long as it does stuff for them, and then the question is, does it do stuff for them? So I think when Meta released the models, they had some evals in there.

I think we should pull them up that are not the standard ones. There are some like personal assistant ones. And Peter, the question back to you is you are saying on the agent arena, Muse wonder is kind of suffering. I got to wonder if my experience with like the personal agent stuff versus the agent arena, where are those tasks falling? Where's the daylight between those tasks? Yeah, it's, it's definitely hard to say.

And I think yeah, it could, the kind of failures that you're talking about, they do sound more like the harness in the, in the broader sense, the connections, integrations, like things like that. So yeah, it's definitely, it's definitely hard to say. One thing to say though, about the, the benchmarks, and you mentioned the different ones.

Um, this is kind of a pattern, which is a funny one. I did a bit of analysis on this, maybe I should, I should publish it there, where I can pull together the different benchmarks that labs were using over the last year, just all of the releases. And the half life of a benchmark that they quote is, is basically like one release, where so many benchmarks just get quoted once, because they get a good number, and then I never seen again.

So I wonder, I'm not accusing them of this, in this case, maybe they're meaningful benchmarks, I don't know. But that's also something to look out for is that quite often, I would open the release, I'm looking at it, I don't recognize even the titles of half of the benchmarks. Yeah, and you know, I'm in the industry. So I don't really know how, how reliable, how are they going to stick to those benchmarks or not.

But if they're inventing new ones, or kind of citing new ones, because it has a good score, I think that that's also not necessarily indicative of the performance. That's true. And I think to some extent, this makes sense, because capability, let me try to find the exact thing you're talking about. But I think the capability moves, moves and different benchmarks need to switch as well.

Right? Like we've seen that OpenAI decides not to do Sweepbench anymore because Sweepbench was like leaked in full and then Sweepbench Verified came out of Sweepbench Pro and like a bunch of like updates to this paradigm. But I definitely want to pull up some of them so we can talk about this meta.

Wander. Nistan, meanwhile, any comments on Muse Wander from your end?

No, is Wander a new thing? I have Muse running in circles all night. I don't know what it does. It tests websites and stuff. It's pretty dumb sometimes, but it's just so energetic. Listen, you don't know what Muse Wander is? No idea. I love how co -hosts of Thursday, I sometimes join as co -hosts with deep insights and knowledge and building things in engineering.

And sometimes they join as guests and just like chill here and like get their news from the live show. It's incredible. Muse Wander is their new model. This is their opus, the beats opus on multiple things. So I think it's time for us to show some numbers. How does it compare to Argon? Holy shoot. Yeah. So this is the thing. They did not do any Argon comparisons.

The Muse Wander full and Muse Wander on max are compared only to Cloud Fable 5 .1, Astra 6 and Cloud Opus 5 .5. And I think, Peter, maybe this is the one that you called out. They have a new benchmark called Personal Superior Dangerous Index. Internal Composite Index of Personal and Knowledge Work. So basically doing kind of like an artificial analysis -ish thing, capability -ish thing.

I'm not quite sure how that works out. And on that index, Muse Wander, the one that we don't get access to on Muse Wander full is 59 and Muse Wander is 54, beating pretty much everybody besides Opus 5 .5 max.

Which other ones here, Peter, would you say? This is crazy. Where is like... Wait, hold on. Just one second. Peter, which ones of these would you say are one times or benchmark or new ones? Well, like if you scroll down a little bit, so we've got GPT, PDF, JobBench. Yeah, GPT, PDF, I have never seen before. JobBench, I have never seen before.

I think we're missing LTG here. He knows a lot about some of these. I will say on the artificial analysis index, which I think a new one is coming, this model does not beat Astra and Fable on that specific index. But also that index, again, it's testing generality of models and I think coding agents and not necessarily personal assistants.

OS World, for example, we know and love, right? OS World is a benchmark that measures computer use and browser use. And on this benchmark, let's see, Muse Wonder, the one that we do have access to, matches Fable and is significantly below Opus 5 .5 on computer use. And even below Cloud. So computer use, this model is not the best.

Which, okay, so going back to what assistants need, computer use is like the number one thing that I need for my assistant. It needs to go and click the right buttons. Otherwise, it's not useful to me. A very interesting set of releases here. And, honestly, you mentioned Argon. We're going to talk with folks from Gemini about Argon in about an hour.

We're going to have Omar back and Genuive. So we're going to talk about that with them.

I think we're good on the Muse coverage or anything else left there? Wolfram, go ahead. Yeah, since we were looking at the benchmarks, as I'm creating Wolfbench 2 .0, I have a crisis moment and I realized something. It's so hard to benchmark agents because they are complete systems. They are not just the model. You have the skills and memories and all this.

And everybody using an agent has a different one, even if it's the same system, because of all the different personalizations, connectors and so on. And even the internet access and the search providers, they all affect the results. And the thing is, nowadays, we are now at a point where you can have your agent create a benchmark for your own purposes and use the same system.

I'm doing it with Hermes. So I can put Hermes in the sandbox with my soul and skill and so on and use a model and do some tests specifically for the users I have. And I think everybody can now do this. Just have your agent build it. You can build all the software in the world now. You can also build an evaluation. I mean, as someone building evaluations professionally, that is a strange moment.

But it is really here now where you can have a customized evaluation for your own purposes. With the agents you are using and test new models, test new agents, all of this is now possible, which wasn't possible. There was a lot of work now. Now it's possible. There was a talk at AIE that calls out this paradigm where you're like, you talk to your slab generator.

You're like, hey, how can I know that you're right? And the slab generator says, well, I can do an eval for you. Here's a few things to test myself on. And then you say, okay, do it yourself. And so the slab generator generates the evaluations for itself. And they're also slab. And where are we if nobody reads those traces? We're going to talk about reading traces in a bit in the cloud section.

This is Muse Wonder. And the call out there for our open source folks is that the new spark model is going to get open source. This is a new llama. And I think that's one. That's great. I need to start using the word wonderful because it's confusing, but I think that's great. All right. Let's talk about Anthropic. Anthropic is big in the news this week.

Anthropic gave us four stories this week. You can trace your agent now, which is a very big one for at least the sponsor of the show. You can now or I guess again, we're so back. You can run the agent SDK on your subscription. So this means that you can plug it into Hermes. I see Wolfram shaking his head very strongly because folks, I have told you before.

So all of this personal agent type revolution. What I mostly need from this is that personal agent ability with Opus 5 .5. And now it seems that while I was able to pay API prices for this before, now I could do this with my cloud subscription. So I'm definitely excited to try this out. I've seen this in multiple coding harnesses and agents.

What else do we have? Anthropic also showed investors a $16 .8 billion quarter. And also rumored Fable 5 .5 in Azure coding, which is a very interesting thing because Opus 5 .5 is kind of beating Fable 5 .5 for most stuff. They're not like super, super strong. So let's talk about some of the Anthropic stuff. How is this lab? One more. Yeah, one more.

One more. There's even one more. They posted an article about strengthening their defenses against distillation attacks. So after clamoring for, oh, we have to regulate and ban this stuff. Now they basically did it by themselves, increasing their defense. They reduced the amount of distillation that happened to the reasoning traces and so on by 90 to 99%.

So it will be much harder to distill cloud reasoning now. And we will see what effect that has on other models. I know the effect that it has on me. There's not a day goes by that my clouds are disconnected. I need to re -log in again. And then the remote control thing fails. And so from New York, I can control my Mac. I don't know if this is the effect of their security on the distillation, but it affects me personally.

I don't want to call this out. Anthropic, please, please let me use my clouds wherever I am. This is getting old. This is really getting old. All right. Let's talk about this.

Let's talk about open telemetry, which is kind of a nerdy subject, but it's an important subject. Open telemetry is this open standard for how systems communicate their traces to each other. Basically the logs. Open telemetry has been around for a long, long while. And then folks like the We folks from Wets and Biases and now the Agent Lens folks, which we're now launching in Deploy Preview.

This is how systems communicate with those platforms. And now cloud itself, the cloud harness is now supporting open telemetry. What I'm saying is, before you had to run something like a OneBeHive mind, which we told you about, that shows you all the cloud conversations. Now cloud itself can send its traces to whatever tool of choice that you have.

Obviously on this pod, we strongly ship the Core with Agent Lens product we just launched. But anything that supports open telemetry, cloud code not supports. I think it's incredible. And I think it's very interesting how what you just said Wolfram, where they try to prevent distillation, while also simultaneously allowing cloud cloud to send his trace in open telemetry.

Though, this was never a problem. Cloud was always writing the conversations to local, right? So you were always able to get it on disk. They were not, they're not encrypted. The reasoning traces are encrypted, but the conversations themselves aren't. But I think it's a cool tool. You can send your traces to observability stacks. And this includes everything, agent turns and tool calls and MCP calls and latency and token usage.

So you can like properly measure how much cloud is costing you. And if you're a company, you can do this at scale.

Oh, there's one stat here from ramp. They said this helped them to cut token use to 10 to 20% on some agents because they learned that some of their agents are running through loops. Will you guys be, are you guys following your agents in any way? Are you evaluating them based on this way? Will you be using this from cloud code?

Nistan and Jan both shook heads. Definitely going to try. Somebody pick it up. Yeah, this sounds good, especially for stopping loops and getting overall agent stuff. I have noticed that cloud code does, does tend to stop often. If you have anything to do with the reasoning now, it just like triggers it as a, as a, as a word. Like if you're working with any type of LLM, like I've been working on WebGPU.

The moment I even like mentioned anything about reasoning now, now it just triggers it right away, which is like for a small model. But yeah.

Still good though. It's, you can't beat Opus.

Opus inside the cloud code harness. I think it's still the best tool for overall tasks and doing some stuff. Opus also help prepare the show. Just Opus low. If you need to do something, just try all in Opus low guys. It is just straight up vibing. So I think because of this reason, Anthropic is also going to do an explosive IPO. They reported an insane quarter, $16 .8 billion, about 45% from the second quarter increase.

Adjusted profitable to the second quarter in a row and cash flow positive. Anthropic is profitable and cash flow positive and gets billions of dollars and going 45% from the previous quarter. Still. It's absolutely insane. I think they have way better opinionated coding than the other models. I think people are impressed more like by the agentic and completeness ability of Astra, but like the actual quality of the code.

It's not that good. I mean, it's pretty good, but it's not as good as Opus. And when it comes to DevOps, even Opus is bad. And Astra is like even below that. Yeah, I think they still have a lot of tricks. And it's mainly because of Haiku, honestly. Haiku is like the most disruptive model I think they've ever released at that price. And they even give you $200 in credits now to just use throughout apps.

That's not going to run out at one cent at 100K. Like whatever app you're going to serve to people, if you just keep it under 100K context for Haiku, which most conversational UX stuff is good enough. That is more than enough. And on the max plan, that $200, that's probably going to support like a thousand users pretty easily. Yeah. 10 cents in, one cent cash.

That's, yeah, it's just way too cheap. So let's talk about the subscription saga. Wolf, I'm going to put you up there. As we said, we would love to use personal agents with the Opus 5 .5 level. Now it seems like it's possible. Can you cover that a little bit? Did you see this? The cloud SDK and the SlashP stuff. That's why I was waiting for the new Hermes update where this is now possible to do.

So I canceled my subscription with Opus when, or with Entropic, when you weren't able to use it without getting banned. Some people apparently got banned for using it. And so that was when I decided I don't want the subscription if I can't use it with the agent I'm using. I pay for this. And if it's cloud code or if it's Hermes, it shouldn't matter in my opinion.

And now it's possible to use your personal agent, be it Hermes, open cloud, whatever you use, and connect the SDK, the agent SDK and use Opus as the main model in your own agent that way. Again, in a legal way without being banned. So, yeah, now it's possible again. And I have to reconsider these abstracting. So the cool thing about what happened two weeks ago is that not only did OpenAI always allow this use case, they also literally shipped a button and integration called like sign with your chat GPT and then the other products could use your tokens from your subscription.

Right. So Devin is one example from Cognition and supports that use case from the get go. Hermes agent was called out on stage. New research. Shout out to New research. Our good friends from there. And congrats on their round again. And Hermes agent is now going to support both. And this was always the use case with OpenAI. It looks like an tropic kind of needs to follow suit.

So now it's going to be possible. Quinn Slack, also a friend from AMP, the coding agent said that this is why now you can use their models. Peter Goster, will you take the risk? Will you use your antropic subscription within external tools? Or will you just say, ah, Cloud Code is good enough?

Yeah. It is kind of a funny one because I think they're clearly trying to push you in the direction of using Cloud Code. Yeah. So, yeah, it's kind of annoying. I don't know. Aren't they making enough money already to just let people be? Like, I feel like there should be more freedom. But yeah, I mean, I don't personally, I just don't. Because I've got enough stuff going on.

I use other models via Pi, for example, if I need to. I find that more convenient. Because, and I'll tell you why. Is that I think with Anthropic, with OpenAI, they clearly, their main priority is to use their own models. Right? And to kind of coordinate everything. And I think it's just easier to use other models in other ways. But yeah, I prefer to just match the harness to the model, basically, simply put.

Harness to the model, I think, is very important. And something that folks miss. Cloud Code and Cloud definitely go hand in hand. Again, Muse Wander and Muse probably go hand in hand as well. Codex is also very good with Astra and less so in other places. And that's why it's for agents like Hermes, it's sometime to catch up. And there's a rumor, and we don't love rumors here, but we will say it, that Fable 5 .5 showed up.

And I think I'm getting tired of rumors. So I'm going to do this for just one second. This is not planned. But hey,

Anthropic is going to release Fable 6 also. And then Opus 6 is going to come out. And then Opus 6 .1 or 6 .5 is going to come out after that. It's going to be even better. And then Opus 7. So if you heard it here first, Opus 7 is coming. We just don't know when or how. But I'm getting tired of this. Like, oh, this model is coming. And this is getting just a lot of...

Okay, when we'll get it? We'll get it. I don't understand why the X algorithm for all of us is so fine -tuned on just like trying to snipe the news of something that we all know. Is on the continuous improvement loop all the time. Just tell me if that's interesting to you when you see somebody and just like, oh, I've seen this leak or whatever.

We know the models are going to get better. We know it's very important for these labs to tell you how better they were going to get. It was especially important pre -IPO. I'm interested in testing them out, playing with them, hearing what Peter says, what Niston and Wolfram and Yam say, rather than just like knowing the news. But given all that, somebody did spot Fable 5 .5.

So I'm very excited to see how that's going to go.

Comments? Did you guys hear my... Do you push back? Do you feel like it's important to snipe and like do breaking news, et cetera, at this age? Or are we getting a little bit of tired of models getting better? Numbers got better. I want to see them run a full factory or something. That's that rumor. Like I want to see a whole bunch of human noise running whatever model that...

Yeah. Yeah. Yeah. The models are going to get better, of course. There's going to be all Opus 6. I think Anthropik is going to not allow everyone to just stay on top of the benchmarks. They always have something in the back end. But yeah. Rumors now are more acceptable for like actual humanoids running a factory, running a hospital or something.

We got to leave the rumors for more like actual physical stuff.

Yeah. I agree. That's not sure why. Like everybody else, it's common sense. I don't know. Yeah. Wolfram, from an evaluation standpoint, do you think that sniping the news and telling you, oh, this model is coming is still important or are we like past that? I mean, right now, the most important thing for most people using agents is to have a way to use a subscription.

So if the new model comes out and it's API only or it's so expensive that you run out of your tokens as quickly and you can't really use it, it's not as useful as a model that is just smart enough. And I think we are reaching a point where a model is smart enough for most users and the much higher intelligence is only for bigger projects and stuff that you as a user of a personal agent are not even needing for book me a flight or plan my vacation and stuff like that.

I think there's a limit to what we really need and everything beyond that is on a company level and not on a personal level anymore.

I completely disagree. I think we're going to look back at this a year later, going to be like, oh, we were so, so stupid and naive. I think it's going to be amazing. And I get your point. Like, I think it's kind of kind of tedious, you know, this number is going to go up. But I think the fact that we shouldn't take it for granted. I think OpenA and Anthropic are the two labs that actually ship models very regularly.

But we haven't seen a full pro model from Google for, what, eight, ten months. I haven't counted. So they're only shipping flash models. So I think, well, I find it perfectly reasonable for people not to pay attention to rumors. But I think there is information in it when companies actually ship. So let's say if there is a scenario that Anthropic doesn't ship a 5 .5 Fable for another two months.

I think that's also kind of newsworthy. Like what happened there? So, yeah, it's, I, the, so I think it can kind of go both ways, which I don't think you have to have rumors for, but I still, I still get a little bit of information out of it. So, yeah, we'll see. Like I haven't counted how many weeks OpenAI, when did the first kind of, how Dastra released.

It's probably what, like a month ago, six weeks ago, something like that. Dastra was around a month ago, five weeks ago, something like that. So, so yeah, you'd expect them to, I'm going to leave six months all aside, because I think that was like maybe out of cycle a little bit, but the next Dastra, you'd think that's probably another month away.

Right. So I think, but if it's two months away, then I think we'll get some information out of it because it's like, they're probably spending more time, I know, aligning the model or something like that. So, yeah. So, you know, I'm very pro -rumor, but you know, I'm also a psycho, so you know, you don't have to be. I think, first of all, the sending opinions is, I talked to our fans in IRL, they love this.

When we disagree, we give like multiple sides of the story and I think they actually do love this. So that's great, Peter. And I do think that you're giving an additional insight there where now the Anthropic is going to be public in IPO. Whether or not they're going to release a new model actually will move the news, move markets given how big Anthropic is.

That's a very interesting kind of thing. So maybe we're after doing this for quite a while. Maybe we need to find the passion again to say, oh, this model is coming. Yeah, let's go. But yeah, we'll definitely get there. So I think that's enough about Anthropic. And given, let me see if I can make this connection here. Given that Anthropic is going to IPO likely on Wall Street and given that I am here very, very close to Wall Street.

And the reason I'm here is AI Engineer. Let's talk about the engineer in New York. Nice. Yeah. Okay. This week, AI Engineer in New York. This is the second New York AI Engineer conference. For those of you who are new to our show and not familiar with AI Engineer, Thursday, I started as a Twitter space and then became a podcast. And the first time that I ever told anyone else when they said, hey, what do you do? I told them I have a podcast was on my way to AI Engineer in 2023 to cover it.

Since then, the conference has blown up and SWIX, the co -host of Latent Space and the creator of AI Engineer category and the conference has been curating incredible folks throughout these conferences. So we went live earlier this year, twice. One time from Moscone West with Wolfram during the in the middle of show of AI Engineer. And then Peter actually interviewed you and we chatted in London as well.

So we went live from London. Wolfram as well. And this time, for some reason, they decided not to do it on Thursday, but we're here in New York after AI Engineer. And what interests me about this specific conference is the zeitgeist is incredible because a lot of folks come there, first of all, to learn so that you know that the speakers are going to deliver something, but also the hallway conversations.

So I have a little bit of a like debrief for you because I know that many people did not make it to New York. None of my co -hosts are here in New York right now. And so here's a little bit of a debrief. This specific conference specific because it was around adjacent to Wall Street. This focused on finance and the eye. The big hedge funds were all here.

Two Sigma. Let me go find them. Capital One and just Bridgewater, Apollo, Cyrodiil, Bloomberg, like all of those folks. And then OpenAI and Anthropical were also here. The big theme from Latent Space where they said was autonomy versus security. Now that the big companies are actually running agents. And when I say now that the companies are running agents to contrast a year ago, 2025 was the year of agents in quotes.

And the zeitgeist was agents don't work. This was before Opus 4 .5. This was the for the kind of the November, January revolution. Just scroll back just a year ago. Agents did not do work. And now all these folks are saying, oh, shit, agents do so much work that we need to like put guardrails on them and do like, how do we how do we deal with autonomy and models crossing the thresholds from now to the end? I mean, and then from some providing non coding work to like making decisions.

There's now agents inside banks. There's golden datasets. There's evals. There's a bunch of stuff that these are not contents with that they didn't have to a year ago because a year ago, agents did not work. Most of the AI in companies were software engineers trying to get lines of code out. Now there's like a gentic things happening inside most of this once.

But two years ago, I think I posted something about I've never seen an agent actually do anything useful. other than Pliny's buying crypto. And that was only two years ago. So just remember, everyone's opinion is always wrong at some point. There was a tweet from me saying, I don't know what people are doing with agents. No, but also we're changing so fast.

It's completely flipped one. And this is what we're talking about when we talk about acceleration. You can't be looking at the current state of the world right now and say, hey, AI is this. You have to start. But this is why we're talking about acceleration. This is why we're trying to bring in the news and the use cases because we kind of feel that this is happening.

So all the banks are now thinking about this very strongly. Not only AI engineers are spending tokens. Banks are actually using agentic processes. Max Kanaat -Alexander. He is, I believe he's a distinguished engineer at Capital One. He was the code simplicity guy. And he did a body show Kubernetes stuff at Google. He's a fascinating dude to talk to.

His talk was the highlight. He said, like he talked about, does code quality even matter? Given the background of this person, I think it's very, very important. Principal software engineer and open source back in Google. He led a bunch of the bug stuff in Google. He basically says the code quality matters in proportion to how durable your software is because AI is about to make a whole new kind of software way more common.

Disposable software. Just write once, use it, and then just throw it away. So when you have disposable software, does quality matter if it works? If you build a calculator or an app for yourself, party planner, does it matter? So this is one great show. And I recommend folks to tune into the AI engineer live stream on YouTube, which you can watch if you want to.

The talks will get released. The other kind of like hallway conversation was, first of all, about evals and whether or not agents are working. But as I said in the beginning of the show, the thing that was highlight for me, which I do want to ask you guys about as well, is the disconnect between how advanced all those AI engineers are in use cases and harnesses and personal agents with the revolution of personal agents for normies.

And how when I talk to some of these AI engineers, they're like, hey, do you use Instinct, Muse, Grokbot, or whatever? And I don't get an answer yes from them. And that's really, really something. Because many of the engineers just build their own harnesses or use cloud code or codecs, and why not use codecs for some of these stuff? And so there's a big disconnect, which is very interesting to me specifically because how big and wide OpenClaw and the Hermes agent got.

And those are harnesses you can tweak and play with. So I did notice this in the halls as well. And specifically, this is highlighted for me because I did talk, like Wolfram does, to every person in the world. There is not an engineer. Wolfram talks to stewardesses on planes to try to AI build them. So I did the same. I took a page out of your book and I talked with, there was the token billionaires launch we told you about before.

This AI engineer had this again. So if you are spending more than 1 billion tokens a week or something, you get this golden card. I think I have it. Hold on. Wolfram, there's a new card that you need to get. The token billionaires card kind of looks like this. And then this gives you access to this, like, lounge. This lounge is kind of like 4 billionaires launch.

There were a lot of service providers in that lounge that were not part of the conference, and I tried to talk with them. Here's the thing that I got from them, not from the AI engineers. All of them know about Muse because Muse is getting pushed down their throats in every Instagram post. Zuck is really going out of their way. So they all know about Muse.

Many of them didn't install it or didn't get why it's good for them, didn't yet catch that. So those conversations were fascinating. Let's see what else. So, hey, engineer, if you haven't been there, the highlight for me, and I tell it to everyone also, if you're listening to the show, you're good enough to submit a talk proposal to an engineer.

And if you submit a proposal to an engineer, you get access, and Peter both can comment that this is true. You get access to this. Where am I? Oh, there we go. You get access to this amazing speaker's lounge where all the speakers that are hand -selected are top of the level of people. And you get also access to speaker's dinner. And to me, the speaker's dinner and the speaker's lounge hangout is the highest tier of folks who understand whatever word I say.

You don't need to explain what harness is. You actually learn a lot. But one example that I can shout out is I just were walking by, and somebody's tag said linear. I was like, oh, who's this dude? Oh, he's the senior, I don't remember the full. I think he manages software for linear, all of linear. Linear is a genetic forward task management software.

And I had a bunch of feedback about linear. We just sat down just right there and talked about how I use linear with my agents and what they can do to improve. This is one of those conversations. So if you are listening to this, but if you have anything to say, I strongly, strongly recommend to apply for a conversation talk at AIE. Wolfram, Peter, any thoughts on that space?

It's the highest level of networking you can have in the AI space because you are not talking to some other attendees of a conference. You are talking to the big shots who are creating the AI news we are reporting about. So this is super, super amazing. And even if you don't become a speaker, I mean, just attending and going to the booth and talking to the people you meet there, there are all the people from Google and Entropic and OpenAI.

They are all there. So it's the best way to connect to the makers and shakers of the AI space. Definitely. Best conference I know.

100%. Peter, go ahead. Yeah. Yeah. Yeah. No, I would say also everyone can apply, but do take time to make a good application because I can't remember what the stats were, but I think Swix was saying, not for this one, but for the one in London, I think it was like the acceptance rate was very low. So I would say put effort in. Put effort.

Yeah. The reason why it's a good networking opportunity is because it is a bunch of good people. So, yeah, it's worth, you know, don't just LLM generate your summary and proposal. Yeah. Take your time. So here's an additional insight that I got, and I think we've talked about this on the show, but obviously this is on our mindset, right? So AI engineer in London, the Zeitgeist, we're talking about, should we still read code? We don't write code anymore.

Should we still read code? I talked about the ZL continuum on my way back. I think it's clear now that barely, barely anybody's reading code now. We're like stressed how to even test it and make it valuable. So the Zeitgeist here was the end of attention. And how do we scale from managing one or two agents directly by steering them and seeing outputs to running multiple, multiple tens or hundreds even agents? And how do we have the attention to attend to all of that and make decisions or whether or not we just like let this swarm run and deal with this? And I think it's a very interesting.

I talked to many, many people. There's not a lot of solutions. Ryan Carson, our, by the way, shout out Ryan. We hung out during this whole AI engineer. Our previous, our co -host, me and Ryan, we were talking to a bunch of people. Ryan gave a great talk about this as well. And he's missing a control plane. He's missing like a one central place where he can like see all of the stuff that's working.

Another thing that was in the Zeitgeist is the move to the cloud. Many people are just getting limited by their local environments. You can see this from the Frontier Labs as well. Now, Codex Cloud just launched. InDev Day renewed, launched. Cloud, you can now hand off tasks to the cloud as well. And they even give you, like I think you said, they give you some budgets, like $100 to try the cloud thing.

Obviously, Devin and Cursor, they have cloud agents. And folks are getting to the limit of their environment, not only because it doesn't run 24 -7. You need to run Caffeinate to keep it running, but also because the local environments are running out of memory. And Peter, I think you mentioned this as well. You're like coding so much in your things that the agents just like killed your computer.

So you got a different box. And when you spin those up in sandboxes on the cloud, you kind of don't have this restriction. You can't go wide. So we're getting to a point where not only are we using more billions of tokens more. My case, definitely Peter's case. Definitely Wolfram's case. All of us here, probably like 10x for the past few months.

And as I guess in the engineer as well, we also need a place for that whole thing to All the tools to happen and that's not happening on local environment.

I heard the engineer. Hi folks, next up we have Omar Sansevro and Genevieve Husk, I believe. They're joining us now. Just before that, as always, given the show is sponsored by one sponsor only, CoreWeave. I would love to tell you about some of the stuff that we have. And then we're going to talk with Genevieve and Omar about, but finally we got the new big Gemini back and it's so exciting to hear about this as well.

Let's go to this week's buzz and just right after that we'll have our guests.

CoreWeave. From CoreWeave. I think that it's time to sunset that transition and come up with a new transition, Wolfram. I think that's awesome because as of two weeks ago, CoreWeave Forge is now the home for the weights and biases as a tool. And also a bunch of other tools, specifically AgentLens. AgentLens launches this week. And if the highlight here for me in AgentLens, Wolfram, tell me if you're also on this, is that Cloud Code now supports open telemetry exports.

And then now you can use our tools to actually look at the agent runtimes that you have and analyze if it did the work correctly. So, would love to just let you know that you can now use that. You can sign up on the forge at coreweave .com. The highlight for me from Fluor Connected two weeks ago was that given the previous life at CoreWeave where you had to go talk to a salesperson, register for loan contracts and sign up.

Forge for the first time is our register as a regular SaaS offering. And then you get access to a bunch of tools from our core of AgentLens cloud. Tools like distillation as well. We talked about model distillation a lot on the show. But now via the open pipe folks, which are great folks, friends of the show, you can get a model distillation for yourself.

You don't have to set that all up. It's all part of forge .coreweave .com.

Wolfram, did you set up your cloud code with AgentLens yet? Are you sending OTA? At the moment, I don't. It's all local right now, but it makes definitely sense to integrate all of this. And the integration is what Forge is all about to have integrations between all the different tools and the whole AI loop from the evaluations to the inference it's running on to the curation of the data sets.

It's all connected. And that is a new thing. I think it's not just Weave or the individual product where you have to export, import stuff. You can jump between them and the connection all on one. I want to call out yesterday. Yesterday, we said at the event that we did, we sat down and we talked about some stuff. And obviously, I was like, okay, this is a core -responsive dinner.

Let me at least tell you about the offerings. And folks who use Weave and Bias didn't even know we have ARIA. And somebody's like, what is ARIA? So ARIA is an agent assistant, agent research assistant, basically auto research from Karpati personified in the product. So if you are using Weave's Advices for anything, people didn't even know that we have this offering.

So we definitely need to work a bit about externalizing our offerings. But if you have seen auto research as a concept from Andre Karpati in his suite, it's now in GA. You can just use it inside Forge on your training runs. All right, folks, I think this is enough about Forge. And now we're going to move on because we have not quite breaking news, but definitely news from the frontier of AI with two great guests here.

And let's go talk to them. Hello. Hi. Omar and Genevieve are joining us from DeepMind, from Google DeepMind. And you guys have news to share. And it's... Can I just say it's about time? It's about time, Omar. We've talked on the show before. And folks, it's about time for a big model. So you guys launched Gemini for Argon. But I would love for you to tell us about the launch, about what's new, about what are the use cases.

I'm all ears. It's out for everyone. It's number one on Arena. And it helped make Gemma three times faster, Omar. That's kind of your corner as well, right? I would love to hear about the use cases. So please give us a highlight and then we're going to dive into use cases. And Genevieve, great to meet you. Thank you so much for coming. Thanks for having me.

Omar, let's go. I can go first. So yeah, Gemini for Argon is our latest model. The LDR, it's a great model. You have probably heard many things about it. My only recommendation is to go try it out. And I think many of you will be pleasantly surprised. Yeah, the model can be used for many things. We've been using it internally. As you said, we've been doing optimizations for Gemma.

We have been able to achieve over 100 ,000 tokens per second with the E2B, so our smallest model. The model has a 1 million unified context. So pretty much you can pass very long videos, but you can also have very long outputs. So that is quite exciting. And yeah, we have been very excited to see what the community has been building with the model.

What do you think Genevieve? Yeah, we've had so much fun building with Argon. It had been a while since we launched a new model. And so most people will know that this is the first model that came out in our Gemini 4 series. By far our largest training run ever. And so Argon, I think, is the first actually quite early checkpoint that already is doing really well.

But we're also super excited about everything that we can build next now. And so Argon is already training our future generations of models. I think we've really enjoyed the fact that Argon can do many more long running tests. And so now everyone is shipping a bunch of stuff overnight and you wake up and you just wake up to really fun things that Argon has been cooking in the background.

And so it's much better at 3D, much better at creativity. Personally, the writing is really pleasant. And so in general, it's just been super fun to build things with the model. And again, we think there's many more great things to come from us in the next few weeks. I would love... Yeah, go ahead Omar, please. Just as usual, we have been working with all of the open source ecosystem to make sure that you can use the model regardless of what's your favorite tool.

So we have been working to make sure that this model works well with OpenClub, with Hermes, with all of the tools,

sorry, Blanchain, Versal and so on. So super excited to also collaborate with all of this ecosystem to get the model out there. I want to say the release definitely is a wide surface release, right? It's available pretty much everywhere. And also in your own Gemini API in the iStudio. Shout out to iStudio folks for always bringing us in to the fold.

Gemini app only for Pro and Ultra, is that correct? So folks who are subscribing to Gemini app on the Pro and Ultra tier are giving this and on to gravity as well. So $2 in and $10 out. This is introductory pricing? Exactly. Yeah. And then what happens then? Extension, new model, another interactive pricing?

Breaking news for next month. All right. And then based on what it says now, then it doubles. One million token context. And I think it's about multimodality. Google and Gemini model specifically were always kind of like the front leading in multimodal, including like watching the video that you send to it. One million token is a great thing for that.

Yeah, you'll see that the model is very good at understanding visuals and the space. And it has a very good understanding of the world and physics. So many people kind of forget that models are not just for coding. Models are to understand the world and the world is highly multimodal, multilingual as well. And in general for Gemini, we make sure that this model works well across all of these kind of use cases.

That's incredible. Let's talk about Vision. RoboFlow called this model is like number one in Vision and Detect OCR stuff as well. Very, very, very strong. Any specific demos around Vision that you guys want to show us? Any cool things? That'd be the... Let's start with the ones that I have ready and then... No, let's go there. ...or if you have anything else, Vision in the background.

The ones that I picked for today are mostly very cool looking. But yeah, it's great to see the Vision capabilities always. If you're building with a model and it can check its own work, if it can record videos and kind of see what's going on, then that is super, super helpful. So even for some of our 3D stuff that we're now doing, it's just so pleasant that Gemini can watch its own videos back.

Even things like motion graphics is very helpful because it can look at a video, understand the style, and create it. And so I made a trailer about Argon earlier this week that took me just maybe 30 minutes and it came out really good. But yeah, in terms of the demos that we've been cooking, the first one that I made was Desk Racer. And so first of all, if you go into Google AI Studio and you click on the gallery, you can find a bunch of stuff that we've made with Argon.

And so Desk Racer came from our multi -modality where I wanted to build a racing game on top of my actual desk. And so the microphone that you see here and then this like cute little cat just gave Gemini some reference images and said, okay, just built me a racetrack. And what's fun about this is, I mean, it's a long running task to understand where to go around the objects, to make it look good, to sort of plan the lighting and everything.

And so now if you can play this, and then Argon even, I can switch the camera view and turn the sound off. That's so cool. Argon even prompted our Omni video model to extract sounds like engine sounds from different race cards, listen to all the sounds, found the sort of audio that was able to loop and then stitch all of that together. And so like, yes, it looks good, but it's also cool to think about the rest that's happening in the background to in the end, come up with a very shiny result.

And so this one I really like because it's where I spend all day every day. But it's fun to see Argon do such a good job and so build all these little other camera views in it. And so this you can play in AI Studio. There's a timed mode where you just drive three laps. And so that's pretty fun. And then this one. Just before you switch, could I ask you for our audience, how do you approach building this demo? Is this all in the AI Studio or using any other harnesses or using like the internal tools? Could you tell us how to get building this where Argon can download the files, then like using Omni to generating things and like extracting? Where is all the tools happening?

Yeah, that's a great question. And actually, I wrote a guide about that earlier this week. And so in essence, I started with the reference images, use grill me. So like ask me questions about how I should plan this project and then just use a big goal command and said, you know, I just want you to spawn a bunch of sub agents, work on it however long you need.

Some things that helped were a using adversary sub agents. And so there was always one sub agent whose job it was to look at the outputs, grade them for accuracy, for lighting, come back with feedback and then give a rating out of 10 and then keep going until the rating was very, very high. So that, first of all, was really helpful. And then secondly, we've all really been enjoying using a nano banana to make 3D even better.

And so in this example, for instance, I'll turn off the audio, but this is a demo that looks really, really good. It's like a pagoda demo. And so if you have a forer render and you give it to nano banana and you say, make this look really good and realistic, and then you pass that image back to Argonne and say, do whatever you need to do to make it look like this.

I'm now using that in all of my 3D projects. And so the lighting that you saw in desk racer, the neon lighting was designed by nano banana because at first it looked very bland. And so this, I think is a really nice way to interact with any model. Just that image inspiration is a really good visual reference. Oh, that's incredible. Yeah, I did write a guide about this.

Sorry, if you look this up. No, no, it's all good. This is like a definitely great tip for our audience as well. Use visual references because the model is very good at multimodality from other models for this. That was incredible. Amazing. Show us another one. So this is great. Yeah. So this one is pretty cool because like, yes, it's fun to do things with a blender and tree .js.

But in this case, my colleague actually asked Argonne to create this 3D scene completely from scratch. So zero library, zero external things being pulled in. And what's cool about this is if you think about how a model needs to do that, is it needs to think about all the light, all the ways that objects might interact with each other. And so the light bouncing off the wooden floor here needs to be refracted in a certain way.

The light shining through the glass needs to exist in a certain way. And so if you think about the amount of real world understanding and then also the quality of code generation that you need to make happen to make a project like this, that we also thought was very impressive. And then the other thing that was fun about this video, I'll spare you the audio, but Argonne also narrated it by itself.

Pick the music with Lyria. And so again, the idea that you can build something cool, give it back to Argonne and then say, now make this into a cool video. That all just ends up working. And I think in our previous models, we probably would have had to babysit that a lot more and give much more guidance on how things should look. And so another one that I think is just a cool example of what you can do.

I want to go back to here with Omar as well, and maybe both of you. I mean, the token output changed as well. Like Argonne generates output significantly larger. Could you tell us how this affects like demos like this on other stuff? Omar would love to hear from you as well. How does that improve people's kind of like results specifically because.

If you think about kind of like evaluating models, a lot of the times the folks are focusing on the Pareto frontier corner where reasoning used to take a long, long time and just like with more reasoning, you get better results. And then there was a little bit of a movement towards improving that so that the model will give great results, but still not reason as much and not waste as many tokens.

But this is not about that. This is like about the model being able to generate like long, long things. So could you tell us about the outputs a little bit? I mean, this is mostly for long horizon or scientific projects where you do want the model to do lots and lots of reasoning. And that's pretty much the kind of task where you would need that kind of context in the day to day.

Most likely you do not need that level of output for input. Maybe right? Like if you want to pass a very long video or lots of PDFs, you may need to do that. But this is mostly for scientific or long horizon. That was awesome. And Genoviv, did it affect any of the work that you did? No, I agree with Omar that in most cases, like, yes, you could write one or several books in one prompt, but that doesn't end up being super useful.

So to us, it's mostly on the very long reasoning. One project that I have been working on in the background, but it needs a bit more work is one 3D project where I'm modeling 55 cities in one go. And so whenever it sends off to the sub agents, then they all start building many, many cities at the same time. But again, in most engineering applications, it's still much better to just do things in slightly smaller batches.

But if you have to think about a very long, difficult math proof for a long time, then you don't want the output window to be constricted by the fact that a model already thought for a long time. And so that's where we see most application. I think it is a very new thing also for us. And so there's also some complexity with how this works, even in things like external harnesses.

But in general, the idea that the scientific community can use this is very interesting to us. Yeah, that is very exciting for sure. So we talked about the visual kind of understanding and number one, Omar and Genoviv, I think you mentioned this, that now Argon is working on next versions of Argon. That suspiciously sounds like RSI. And Omar, you're talking about Argon training GEMA models, and that's like parent teacher, not necessarily distillation, but it's also in the process of training them.

Give us a little bit more of that. I think people do want to know whether or not this model can help train other models and how that works. Specifically around GEMA work, how much of this was autonomous, Omar? How much of this is like intervening? Could you guys talk to us about model improving other models in the future? Yeah, I think just as we think about AI for science, AI for math, AI for biology, AI for quantum computing, we can think as well about AI for AI, right? And I do find this a very exciting area.

This is something that we as an ecosystem have been doing for some time. It's just that as these models are getting significantly better, it's quite exciting to think about ideas. So for example, in the context of GEMA, what Argon did was optimize BLM, which is one of the most popular open source libraries for server side inference. And it did a bunch of BLM optimizations.

It found a bunch of areas where we could improve things. Similarly, you could see how models could help on co -designing training code or co -designing architectures.

So, yeah. Do you have anything, any last comments before we let you guys go? No, I mean, I think, yeah, the sentiment in the market about Gemini has varied. I would say, please give Argon a shot. It, to us, feels like a very different model from the models that we've launched up to now. And so I hope everyone gives it a creative shot because there's so much I think that you can do with it.

For us, I think it's the start of a new era without sounding too commercial. But yeah, I think this is a very exciting first step to what hopefully is going to be a couple of fun weeks and months. We've been waiting and we're now very excited to have it back and to see that Google is back. So, folks, thank you so much for joining us. Genevieve, nice to have you on the podcast.

Feel free to come back with awesome demos, which we always love. Omar, as always, has been on the pod for a long, long time. Thank you guys for joining us. Thank you so much. And congrats on the release. And we know that you have a lot of stuff to catch up on during the release week. We'll see you here next time. Cheers. Thank you so much, guys.

Bye -bye. Bye -bye. All right, folks, from one conversation to another. Now we're going to chat. Now we're moving forward and I want to introduce you to Nathan Benyash. I've followed you for the longest time, but I think we've never talked like this on video. And I followed the state of the art report. So let me just let me just do this reading.

Folks, the state of the eye report turns nine this year. Is that real? Yes. Nine years. I feel like I've been doing this for three and a half years and I feel like this is a long time. Nine years. The word AI was not familiar to many folks back nine years ago. Yeah, it was the era of like AI -ML and data science. Yeah. And this year's 244 slides, a scorecard on last year's predictions.

We'd love to hear about last year's predictions and whether or not we can hit them. And nine new predictions, including AGI 27. Nathan Benyash, the founder of Ash Street Capital is with us. The author of state of the art report. Nathan, welcome. Thanks for having me. Peter also joined us for this interview. Peter, thanks so much for making this connection.

I want to talk about some like the highlights, some favorite numbers. I have cloud led 27% of measured model R &D and Anthropic in August. We just talked with folks from DeepMind about Argon and like RSI, whatever. Would love to hear from you, Nathan, how this kind of like shapes, takes shape between last year and this year. People do talk about RSI being a thing.

People do talk about models training other models, whether or not we're going to hit this like thing. What are you seeing from there? What are the report says about RSI and continued improvement by the models? Yeah. Yeah, yeah. I would say like the major story in the last 12 months has been RSI and super intelligence, moving from AGI to super intelligence.

And then, you know, increasing evidence, you know, at the frontier that shows like we're kind of making progress towards this idea of like AI, building AI. I think what if like the two experiments I found most interesting there, one of them you cited, which is just the general stat of like how Anthropics like R &D team, when they're doing core R &D work on the model, it's like 26% of like the directions in research are being guided by cloud.

And that's up from basically zero a couple of months ago. Not to oppose this, but I think the code generation stat that I remember, maybe you have an updated one, but as far as I remember, like 80 or 85% of code is generated by cloud, maybe not directed. So yeah, yeah, yeah. So that, yeah, that was like a few, a few months earlier, you could see like a chart of like code commits that were, that were basically accepted by engineers, as a function of like how models have been changing.

And it was sort of like flat for a while. And then like the mythos class models came and then now it's like 8x the average, but you see a similar stat at OpenAI.

And then the other, the other data point that I liked that we made a slide on is this, this like experimental setup where a human researcher engages on a, on a, on a research path, kind of goes through like several leaves down like the node of that research path. And then at some point the path forks, and then one of the fork is a model that continues the research.

And then the other fork is a human that continues to research. And at the end of like both of those leaves, a human rates, which one was better. And, and you saw that like the old, old school models were definitely not better. Like I think 10% of the time humans preferred them, but then the mythos class model, it was like 60% of the time humans said the mythos suggestions were better than humans.

So I think that's like also, also interesting. And it gets to this point about scientific taste. Um, which I, which I think is like on the, on the tip of everybody's tongue, like no plan intended. Scientific taste. Yeah. Yeah. Just, just the idea for the, for, for those who are like less familiar.

It's like, um, I did a PhD in biology and what I remember were like the great papers were the ones where you would, the grad student or the postdoc would, uh, would be, uh, running a couple of experiments and then every week going to their supervisor and showing the experiments, the supervisor, who would look at the work and, you know, kind of mull it over and say like, okay, the next, the next best thing you should do for the story is this.

And I think the best PIs would be the ones who'd suggest experiments or ways to interpret the experiments that you did and the results you got in a way that you thought was like novel and interesting and elegant. And then the bad PIs like do that very poorly. And then you see this in the result of a paper where like the story kind of reads beautifully.

It's very logical. Like each experiment adds additional signal that the previous one didn't have. And I think that's like what people are trying to encompass in, uh, what they call taste. So can a model like approximate that?

Yeah. I think there was, there was one more from prime intellect, I think where it was the non, uh, GPT, right? Where, where, what sort of did it almost get to the human level, right? Yeah. But I think there was something that it was still narrow the scope of the work that it was there. Yeah. Yeah. There's a, like to get into this more of what I, there was, um, a study, uh, from open AI in the report where they looked at trying to categorize like the, the token usage of their, uh, of their engineers and researchers.

Um, and, um, and what they did was like identify some tokens that were about, uh, helping decide the research direction. What experiments should you run to test that research? Um, and then other kind of category of tokens, which is like implementing those ideas, like designing the infrastructure, engineering the infrastructure, doing the data pipelines and doing the data analysis and just writing the results of the study.

And the vast majority of the token usage was on just like the raw engineering or just like get the thing working and then writing about it. And, um, and very, very small amount of tokens was on what research should I do in the first place? And like, how should I test it? And, um, and so mine, I think there's different ways you can interpret this, but, but one of the interpretations to me is there's still like so much more to go on improving, uh, you know, the use of models for like ideation in brand new, brand new areas.

Um, you also gave yourself a scorecard. Can we talk about that from last year's predictions and misses? Uh, what stuff did you hit correctly? What's that, or like the report hit correctly? And I'm assuming that's you as well. Uh, what stuff did you miss? Yeah. I'd love to hear. Um, yeah, like the stuff I got right was, um, let it be a push back from big labs and new labs into open source.

And this is, and, um, in the U S and the West, and the idea with that is it was very clearly part of the national AI strategy from the Trump administration to sort of regrain, regaining supremacy of, um, of American AI and like open sources, a way of doing that. And very soon after we published that report, like reflection AI sort of went from the coding agent work that they were doing towards saying, we're going to publish an open, open weights model.

And then in the, uh, supercharged all of that. Um, by, by, by, by, by, by, by, by, by, by. I think it's rumored. I'm not sure if this is quite there yet, but by saying that the NBJ is going to maybe like get reflection in there after getting hugging facing there also this year. So yeah, you're kind of full side and full side and other potential ones.

And, uh, so you're bang on on that as well. So that one, and then the second one was, um, on NIMBYism for data centers, like not in my backyard. Like I don't want to have a data center next to my garden, um, that this would be a real problem in the, um, yeah, you know, hampering like the build out of data centers, but also influencing part of the midterms, which we'll see in the next couple of weeks.

Um, we substantiated this stuff with like a survey that we found where Americans say that they're much, they'd much prefer living near a nuclear power plant than a data center. Which is quite wild. Wild. Yeah. We covered this on the show as well, and we still can't believe it. It's just, it's just the amount of things that are going on and just people on the street talking about just AI boiling the planet.

Uh, yeah, yeah, yeah. And it's not just like in the U S it's certainly happening in Europe and in Korea and other places where, you know, there's been like a small proportion of the population that's benefited enormously from AI. And many people who haven't, and, um, you know, there's some social issues with that. Uh, I think the last, the last one that I got right was, um, you know, Chinese labs would overtake us ones, uh, in some major leaderboards.

And, you know, I think it depends on which, which timeline you actually look at and Peter probably knows this by heart, but, um, but certainly, you know, Kimmy and GLM and others have done really, really well. Yeah. What about some of the misses? I'd say like the, yeah, yeah, yeah. Well, um, there's been so much talk about like AI sovereignty, which could be interpreted in different ways, but the broadly speaking, the one of I, I, as a country, I want to have independence or control over my own destiny when it comes to, uh, building models, um, uh, training them, serving them, having the infrastructure for it and the energy, et cetera.

Um, and, um, and, um, and my bet was there would be this concept of like, uh, you know, I'm AI neutrality in that, that kind of mirrors like, uh, uh, sort of mutual defense, um, agreements where a country is like too small to be able to afford, uh, you know, its own military or its own defenses. And the same way it's like too small to be able to build its own data centers and have people build models, et cetera.

So it just forms like a strategic defense, uh, uh, you know, agreement, uh, or AI, um, provision agreement with a bigger neighbor and ally that says, you know, I will, I will, I'll promise you to give you all, all the AI in exchange for something else. Um, I haven't quite seen something like that yet. There's spent a bit of a move with a US PAC silica policy that, um, that tries to like bring, um, a collection of countries that have, uh,

access to core, um, parts of the chip supply chain. So it's like minerals or fabs or lithography or whatever. Um, but, but overall the AI neutrality thing hasn't really taken on just yet. No, no, no. Um, yeah. And overall I'd say the predictions are like directionally well, like this as a guy on LinkedIn who like looked at the last like nine years, uh, and then tried to see like over that nine years, which, which, which, what proportion of all predictions eventually happened.

It was, I think like over two thirds. And so sometimes it's just a question of timing more than just like, this is ludicrous. Well, that's what I like about your prediction is that I think it's quite easy to make predictions that, uh, you're gonna nail like 90% correct. You just kind of predict roughly what just happened. And then, oh yeah, it's gonna make more money next year.

You know, it's like, okay, that's not, it's not very interesting. So yeah, I find it interesting. You've got the hit rate of like 50. Yeah. Yeah. Yeah. So is it like, how is it difficult to, to come up with the predictions? Are there some that you, you kind of decided against, I don't know, too wild that, that you didn't end up making this year? Yeah.

Um, there's one I couldn't quite formulate around, um, around, uh, one is like cybersecurity and like, can AI somehow disrupt, um, encryption on the internet? Um, as it might be, uh, as that might be accelerated by the fact that AI solves math. So I, I don't quite know how to, um, yeah, I basically just ran out of time for like thinking about how to formulate it properly to be able to defend it.

Can I, can I ask you, so you, you speak to all kinds of people and we're a little bit in the AI engineer kind of bubble. Like what, what are the sort of things that people ask you about? Cause you've got different parts of the report about even going into politics and things like that. What do, what do you hear people care caring about?

Um, there's a segment of people who really care about what we discussed, like the, the RSI, like rapidly kind of accelerating progress. Like, what does that mean for the rest of us? Basically. Um, there's, there's some who really, you know, who, um, you know, are just like increasingly, uh, uh, pessimistic over like software companies still because, um, you know, now, like if you're gonna believe that RSI is coming and basically the, the quality of the software you can produce or the work you can produce is like really limited by what model you have and how much money you're willing to spend on it.

Then, um, there's certain people that will just like organizations that will accelerate way faster than others. And if you're not part of that, like what do you do? Um, there's always like chatter, you know, every two weeks or something. And I think now we're in one of those negative cycles around, like, is there too much compute circular financing? Like, how is it financed? Like, is the debt going to be able to be serviced? And, you know, there's a bloodbath in Neo clouds in the last like two, three days, because we're just in that part of the cycle.

Um, um, you know, it's always, always debate around like, will, will, will the labs monetize? How will they monetize? Or these like funky financing structures with multiple stages of different prices all within a couple minutes of each other? Or is that like a sign of, um, I don't know, just like rabid demand and not very much thinking? Um, how about normies? I have to ask you, Nathan, how about like just regular folks on the street and how they view AI? Do you track that instead of AI? Do you talk to them? How do you see that opinion shapes? So we did talk about the change in the public opinion about water use, et cetera.

Uh, what about the kind of the use case for AI? What do you see there? What do you track there? The rise of personal agents is now upon us, Glockbots and Mules and all of those. Do you consider those in the report? Yeah, but that's still like in the small minority of tech people who know, who know what it is. Um, you know, I, I would, I would still bet that like 90% of people are using like a model to maybe at best, like read their email or answer to their email or, you know, ask questions in the same way they would ask Google, but they probably never click the work button.

Like, what does that even mean? Like, do I want to open a consumer app and like work? Yeah. Not really. You know, and most people don't even know you can swipe left, open codex, when you're on your machine and like talk to your machine and make it do work for you. So like, I think that's so far away, which is one of the reasons why like I've just started saying, like, I think these big labs should open like a genius bar or like an actual physical store and just like start walking, you know, regular people through the magical things you can do with AI.

Right. That, this is the kind of also the geek squad genius bar comment, right? Like, exactly. They need handholding and imagination unlock. Um, yeah, that's what we're trying to do on the show, though. The show does cater to engineers. And so if you are still listening to this, uh, you are in the very few, I don't know if you saw the corporate as well about the talks about this as well.

Yeah, exactly. When he segments the world. Exactly. It's, it's, it's, we're still very much, very much like that. Um, and there's an argument that like that gap, that gap expands even further if like capabilities get better and RSI happens and a lot of stuff. I mean, you can see it a bit in like the, um, you know, the ramp data that we show with the 70 ,000 us customers that they have and tracking how much money they spend on models.

It's like, you know, the average spends like 12 bucks per employee per month. And the top 1% is like 7 ,500 roughly speaking. So it's like 580 times difference. Between the potential permanent underclass. If If this continues, yes. And the top spenders, the billion token ones. Nathan, thank you so much for joining. The thing that I do want to ask you, because you've been doing this for a long time, again, from AI ML days to then generative AI to very small context windows, like huge ones.

We've been doing this for three years and it feels like an incredible time. You've been doing this for much, much longer. Your personal beliefs and whether or not your PDOOM stuff. I don't like the PDOOM nomenclature, but about the bubble stuff. I would love to just hear from you. How do you see your views change throughout these years? If you can talk about this for a little bit briefly.

Yeah, I think the major... Also, my PDOOM is very small. I'm not a big believer in X -Risk.

I do think along that angle that cybersecurity is very precarious and should be taken insanely seriously. But I think that's different than existential risk of destroying humanity.

I think the major change I've had in the last two, three years is I generally believe that you use the right model for the task in the same way that you don't drive your Ferrari to go to a coffee shop or maybe you do to show off or something. But it's not the most reasonable thing to do. Just walk.

Yeah. Not all my emails, but the ones where you have to think about it. Why would you take the risk using a shittier system? Or potentially shittier system? So yeah, there's that. And then will we ever make money on Frontier Intelligence? And I think it's starting to show that we will because the value of the tasks that they're doing is increasingly high.

And just the more people from all sorts of different parts of the technology world who come into a world that's going to be a lot of money. And so I think that's really important to working in AI, bringing all of their knowledge and optimizations to the fore. And just human ingenuity plus lots of AI equals more progress. So I'm like quite, quite bullish on like margins improving.

And, you know, I don't know what anthropic is adjusting when they report their adjusted numbers, but you know, it's probably not like community adjusted EBITDA like we work. Like I think it's a serious company. So, um, um, so I'm, I've changed my mind on, on, on that. Like, I think you can really make money on frontier AI. And I'm increasingly convinced that it's the open source versus like frontier labs.

It's like, um, iOS versus Android. It's sort of cool for a while to play around with your Android and mod everything. And some people want to do that. But at some point you just want product that auto updates is like reliable, connects with everything and it looks nice.

So, all right. Those are great stuff, uh, folks, Nathan Benius. Thank you so much for joining us on Thursday. I will always welcome you back after the next one. Are you planning to keep going? Yeah, yeah, yeah. Yeah. Well, we'll keep going. I mean, you know, if AGI 27 happens, I'll be on the 10th year anniversary and then I might be able to stop, but, uh, all right.

Yeah. In the meantime, you can find everything online. It's all free at state of that. It's a state of that AI. Yes. It was a great URL. Nathan, thank you so much for joining us. Yeah. Thanks guys. Bye. Bye. Um, that was super cool. I don't, we don't often, we don't often get somebody here that does this longer than we do. Uh, and LDJ joined us as well.

Uh, this was great. Nathan, uh, Peter, thank you so much. What's your take on. Um, I, the, the most, the, the very cool thing about this is that there's obviously, uh, I find this really helpful to keep up with this stuff every week, but there's something really valuable. I think that to just look back at the year and you're just like, Oh, I completely forgotten about this.

Or I didn't actually realize there were papers there that maybe didn't quite make the splash this week. So we skipped it. Cause you know, there's too much to cover that actually they turn out to be quite important. So I really value his work and I've been reading his stuff for years now. I can't remember when it was the first one that I read, but it's just so useful to just look at it and, and just read the stuff.

So I really recommend just book, I don't know, an hour, two hours in your diary and just scroll for this. You don't have to read every single thing, but just scroll for this. I know it's like 250 slides. You can do it in an hour quite easily. Don't, don't, don't ask your agent to read this. Don't ask your agent to read this. Just read it, read it, put some effort in.

No, it's worth it. Nistan, how's your a hundred K city going? Are you still AI build? And Peter, what new and interesting did you start using or, or building this week? Just whoever wants to jump in, let's just chat. Hell yeah.

There's a, I got, I got some, some, some crazy stuff coming up for a release.

It's yeah. The entire decompilation of games and stuff is enabling full out simulations. Can we talk about this? So LJ, we mentioned this like last time that people are putting things in other games and now there's a whole decompilation thing happening. Somebody told me somebody decompiled or reverse engineer the airdrop protocol. What the fuck? Guys, it is crazy.

It's like a Renaissance happening right now, right? Yes. All right. Here's, here's what's happened. The entire internet, the last two months has just gone completely nuts by using like the NSA Ghidra protocol, which allows you to like reverse engineer any piece of software. And then there was also the law in 1996 from the Phoenix bios or ABC, I might be wrong on that.

That allows you to actually provide like visually functional pieces of software, as long as it does not violate any trademark. So you see a lot of these things called photocraft, gridcraft and stuff, which are like Photoshop or Excel builds. Clean room builds. Yes. Yes. So those are mainly done in Rust. But yeah, it's a complete renaissance in apps.

It's going to be a very interesting next three months as entire countries and economies figure out what kind of office suite do they need to run. And the very interesting thing is that actual professionals and creatives tend to like the Vive coded versions more because they're so much more portable. And they're fun. Yeah. And they're fun.

Again, they're not perfect. For example, in Photoshop, the reviews were that the magic wand tool is not very good because it used like an older machine learning model to remove the background. That's going to just be fixed in the next like two weeks. The second he turns his slop cannon towards that issue, that issue is solved. Yes. That issue is solved.

Yeah. They're just, they're throwing entire gauntlet prompts. And they're, I've seen entire apps, the entire Adobe suite, InDesign, After Effects, Lightroom, all just running in the browser, very performant too. And you can run them without any telemetry or crap or spyware. Without Adobe paying also a bunch of money to Adobe. Listen, I do have to pause you because LDJ's bar is breaking news.

Let's go. Hey, I breaking news. Coming at you only on Thursday.

All right, folks, we have quite exciting breaking news. LDJ, you want to take this one? Sure. I can. So, yeah, I do feel like overall these past few months, we've had a bit of a renaissance of Western American open models. And Meta just announced literally within the past 20 minutes that MuseSpark Open is released, which I presume is maybe like partly censored version of MuseSpark.

It's 816 billion total parameters, 42 billion active parameters. So this is around three times smaller in total parameter size than Kimmy K3, while being approximately half of the active parameters of Kimmy K3. So you could expect it to be a lot more practical to run a lot faster. I would say probably in terms of parameter count of active and totals, probably most fair to compare it to maybe the full GLM 5 .3.

And I think it is maybe beating GLM 5 .3, though, or at least trading blows with it in the benchmarks. So I guess we'll have to scroll down or kind of wait for people to benchmark things. Yeah, let's take a look. Let's take a look. They're comparing themselves. MuseSpark Open to Inkling, to Nematron, to Beam, to Mistral, and only then to QN and GLM.

Right. So like they're starting with, is it Inkling? It's thinking machines. Oh, yeah. Yes. Okay. So look, they're literally saying, hey, we're comparing ourselves to American counterparts first, then to European counterparts. Only then, QN and GLM and Kimmy. Yeah. I think what's interesting, if you scroll down into the coding area and compare it to the Chinese models.

So we've got the terminal bench 2 .1. So MuseSpark has 76 .4. Then QN 3 .8 has 82. And GLM 5 .3 has 83 .9, Kimmy 85. So it's like 10 points behind. Well, not quite 10 points, but like 7 and 8 points behind. Then DeepSwee as well, basically behind GLM and Kimmy K3, which, by the way, been out for what, a month, two, something like that? Yeah.

So, yeah, definitely better than the U .S. models, but not necessarily at coding versus the top Chinese models yet. Yep. It doesn't look like they were going to break SOTA. Oh, somebody says, yeah, but they're like 3x the size, right? So GLM and all those are like in the billions of tokens. I think GLM, not so much, right? Isn't it like about a simple size? Yeah, it's quite large here.

Let me find the exact parameter count real quick here. Because Flash is around 300 billion parameters. Yeah, okay. So total parameters for GLM 5 .3, the not Flash version, which they're showing the not Flash version in that chart. It's 753 billion total parameters. Oh, so that's comparable. Yeah, and 40 billion active parameters. So almost, yeah, very, very similar.

Very similar. And gets bitten. Well, GLM is better at coding, like quite a bit. Yeah. Quite a few points. So, yeah, I think it's... Well, what do you guys think? Is this... What's the relationship to Spark, Muse Spark 1 .123? Any relation? Oh, that's... I'm assuming that this is like a cleanup version to some extent. Yeah. Yeah. But they're not talking about this.

But yeah, Muse Spark from 1 .1 to 1 .3 did like quite heavy jumps.

And this one is a multimodal text, image, video, and audio output is only text. With 1 million at the inference context window. And a great vocabulary size.

This is super cool. Lama is back, essentially, is what we're saying, right? This is like the first release from Meta for a long time that's in an LLM space. Nistan? Thoughts? Excitements? I'm just going to... I just need to try it. The weights are available to download. The model is available in BF16 and F... It's pretty big. ...and VFP4.

To run on your system, the model requires a GPU with at least 700 gigs of VRAM and VFP4 and 2 terabytes for BF16. It also requires an inference deployment framework, supported frameworks, SGLang, VLM, Hakenface, and Lama CPP. We're definitely going to chat with our inference folks to try and provide this inference for you. Because we do believe this is a good model.

Yeah, yeah, yeah, yeah. It's... Especially for computer use, I think this will be pretty key right now. I'm just reading through the license. It just asks that anything you do with it, it should say, built with Muse Spark Open in the interface of each product or service that uses the software or any derivatives. That's the main license. And everything else is like an open usage policy so no violence and drugs and stuff.

Or not impersonating and misinfo. Yeah. Otherwise, it looks close to just like an open license. There's no restrictions on like how much money or anything like that. This is... Yeah. Oh, no, there is a restriction. Where the third -party outputs of such models, models of service, and the aggregate gross revenue of the licensee and its affiliates exceed $20 million US dollars or equivalent during any consecutive 12 -month period.

So as long as you make less than $20 million a year, you can just offer it. Yeah. And then probably a license. Yeah. Those are the things. But this is amazing because it will be great for generating data and stuff. Yeah, I'm pretty sure companies will offer it. It's probably one of the best computer use ones. That is so cool. Let's go. I wish I had my arrow button, folks.

Llama is so back. Llama is back. The promises of Space Onk where they release a new Grok, the old Grok will be open source. That doesn't often come to fruition. I don't remember the last time Grok was open source. Zak and Alex Wang did say that the open source is back after the manifesto and everything. And so now they're... After they're releasing Muse...

So two huge news from Meta this week, right? They released Muse Wonder and Wonderful. And that's... Folks are split on that a little bit. And now they released the open source version of Spark and Muse Spark opened. 816 billion parameter, 42 active billion parameters. We don't have the club buttons here. Also the applause. But like every lab that releases stuff with the license that's open, we applause.

And this was our attempt to adjust the conversation. The breaking news game. There was a bunch of other stuff this week that we didn't get to, including the voice. So Grok voice released and then Tinyfish as well. So let's see what else is interesting here. Yeah, I think it's pretty much it. We've covered most of it besides the... Oh, and multimodal JEV.

Didn't get to. I really wanted to. JEV is now multimodal. The original JEV, not the 700 open source or so clones, including the one from Microsoft. It's called Decision1. Including the one from OpenAI. It's called OpenAI Decision API. JEV, the original JEV from TypeSafe AI that we told you about, what, a month ago? That's it? That's only a month since JEV came out? Yeah.

Pretty much literally a month ago. JEV is now multimodal. So you can look at images and return bounding boxes. We also told you about Gemini Argon, which is Gemini 4. And we had both folks here, Genevieve and Omar Sanciro from the developer experience team at DeepMind to tell you about these models as well. Coming to you live from New York, AI engineer, the conference had a lot of folks who listened to Thursday.

And a lot of talk about the type of stuff that we struggle with, including attention, including how to run multiple agents, including when to sleep and how to sleep when you're just about to send something to an agent and it can reply and wait for you the whole night. This is also a part of the conversation as well. Well, signing off from here.

If you missed any part of the show, the show comes to any podcast app that you use, Apple Podcasts, Spotify, etc. And on YouTube, including shorts. If you haven't subscribed, please like and subscribe. And if you have, help us on other platforms as well. It really does help for exposure. Wolfram, go ahead. Wolfram, go ahead. I just wanted to say the news are coming so fast, but it is now the time where it is the easiest to actually implement the stuff.

When we started with the show and I saw we were talking about Croc voice, I told my agent, put this in my voice app. And it's now running on my phone already. I can talk to Amy with the Croc voice. And it would be cool if the audience would also tell us next time what of the last show did you implement yourself? If you have your agents, what is the stuff you are actually using? What about the things we reported on are you using now? This is super interesting to us, not just reporting on the news, but actually the uses of the news items, what our audience is doing.

That would be very interesting to hear. Yeah, please send and comment and tag us on social media. It's going to be great. With that, we're signing off, folks. Thank you so much. Peter Gosteve, Wolfram, Rameen, Wolf Nistum, Tahira, LDJ, Yama Pelagos here before, and our three guests, Nathan, Benyash, Genevieve, and Omar Sansevro from DeepMind.

And all of you for tuning in, for listening, for sharing, for liking, for sending, subscribes, and sharing with your friends. It really means a lot to us, especially if you come up and tell us that you're a listener. Thank you so much, and we'll see you here next week. Bye -bye, everyone.