The AI Race Has a Leaderboard | Arena CEO Anastasios Angelopoulos

0:00

Anthropic revenue has been booming, booming, booming on the API. But to some extent, it's also easy come, easy go. If you are spending that much money that quickly, there's no chance that you're not able to switch vendors. And so if an open source model comes by and it is able to do 90% of the work at 10% of the cost, which is totally feasible that that might happen over the next couple of years, I think that's a pretty structural problem.

0:23

People are assuming a degree of lock in the APIs that I don't think is necessarily gonna exist. We work with labs to help them improve their models. The product that we give to them is basically an analysis and insights product called an evaluation. In the process of developing models, Laz will come to us and say, hey, we have like 50 checkpoints this week.

0:40

How good or bad are these at different types of tasks? Is this good at front end? Is this good at full stack? Is it bad on tool calls? We take those checkpoints, we evaluate them on our data, which represents real usage, and then we hand those results back to them as an insights dashboard. And so this is the product that scaled us to over 100 million in revenue.

0:57

This has proven so useful to labs. How can we take an arena and put it within every business in America and every business globally?

1:16

Welcome to Grit. I'm Juven, partner at Kleiner Perkins, a show where we go beyond the highlight reel and explore the personal and professional challenges of building history-making companies. Today on Grit, I'm talking with Anastasios Angelopoulos, co-founder and CEO of Arena, the platform the entire AI industry watches to answer one question, which model is actually best? Anastasios went from a Berkeley statistician to building Chatbot Arena as a research side project to running a company that hit $100 million in annualized revenue this June at a $1.7 billion valuation.

1:47

My partner, Mamoun Hamid, joins me as guest host. You were born here. I was born here. Yeah, so LA kid. LA kid. LA's the best place to grow up, you know? I was awesome. A lot of art, a lot of music. Yeah. I didn't think I was going to be in tech. Well, the first time we met was actually on the Berkeley campus. That's right. In a... I don't know if I should call it charming or depressing little rectangular box.

2:14

Yes. With five people. The, the like, uh, yeah, we were like in the basement of soda hall and yeah, that's what we'd like pitched you guys. That's where you pitched us. It was awesome. Me, Yonway, Lynn, and then. Yeah. A couple of us. Yeah. Lee Marie and Ilya, I think was it right? Bucky too. Bucky. Buck. Um, why did you meet at Berkeley, not Stanford? Well, he was finishing off his PhD at Berkeley.

2:37

Dude, this was never meant to be a company. We were like, we started this off as a student project like three years ago at Berkeley because at the time we didn't know how to evaluate. It was like kind of a side project of a side project. I, meanwhile, was doing my PhD in theoretical ML. I was like proving theorems in a basement at the time.

2:56

And really I was happy doing that. And that's also how my, my wife, because we were both theorists and that's how we both also got, you know, our jobs at Stanford and that's, um, and then we, you know, in the meantime, we started arena and it just grew and grew and grew into this. project. And then I remember what happened was that at the beginning was just open AI.

3:16

And then Google came in the mix, Anthropic came in the mix, and it just exploded because people wanted to compare. People wanted to understand the differences. And they were kind of surprising because you would test them on like MMLU or whatever multiple choice questions. And it would be like, the model that does well on the test is not the same model that I like to use.

3:33

And that's doing well in the real world. That's because they all trained at the test. Um, and so we had this philosophy that it's really about reality. Reality is the only judge that you can actually trust. So we built arena, which is this platform where people go to use AI in the real world. And by using it, they're giving us the feedback that we need to make these leaderboards, which you'll see online.

3:55

Uh, and it just exploded from there. Um, but we met very early in the days back when we were still basement dwellers. Yeah. And do you think it was, at that time, you know, this is a year and a half ago or so now, you had OpenAI, Anthropic, Google, and DeepSeek was relatively new.

4:18

Was there a moment around DeepSeek that made you think, hey, like, I need to go start a company because there'll be not just three models, but there'll be You know, not infinite, but lots of models. I wouldn't say it's deep, deep, deep seek specifically, but I, you know, throughout the history of arena, there've been all these times when people kind of be being like, do you think that they're going to consolidate? Do you think that the models are going to consolidate? People still ask me that question.

4:43

Like once a week I get, oh, but what if the models consolidate? And it's always the opposite. People are just not understanding that AI, because it transforms every industry is going to lead to like many flowers blooming. Um, and so, yeah, for sure that that was a trend that we didn't fully appreciate at the time, but it has proven to be really helpful for our business.

5:07

Yeah. Because at the time it was still the leaderboard for those, the obvious company. Yeah. And we had like, you know, 20 models, you know, we started with like eight models. Yeah. How many models do you have today? 500. And when you say have, can you explain what have means? Oh, we just like rank them on the leaderboard. Yep. Really? We've started deprecating some of the older models.

5:29

Yep. Um, but you know, what's happening now is that every week there'll be like 10 releases. That's insane. It's insane. Today, new release from Quen, Quen Max. And there's more coming from Quen.

5:44

The dragon has awoken. Yeah. Yeah. You've become somewhat of an expert on some of these open source models. For sure. And like, I think you guys are sort of leading the charge on, you know, almost like, not actually anointing, but like, really understanding deeply like what's happening with Kimmy and Quinn and others. Yeah.

6:11

Where does that expertise within the company come from? Well, it all comes from the users. The point of Arena is that real users and the utility that these models provide to real users should be at the center of the AI conversation. There shouldn't be some random benchmark that's made by some dude that thinks they're smart. Some researcher out there that's like, oh, how does AI do on this weird prediction task? And I think that long-running finance workflows of XYZ type are the thing that determines AI progress.

6:44

It's like, dude, Get your head out of your ass. You're the cow sheep for AI. We're the cow sheep for AI. There you go.

6:54

I'm serious though. People make all these benchmarks and they're so weird. Why are they making those benchmarks? Because benchmarks attract attention and people don't know how AI models, people don't know how to rank them. People don't know what performs best to their task. And so they're creating all of these like wacky ways of measuring them.

7:19

We were all like for the last year on Twitter looking at pelicans riding bicycles because people were making benchmarks around SVGs. You know what I'm talking about, right? The pelican riding the bicycle? No. Well, there was this common benchmark that was made, you know, there's a particular individual who likes benchmarking things and made a pelican riding a bicycle.

7:40

And then everyone on Twitter is looking at this. Every time the new model comes out, oh, where's the pelican? It's like, let's show you the pelican. cares about the pelican dog i'm not kidding you i mean it sounds ridiculous because you guys haven't heard about it but it's like yeah i've not no you should you should be i'm so happy to hear that yeah it reminds me of like when you were in the graphic like graphics industry in i don't know circa 2000 everyone had like the spinning um like video of venus de milo that was that was a test Yeah.

8:10

So this is the Pelican riding the bicycle. This is how far we've gotten is from mythology to. Yeah. A hundred percent. Now I think that benchmark has been saturated because it's made us wait. It's the training data. Now people know how to generate SVGs and Pelicans riding bicycles, but people will create all sorts of like random little tests and toys examples, but what it doesn't speak to is real user value.

8:33

What happens when you go try to use them all for your actual workflow. You're trying to code with it. You're trying to do your finance workflows on it, spreadsheets about which companies to fund and which companies not to fund, which you guys do every day. That kind of stuff, there should be benchmarks that measure it. And it can't come from just somebody writing down a set of questions.

8:51

It needs to come from you. You need to be the user and you need to tell me, hey, this is good, this is bad. And it's not because you are trying to benchmark, but because you're in the natural flow of work and you're getting your job done. That's what Arena is all about. It's measuring utility by providing utility. The point is that the user incentive and the platform incentive are perfectly aligned.

9:13

Because we're trying to measure utility and the only way to measure utility is to give utility to people. So it's like Cal she meets Reddit where you can do some hierarchy upvoting to make sure that the users are the ones that are actually deciding where what's good or not. Yeah. And I think, um, a different way of thinking about it would be like a, uh, model agnostic chat GPT, um, or a model agnostic cloud cowork because, uh, the insight is that the user of this product should never feel like they have to give feedback.

9:48

they shouldn't feel like they have to upvote, downvote. And a lot of times the feedback that they give is not of that nature. It would be implicit conversational feedback. You know, I like this. Can you keep going? Or, you know, what a terrible response, you know, undo it and give me something else. Anytime you're doing that, that's explicit natural language textual feedback that we can use for our rankings.

10:13

And then there's implicit natural language feedback. You ask a question, you try to get your response, it doesn't work. So you try to ask it another way. Almost like in the days of search, you have query reformulation. Like crew there's query or formulation also for cowork type products where you're just trying to get like, how many times have you had the experience where it's like, I'm just trying to do X with my AI and I keep prompting it to do X and it's doing it wrong.

10:40

Or it's doing X plus Y and the Y I didn't ask for it. It's like, I've been trying to like, you know, figure out where to put a couch in my living room. And it's like, I want the painting here and I want the couch there. Can you do that? And it's like, sure. And then it's like, I'll do that and I'll also modify the window. What are you doing? And so I'll try it again.

10:59

No, do it again, but don't modify. And so that is a piece of feedback. Every interaction is a piece of feedback. Download button, that's a piece of feedback. And so if you take all this together, you can learn about which models provide utility to users. You know, which ones are they actually getting their job done? Where they're not having to reformulate their query 50 times.

11:18

Which ones are they downloading the files and then actually using them? Which PRs are they merging into their code base? All this stuff we measure, and it becomes the highest quality signal of model performance on the market.

11:30

What kind of people use Arena on a regular basis? So this is very interesting. What would you guess? Actually, you don't know our business. What would you guess about Arena users? Engineers? Why do you guess engineers? Hmm.

11:48

Like, uh, assuming maybe that if the breakaway application is coding and having spent a bunch of time with engineers over the last year, they seem to care absolutely more than anyone being on the frontier, depending on what coding model they want to use. Yep. That's why I guess. Totally. And you're right. It tends to be more knowledge worker professionals that are using Arena.

12:11

28% of them are software engineers. By the way, at the scale of tens of billions of users, this is a lot of freaking software engineers. Yeah. So 28% software engineers, 17% scientists, 15% finance, 6% legal, 6% medical.

12:27

It just follows the categories that are, like you should expect as these new categories. Yeah. Like if Harvey and Sierra ever catch up to Cursor, you would expect those numbers to probably start going up in Arena. For sure. It's no joke. It's no joke. No joke. So it's not just a leaderboard. It's not just a leaderboard. There's this massive community consumer following that powers all the data and analytics that we show.

12:57

And I'm curious, like, okay, let's just assume that you have more ground truth on model quality and progress than maybe anybody in the world. What has surprised you over the last year on how the leaderboards have changed? You know, the easy answer would be that the pace of development is just extraordinary. I think it's faster than any of us could have predicted.

13:19

But I think the number one thing that's starting to happen now is there's a phase shift between closed and open models. And I think that the implications of this are not well understood by our industry. The fact that Kimmy was number one on our web development arena, was huge national, international news. I don't know if you saw it online.

13:44

But the fact that Kimmy is number one on web development, it's the first time really that we're seeing a violation of the distillation narrative. The distillation narrative being that the only reason why China is keeping up is because they're distilling American intelligence. But how can that be true if they're number one and America's number two?

14:02

In a very important category, by the way, most software developers are doing web development. So it's not a nothing category. That's where most of the economic value is coming from. And so what that means is that yes, of course they may be distilling, but distillation is not the only part of the story. There's something else that they're doing to actually to push the frontier beyond American labs.

14:23

And so this is what we're getting into. know i don't think americans are fully appreciating the intelligence and creativity that's coming from chinese scientists but that is being shown it's being proven as we speak and quen is coming out you know quen just came out with their max model it's a frontier model um and there's going to be many more of these and so i think that we're we're starting to get into the flippening The flippening of the American and Chinese.

14:51

And so what happens when this actually comes to fruition? If we actually get a Chinese model that's as dominant as Fable has been over the past couple months in the sense that it's topping every leaderboard. You know, you should tell me what's happening to the capital markets. I think it's not going to be good for us. Um, but the other aspect of it is that there is a silver lining, which is that I think that there is room for a trillion dollar American company based on open source.

15:21

Mm-hmm. I think everybody would probably agree with that. I don't know. Would they? Well, I mean, I think everyone here would like to agree with that. Like, I'm curious if the leaderboard suggests that there's anything even close. Nope. Nothing. Even close? No. Thinking machine's the closest. If you look at open source only models, they would be number 10.

15:46

Wow. And then there's nine Chinese above them.

15:50

Nine. And actually there's probably more now. They're probably not even top 10 because of Quenna. And so it's very, very, very competitive. And we are way, way the hell behind.

16:04

So what gives, like, what are we missing here as, you know, the East versus the West? Is it,

16:19

not putting enough resources behind distillation, but also other novel techniques that allow Kimmy and Quinn to produce these frontier models. Yeah. I think, you know, there's some aspects of this that are incidental because Lama was a pretty good model at the beginning and then they killed their program. Yeah. They like completely imploded, um, because of some issues that they were having internally and restarted.

16:42

And maybe if they get back into the open source game, you know, things will work out.

16:48

Um, but we seem far from that. My, my take is that there's business model issues that have not yet been resolved or that are starting to be, we're starting to see a path for those companies monetizing now, but that wasn't clear earlier because what happens, I'm going to spend $5 billion training a frontier model, and then I'm just going to give it away.

17:07

Sounds stupid, right? It sounds like a stupid idea. And so not a lot of American companies did this and the Chinese can do this because they have different corporate dynamics in China, which you guys are very familiar with. But in the US, it's like, well, how am I going to make the money back that I used to train this model? And there's a couple of ways that people are approaching it now.

17:30

One way is to have licenses that are kind of open source and also kind of not. that have like revenue thresholds. Hey, if you use this in a product that's over $20 million in revenue, we're going to do a rev share. If you're an inference provider and you want to serve this model, then we're going to do a rev share. That's one way of going about it.

17:47

And then the other way of going about it, which I think is becoming more and more common with like Thinky and Reflection and Mistral and others, is to take the open source and then use it as a way to insert yourself into companies and then build basically an AI modernization motion around that. But to Mamoun's question, like let's take Thinky, for example, you're saying it's barely scratching the top 10, but it seems to have extremely competent.

18:14

Sorry, barely scratching top 10. Open source. Open source. Open source. It seems to have raised enough money to be dangerous and have all of the right people around the company. Like why, why hasn't it been able to break through the top 10 and open source in the benchmarks? I mean, listen, the talent game is very, very, very competitive. The model game is also competitive.

18:37

They're, they're starting from behind. Let's be honest. I mean, you saw they restructured the team like seven months ago. So they like blew everything up and then restarted. And then seven months later, they're at a model that's top 10 open source. I would say that's actually not bad. Yeah. It's like good, strong momentum. Um, but, but like a company like open air anthropic has built such a mature pipeline around hardware around data.

19:05

Um, you know, they, they've accrued advantages that they need to be overcome. There's no question that thinking machines is underdog.

19:13

This might be a silly question, but do you.

19:17

get pressure from these labs? Like, you've become very influential in, let's call it, engineers' perception of their product. Yeah. Pretty much more than anybody. No, they care. They must care a lot. They care a ton. How do they show you they care? Well, they care about the results. And so they're coming to us and asking, hey, can you co-release with us? Or, you know, what's my score gonna be and all that stuff.

19:45

So we work very carefully with them. The trust is a good asset for our company. But you have to recognize these people are smart, so they understand incentives. Their incentive is the same as our incentive, which is for the platform to be trusted and neutral.

20:01

If the platform's not trusted and neutral, it doesn't have value to anybody.

20:06

And so we've never really had a model for how to come to us and try to say, Hey, will you do this under the table?

20:14

They have too much on the line and they know that if they do that, that damages the platform forever. Yeah. Um, so we don't have issues like that. Yeah. Perhaps surprisingly, a lot of big personalities. You would think that they'd be like trying to jockey for sure. Yeah. But people, people want to play on a fair playing field right now. What's the biggest criticism of the platform? I'll tell you historically.

20:40

Arena started with battle mode. So you would get one pro you input one prompt, you get two responses, and then the user would choose which one is better or worse according to their preference. And we built an incentive so that they would vote their real preference. The reason being that the answer they vote for, they continue with.

20:59

which means that you shouldn't vote for a bad answer because it'll pull your context. I would say the biggest criticism of the platform historically has been human preference is really an incomplete signal of moral equality. Like, why should I care if humans prefer it? I care if it's verifiable. I care if the code's high quality. I care if it's factual.

21:17

And humans aren't, like, checking all the facts when they vote for answers. They might just vote for what they like. You might vote for something that confirms your pre-existing biases. And so what we've been doing is we've been going way, way beyond the battle mode. to address these concerns and to take advantage of like richer signals. So we've moved largely away from battles as the main signal on arena for this reason.

21:43

We incorporate human preference still, but the number one signal that we have now is task completion. Are you getting your job done? And you don't necessarily need to tell us. We can also measure it straight from the data. So it also fixes the gap between the stated and revealed preferences, you know, that we have. Task completion, steerability, hallucination rates, factuality.

22:01

We now rank factuality by using pipelines that we build that are automated, extracting all these factual claims and then shoveling them into a bunch of search models that go search the web to see if we can find any evidence to support or to disprove any claims that are made by these models. You'd be surprised. They're not all factual. Despite the way that they're trained, or maybe because of the way that they're trained, they're memorizing a bunch of lies on the internet.

22:28

which you can find. And so we've been trying to neutralize the concerns around human preference by going way beyond it. And so now all the rankings that we have on arena, um, go far beyond human preference. Yeah.

22:42

So you're not a leaderboard. You're not just as a, for a consumer, you are a model agnostic chat GPT or cloud. But as a company, as a business, you're a data company. Yeah, absolutely. And by a data company, we mean that the value in the products comes from an accrual of a massive amount of data, high quality data that we can collect through our various platforms.

23:11

You want to talk more about what you guys sell? Yeah, sure. So we work these days primarily with labs. and we work with labs to help them improve their models. The product that we give to them is basically an analysis and insights product called an evaluation. So the first thing to note is that the,

23:30

The numbers you see in the leaderboard, like the leaderboard that you see publicly never touches money. It's a hundred percent. We basically run it as a charity, uh, so that the world can see, of course, it's great marketing for everyone involved ourselves and others. Um, but we, uh, but, but like we valued all the models on there for free.

23:47

Now, where we do charge money is in the process of developing models. Laz will come to us and say, Hey, we have like 50 checkpoints this week. how good or bad are these different types of tasks? Is this good at front end? Is this good at full stack? You know, is it bad on tool calls? What can you tell us about where we need to improve? And we take those checkpoints, we evaluate them on our data, you know, which represents real usage.

24:11

And then we hand those results back to them as an insights dashboard. And so this is the product that scale is to over a hundred million in revenue. And a lot of the future of the business is taking, is saying, okay, this has proven so useful to the labs. How can we take an arena and put it within every business in America? Every business globally.

24:31

How can we have every business take advantage of their data in the same way we are to know which models are best for them, which models are best for their users, which models are best for their employees, and then how to save money. Because that's a big part of it is the trade-offs. People don't know how to make the trade-off between performance and cost.

24:45

They don't even know how to define performance. Right. Cause it's not like you just, it's just speed. You know, it's, it's not like just a, you know, a CPU or something that you can just hardware benchmarking. It's very, very mushy. Like, and you need to know whether people are getting their jobs done better. Uh, and whether they're getting more satisfaction, like CSAT type measures from the actual, you know, organic traces that are being generated within your company.

25:14

Uh, and this is sort of representing the future of the business. How quickly did you get to that a hundred million? Eight months.

25:24

How big is the company now? People wise? About 75, 80. And how big was it eight months ago?

25:32

Eight months ago, it was probably like 15 to 20. It's insane. It's completely insane. Absolutely insane. Yeah. Imagine what I felt like coming out of the basement where I met. That's what I was just thinking. Yeah. As a PhD student, just proving theorems. And then now I'm here running this company. It totally feels completely surreal. Surreal fast.

25:54

Surreal fast is surreal. Just like weird. Do you enjoy it? Yeah, I enjoy it. Which part of it? You know, I mean, obviously like it's so exciting to be able to participate in the creation of this technology. That's gotta be the number one. It's all of us. I mean, you guys included have to be just so excited to be like in the nexus of the most important thing that humanity has ever invented.

26:19

It's like gonna be bigger than electricity. Mm-hmm it's just gonna touch everything. It could be even result in like a new species guys. I'm not kidding. Like, I, I, I feel weird. Like even with those words coming out of my mouth. Yeah. It's like, yeah. You're like chant, chant the name three times and it appears. Do you know something that I don't know? Yeah, I do.

26:42

I do. You do? I know, I know what's happening inside the labs. They're, they're making aliens. Yeah. We've, we've had contact with RSI, RSI. RSI is here. No, I, you know, more than me. I don't know. Um, but yeah, I think that that's number one. It's just like, so cool to be part of this. Um, and then there's things about being a CEO that I love.

27:05

I would say I love working with people, a diverse array of people from all sorts of walks of life. know, as a, as an academic or as a PhD student, um, you interact with technical people, people who are experts at their craft, really honest people, sweet people that have no idea about how necessarily productionize something, but that are, uh, brilliant at what they do.

27:29

And I love those folks, but it's only like a tiny, tiny slice of the world. And now I work with BD people who everything is a negotiation and I work with, you know, uh, like lawyers and I work with investors and I work with, uh, you know, engineers who are like great at building things and taking something from an idea into a product that you actually love to use.

27:50

And I work with marketing people and all of the people that we have at arena are absolutely best in class. They've built generational companies before. Yeah. And so it's, it's really cool to be just. you know working with so many amazing people and then people have problems and that's my job as ceos deal with people's problems i'm sure you guys know you know you're really good at it you're actually you're exceptional at it from from the basement as a phd student to a year later running a company at this scale uh i think just from point a to point b i don't think i've seen a slope higher than yours That's so sweet.

28:28

I really appreciate that. You've seen a lot of slopes, so that's a high compliment. But I'll tell you, the people problems, of course, they take up more and more of the time as time goes on, because everything ultimately is a people problem. A company, you design the company, the company builds the products, right? So, I mean, there's some truth to that.

28:46

I love being part of the details, so I'm always going to be part of the details, but at the end of the day, you need to make sure the right people are in the right place, right time. and are pointed in the right direction and make things happen. And that they're not caught up with like dumb crap. Um, but I kind of love dealing with people's problems too.

29:02

And it's because, and you just gotta love people. You're like, oh, this person's having this problem because they are emotionally feeling this. And oh, how can I help them like fix this insecurity that they have or this like interpersonal, this argument that's happening because it's something stupid. How can we help resolve that? And I kind of relish that.

29:19

No, this person I don't like. Parts that I don't like about being a CEO, primarily that my calendar is just filled with meetings. I just have like 10 hours of meetings a day. I do 10 hours of meetings. I go home. I do whatever work I couldn't do in the meetings. And then I rinse and repeat. Meetings are not always pleasant. Sometimes you just want to be a little like,

29:42

I love doing technical work too. Um, so I try to carve out five to 10% of my time to do that, but you know, that's always last, last on my priority. Can you steel man, why you think this is more important than electricity? Like, I'd be very curious your argument for why. Okay. So why is it the most transformational technology that humans will ever invent? Obviously like electricity invent or ever have invented, ever have invented, ever have invented.

30:15

Um, obviously electricity is foundational, right? We can't have AI without electricity. So in some sense, like if you think about it hierarchically that way, um, electricity would be more important. Now the counter argument to this would be just in terms of the realized impact. That sort of the incremental value that AI would add would be greater than that of electricity.

30:35

And so why would that be? Um, AI has the potential of literally, um, creating, uh, intelligence on demand, which is something that humans have never seen before. Um, and could be influential in so far as it actually even deprecates humans.

30:58

They could literally make humans obsolete

31:03

in the sense that we're able to create an organism that can pursue any goal faster than we can, that can create productivity at a greater degree than we can, that can explore the universe more efficiently.

31:18

The impact of that, I think, sort of if you were to try to decouple everything else electricity can do versus everything AI can do,

31:28

that impact may be greater than electricity because it will just supersede everything that humanity has ever accomplished. That would be the argument. So to be clear about the thesis, it would be if you take everything that electricity has done and will do outside of AI and compare it to the impact of AI, that the latter will be a greater.

31:50

I'm curious, what's your reaction to that argument?

31:54

It's beautifully said, and I fully agree with you. There's nothing to disagree on. I think, you know, we've talked about this for the last few years, that this is the coming together of all the pieces of technology that have been created over the last 60 years, from the transistor and silicon to

32:18

firmware, software, operating systems, the internet, mobile cloud everything pulled together allows this moment to exist yeah right so it is the it is the coming together the culmination of everything that's happened in our industry ever since you know our firm was started 55 years ago um and it is greater than all those combined And I actually love the framing of like, if you took electricity and took to one side what electricity has done for humanity and put to the other side what AI will do for humanity and the way you said it, the latter will be greater than the former.

33:05

Yeah. No debate. No debate. No debate. Do you have a different opinion? No, I mean, I hope you're right because something's got to justify these valuations. So, you know. I think the real question is what happens to money.

33:19

Yeah. Will money be, so it'll, it's interesting because money there's, there's something that has made capitalism work so far, which is that capital and labor are coupled. Like you can earn money by through labor and that is no longer really going to happen. Right. So that's where this capitalism start stops working for people. And then you have to start thinking what's going to happen to people.

33:43

I think that's the fundamental question is. is the decoupling of labor from capital. Yeah. Very fundamental. That is really the crux of the matter for humanity is the decoupling of the two. Historically, the stock market revenue for companies, which is tied to the value of the stock market, was tied to the labor. Yeah. Input was labor. And as labor becomes detached from the creation of value,

34:12

the stock market capital value diverges from labor input totally right and that's really what we have to all grapple with as a society over the next decade or so which i think we have to flush this stuff out um There's just one stat that really jumps out is if you look at the overall market cap of the stock market, it's like I think the U.S.

34:40

stock market is like worth $100 trillion or something like that in that order, roughly. And it's sort of in line with the GDP of the world. And like 20, 30 years ago, the overall value of the stock market was way lower than the overall GDP of the world. Capital value has appreciated a faster rate than GDP. Yeah. And it's going to accelerate in that way.

35:11

And it's already actually, if you look at the U.S. stock market, a lot more of the value accrues to... The delta is like the U.S. GDP is like 22 trillion. The U.S. stock market is like 60 trillion. So it's like a 3X. Yeah. And the overall worldwide is like a 1X. So the U.S. is already at a 3X. capital value to overall GDP. Totally. And that's going to accelerate to a crazy degree.

35:35

Right. And so it's the separation of labor

35:41

from capital. And so I think that is what we all have to sort of grapple with as a society is how do we live in this world where more and more of the value of the world will accrue to potentially fewer companies. Totally. And also, I think the idea of how we assign values, another thing that we are going to need to think about, because the assignment of value comes from scarcity, and scarcity comes from the coupling of capital and labor.

36:11

The fact that you can only make X dollars because you're on a salary and therefore you have to spend that money carefully. And therefore value is a signal, like price is a signal of value, right? The whole point of like this, these economic theories is that of capitalism is that price is, is information. The price is the information about how valuable something is.

36:38

And that information is driven by scarcity because the sort of like decentralized market of consumers of whatever that product is, they go and they pay whatever it's worth to them as opposed to other goods. But what happens when those two things are decoupled? And I think there's, there's two ways of seeing it. One way is what happens to humans? How do humans assign value in a world where let's be honest, like there's going to be a big fraction of people that just can't make money.

37:06

And then there's another side of things, which is agents. What if agents can assign value? Do they have the right to assign value?

37:15

You know, like is value just something that human, like within the construct of a human, if you're making life better for a human, that's valuable. Is there a con or is there a construct under which you're making life better for an agent? And therefore an agent with a wallet can pay for that. We call that value and that the money that the agent is spending doesn't need to come from a human.

37:37

So can agents autonomously be earning value? And that's something we're going to need to decide as a society, is whether or not we're even going to allow the possibility for autonomous agents to directly assign value themselves.

37:52

Is all of this assuming that the models will continue to get better at the rate that they have? Yeah, models are harnesses.

38:00

Not necessarily even at the rate that they have. This might happen in 20 years. It might happen in two.

38:06

My guess is going to happen closer to two than 20. So I think we better start thinking about it now. Why is that your guess? It's just the rate of improvement hasn't decreased. If you look at the arena, like, you know, we're seeing at times bigger gaps between successive generations of models than we have in history still to this day.

38:29

You're saying that the newer models from Anthropic and OpenAI are taking bigger step function leaps and improvements in your leaderboard than like GPT 3.5 to four. Yeah, it's happened before. I mean, GPT 3.5 to four was another one of the biggest leaps in history. Um, but we are still seeing some of the biggest leaps in history to this day.

38:50

So for example, GPT image two versus 1.5 was I think the biggest gap that we have ever seen in history on arena. It was nearly a 100% win rate. And by the way, it's almost impossible to get people to agree on anything. Like it's almost impossible to get a large number of humans to vote the same way on any political issue. on, you know, basically anything at all, but a hundred percent of them are voting GPT image two better than every other model that we had ever tested at the time that it was released.

39:21

And we're seeing phenomenal like that's still happening. So yeah, I don't think, and RSI, you know, it's the buzzword of the day. If this happens, um, we'll absolutely be seeing acceleration. Question for you.

39:35

What are you long and what are you short? Hmm. I think that the API layer, people are assuming a degree of lock in the APIs that I don't think is necessarily gonna exist. Anthropic revenue has been booming, booming, booming on the API. But to some extent, it's also easy come, easy go. It's like, if you are spending that much money that quickly, There's no chance that you're not able to switch vendors.

40:00

And so if an open source model comes by and it is able to do 90% of the work at 10% of the cost, which is totally feasible that that might happen over the next couple of years. And that certain businesses might have all of their capabilities be able to be done by the mini light version of whatever the next Kimmy is. Well, I think that's a pretty structural problem.

40:20

And so what are people doing? What are labs doing to do this? You know, OpenAI Anthropic, OpenAI has this huge consumer business, which is kind of like going to be able to keep it afloat anyway. By the way, one thing I'm long is the consumer ads marketplace on OpenAI, which has obviously not been fully utilized to this day, but I think it remains one of the biggest opportunities in AI.

40:38

Because the question is, how are you going to generate hundreds of billions of dollars in revenue for them? right? Like annualized. And there's like only a few businesses that know how to do this. And it's like ads and great hardware businesses. And I actually don't know anything else. You probably know better than I. No, those are your, you know, take Google, you take Amazon, you take Nvidia.

41:03

Exactly. Right. Tesla. Tesla. Yeah. You know, consumer or great hardware. And like ads are hard. If he's right, let's just imagine like if open source,

41:14

delivers breaks out. Do you think that value goes up the stack? Like let's just say right now, so many of our application layer companies have very expensive costs to serve.

41:28

And if you got rid of the gross margin problem, these would be some of the most incredible businesses of all time, like ever. So I guess if open source can alleviate the gross margin problem because their cost to serve goes down by 80 to 90%, then like naturally does the logic follow that value continues to go up the stack? I think that's been our fundamental belief here that we're at maxis here because we believe that the models will continue to get better.

41:58

The open source models will be even better and the cost will come down. And we already see that the army of lawyers that AI lawyers or whatever we're going to call it at Harvey, is the use just continues to go through the roof. Or take any app layer company, there's an insatiable demand for the apps that get the job done. And you don't want the half job done.

42:27

You want all the particular needs of the legal industry met or the medical industry or the coding. So

42:39

yeah, fundamentally, like all the things we're talking about benefit the app layer companies. And that's always been sort of the, that's how it's always played out in every sort of cycle of the industry is that companies, big companies especially, buy from other companies that package up a really nice solution, you know, show up with the person who's going to help them implement it, get it up and running, getting it, getting the value delivered to the end user.

43:12

And that usually comes in the form of an app player company doing that for them. And so I think the tailwinds for these app player companies

43:21

are totally there. I know they were called rappers for a very long period of time. And I think that's, it's always the, you know, the very lazy way of, you know, identifying these companies or, uh, uh, describing these companies. Uh, but, but yeah, I'm, I'm in the camp of,

43:42

there's just a lot of value that will accrue to these app layer companies. Well, I think that this is a really interesting point. So then how do you view the labs? It's to your point, like, you know, especially as open source, unless like we all just locked down open source from China. Right. Which I don't know. Regulatory capture. Yeah. Do you think that happens? I think that there will be serious regulation on open models, unfortunately.

44:08

Yeah. Yeah. And, and I don't know exactly what shape that'll take. I think in part it's being shaped right now and that people like you guys probably have the power to like make sure that's done responsibly. Right. But I would predict that they, that we regulate those. There, there will be regulatory capture. Yeah. Maybe the silver lining is then we get a good open source model here.

44:27

Yeah, I think that that is the, that's the saving grace. And to be clear, I think the commercial incentives favor this anyway, because we are going to need American models so that people can build on a reliable supply chain and also one that doesn't have attack vectors from our, frankly, a foreign adversary. So we're going to absolutely need American open source.

44:47

Maybe this will accelerate it. Um, but I just hope the regulation doesn't slow things down too, because it could also slow things down. Would that be the worst thing for your business? Oh, for sure. Our business sucks. If we only have one model provider, you know, and if we have two, we're barely making it, we want 10, we want 50. Um, yeah, I think we could probably do with three or four.

45:15

Um, but you know, regardless of our incentives is bad for everyone. I mean, who, who wants an oligopoly? Yeah. Like the whole point of this like capitalist system is that we should try to encourage competition. And so now if we have the winners going to Washington and convincing politicians that we should create these regulations that are going to be harder for, you know, they're basically only incumbents are going to be able to satisfy them and make it harder for other upstarts to be able to increase adoption and get their models out there.

45:45

I mean, come on, like this is the most anti-competitive possible behavior.

45:52

Yeah. You know, um, I was listening to, um, Brett Taylor. We're at some thing and he was talking and like, I would consider Brett to be more in the know than like anybody. And like about as competent as competent, like whatever competent is, that's him. Like he is, I have so much admiration for him. And obviously he's on the board of like, he knows more than most of us.

46:17

Yeah. And whenever he talks about anything AI related, he's like, listen, I have no idea. Like when Brett talks, he's like, I genuinely have no clue. Because he's like, I've never seen anything like this. And he's like, I know more than anybody in this room. And I have no idea. And I'm like, you know, I kind of respect that. You know, I'm just like, it's just, we've never seen anything like this.

46:40

Yeah, we've never seen it. It's insane. Yeah, completely insane. Going back to the, like running your company for a second. Sure. How do you think about, like, if you did a hundred million in eight months, I don't know what the goal is in a year from now, but we don't either because we keep beating it a lot more. We keep setting it. We keep beating it.

47:00

How do you support the company? Like, how do you, um, I don't, um,

47:06

How do you support revenue growth that is going parabolic like that, like from a company perspective? Like how do you even build the scaffolding around a company that's growing that quickly? Yeah, I mean, the good thing is that I think AI has made it easier to run a very high revenue company with fewer people resources. So we don't need to scale people linearly with revenue.

47:34

Fortunately, we've avoided doing that entirely. I would think about it more like, how do we put the right people in the right place at the right time to make great products? And how do we attract the best people in the world towards a small team that can support huge amounts of revenue? Um, and of course we need to fill in to certain GNA things as we go.

47:53

You know, we probably need more lawyers because we have more contracts. The number of contracts and number of lawyers kind of scale together. Um, but it's not true of, of, of the raw revenue. Yeah. Or the product, you know, the product runs on its own. It's arena.ai. It's this website. It creates, has all these users. You know, of course we want to keep making the product better for them.

48:11

We want to keep refining the user base and, you know, making the users better, giving them more tools, um, making it, making it safer, uh, decreasing abuse for the platform and so on. So we want to hire for all these things. We have our enterprise play. We want to hire great infrastructure engineers for, but it's not like that's a revenue scaling problem.

48:30

Um, it's more of a product scaling problem. What's your worst day at the company so far? Man, worst day of the company.

48:39

Such a positive guy. I know. Well, it's easy to be positive when you recruit 100 million in eight months. No, there's some low periods. Yeah.

48:49

You know, at the beginning, in the early days,

48:56

it's so hard for me to talk about. There was this moment.

49:01

Uh, we were just, uh, you know, in the early days of trying to start the company, trying to get it off of the ground. And we were still supporting labs and open source and other academics sort of on an ad hoc basis, giving people data, helping people, you know, do their research. And, um,

49:21

we were about to raise our seed, which was of course stressful for me. I had no idea how to do it. Yon was helpful. You know, you of course support us in that time.

49:31

And, you know, very stressful period for me. And then my father-in-law passed away suddenly. He passed away of a, just a heart attack, I guess, in Serbia, my wife's Serbian. And so I kind of had to drop everything and go to Serbia. He was also her person. They were so, so close. And they're similar people. Like when I talk to her, I see him.

49:57

And I was close to him too. I really loved him. And so he died while literally while we were like in the, Hey, term sheets coming

50:06

stage of our seed. And so I had to call people like moon and say, Hey guys, like I need a few days. And then we went through the funeral and then two days after the funeral, I was on calls again,

50:18

like negotiating the deal. Right. Cause I needed to, because the company needed me to do that. And maybe I didn't a hundred percent need to at the time, but I thought I did at the time that people wouldn't wait. And that the deal was hot and that, and I didn't know what to do, but I knew that I needed to do what's right for the business. Cause there's people's livelihood relying on me.

50:38

And then after that, you know, I think it took a long time for my wife to forgive me.

50:45

for the fact that I had to go be there during that period. And, you know, she was mourning so deeply. And I would say that she cried every day for many months.

51:01

And then eventually it was every two days. Maybe six months later, every two days. And then three months after that, it was every four days. And now she cries every so often. She says, I remember my dad. And I'm starting to forget my dad.

51:16

I forget the sound of his voice. And I was, that's just so difficult.

51:24

Mamoun knows because he's been through a similar experience. Yeah.

51:28

And then on top of that, you know, a couple of weeks later, there was this awful moment where there was this paper that was released called Leaderboard Illusion. People don't know this about Leaderboard Illusion. You guys might've seen it. Not only is it filled with a bunch of sort of like, sort of lies, like lies with data, basically incorrect analysis about ARENA.

51:50

that sort of is you know it's this coalition mit stanford you know whatever so it looks like it's a reputable source but actually um when you like dig into the analyses that that i had about arena at the time it was about oh this company is favoring closed source models over open source models and blah blah blah from cohere of course go here was like number 70 at the time so you know you say what you want about the incentives we've always been fair to everybody And that was very difficult because it felt like a stab in the back from some people that could have been my future colleagues if I ever decided to go to Stanford.

52:24

And was another difficult experience on top of the sort of personal challenges that we had at the time. It was very hard for me and my wife. And it happened a few months before our wedding.

52:38

And so we thought that our wedding was going to be a sad time. We thought that we were not going to have a good day. Um, but you know, we ended up pulling, she ended up kind of like rallying for that day. And we both had an awesome time at our wedding. Thank God. Cause it's only something that you get once. Um, but there've been some very low points in arena startup life.

52:57

There's the lows are low and the highs are high.

53:02

I got goosebumps. Thank you for sharing that. Of course. Yeah. Can I ask you about, can I ask another question? Of course. Yeah, yeah, yeah, yeah. I'm an open book. When you, um, I thought the framing of how you said it took a while for her to forgive you for that. Was it her forgiving you or was it you forgiving yourself?

53:20

It was probably both, but she definitely needed to forgive me. Yeah. I think for her prioritizing the company. Yeah. Right. Right. I think it felt like, uh, a betrayal to her that I would prioritize in the way that I prioritize. Mm-hmm and probably she also had more clarity of thought and was a little further away from the fundraising details for me to say, Hey, maybe I get why you would do this meeting, but I don't get why I get why you would do this other meeting.

53:48

Mm-hmm And like not all the decisions I think made sense to her, probably because she was right and they didn't all make sense, to be honest. I don't know, for what it's worth, I got to see a very small, maybe like 0.01% of you during that time in terms of what came through to me. But the level of care and love that you showed towards her and your father-in-law was actually unique to hear from a founder CEO who's building something in AI, there's a level of humanity and

54:28

empathy and compassion that you had towards others that you actually don't see from founder CEOs. There's so much egocentricity. So it made me, and we'd just gotten to know each other, respect you so much as a, not as like human being, but like someone who is more than just a a

54:52

phd stanford professor founder in ai it was just like they're like and maybe that's why from point a to point b you are you have this slope because you you may be the full package i appreciate it no i think we really resonated together on that personal on that personal note that we've been through some similar difficult experiences and um Yeah.

55:14

For, for better, for worse, I have a big heart. Um, so I, you know, I appreciate that you value that.

55:25

Man. Thank you for sharing that. Of course. Um, what does the rest of this year look like for you and the company? Yeah, a lot of big things come in for Arena. Two big priorities, I would say. The first is agents, and the second is building out the enterprise side of our business.

55:43

So on agents, you know, we've been historically known as a platform for evaluating models. But when you get into agents, it's just such a more heterogeneous ecosystem. It's not just the model. It's also the harness. And the harness can be very, very difficult to manage. you know, evaluate. It can be a multi-component system. It includes a bunch of sub-models and sub-agents.

56:06

And it's something people don't know. You can't go to somebody today and say, hey, you know, which model, which harness combination is the best? But we can actually do that at Arena. And so what we do, if you go to arena.ai slash agent, we have agent mode where you can do co-working type tasks like cloud co-work type tasks on arena and browser.

56:25

You can connect your code base, you can do everything. And when you type something that will basically give the agent a computer and it will go in and it will

56:34

allow you to do, you know, open-world tasks. And as part of that, we can randomize both the orchestrator and the harness for the agent. So we can collect all of those interaction effects that make it challenging. The goal being that people shouldn't really have to think about what harness to use or what orchestrator to use. It should just be an automated system.

56:52

We can even do that via routing, which we've also built. We've been building routers for the last year. Routers are now hot. But we were in the riding game. So hot. I know. So hot. We've been in the riding game for quite a while. And then the second would be, how can we take these ideas, the sort of like self-improving data flywheel that we built on Arena that allows us to build the best routers in the world, and then copy paste them into every business.

57:15

Help businesses take better advantage of their organically occurring data so that they can build the better routers, avoid vendor lock and get AI sovereignty and so on. And so those are the two big priorities. Are you hiring towards both of those priorities? If people are listening, what are you hiring for? Uh, infrastructure engineers, AI engineers, um, those are probably the most valuable.

57:35

If you can come help us build out our agent product and our enterprise product, which requires really high availability, high uptime, you know, this routing functionality, data engineering, um, that's all really important machine learning research. You know, I come from a research background myself, so I know a lot of ML researchers, but if there's people out there that are interested in these fundamental problems around, um, how do we optimize basically? How do we measure? And then how do we optimize? Because you need to know what target, you know, basically you need to know the target in order to optimize it.

58:07

And that's what evaluations are all about. I view Arena as being the system of value. That's where I see us in the ecosystem, the like AI ecosystem. We can help developers, businesses, and the whole community define value through the real world.

58:27

And then once you have a value system, then you can hill climb it. And the whole game is about picking the right value system. Um, and so our, our value system is utility to real people. Uh, and so anybody who's passionate about that mission should come join us.

58:45

Thank you for doing this. Um, anything else for you? Last one for me, then if, when you hear the word grit, what do you think of? Oh, grit. I think of people that never give up. I think of people that day in and day out, they are coming in and they have their nose to the grindstone and there might be terrible things happening around them or they might be pulled in other directions, but they're never going to lose focus.

59:12

They're always going to pursue the goal and it's never going to be about themselves. They're never going to have any ego about it. Grit means somebody who

59:21

is always putting the greater mission ahead of themselves and is willing to do anything to get the job done. My, um, I was going through howling back on the company over the last couple of weeks on some stuff. And my fiance on the mirror of our bathroom wrote the man in the arena quote. Yeah. The full Teddy Roosevelt man in the arena quote, just literally in marker on the entire, on the entire, um, on the entire mirror.

59:52

And, um, yeah, it reminds me of what you said. Like sometimes you just got to be in it. Totally. Appreciate you doing this. Thank you so much. Amazing. Thanks for being here, guys.

1:00:04

That's it for now. If you liked the episode, please leave us a review or go back into the archives where we've done more than 200 episodes with some fantastic folks. This podcast is a Kleiner Perkins production, and I'm Jubin. Thanks for listening.