Smash Notes
Why Does 97 Percent Accuracy Matter on Loan Documents? · Training Data

Training Data

Box's Aaron Levie: On Reinventing Yourself in the AI Age and Enterprise Diffusion

Starting a company is hard. Reinventing your company for AI as a public company with quarterly earnings results is even harder. Aaron Levie has pulled off the transition with Box and offers hard-won advice for founders. The cofounder and CEO of Box argues the value isn't only in the model; it's in the bridge from a model's raw capability to the actual…

Notes

  1. Aaron Levie says a model wrapper is working out in the enterprise because organizations need a bridge from a model's capability to the workflow the enterprise… Full answer

  2. Aaron Levie says applying intelligence to real work will still be a huge software market, because cloud infrastructure and the software built on it each… Full answer

  3. Aaron Levie says the Box agent reads company files to answer questions, extract metadata, turn unstructured documents into structured data, and automate steps… Full answer

  4. Aaron Levie sets aside the top engineers who bring deep taste and incredible judgment and treat code as art as much as science. Full answer

  5. Aaron Levie says Box's Complex Work eval is a document-centered test of domain-specific work in life sciences, financial services, the public sector, and tech. Full answer

  6. Aaron Levie says the high degree of complexity in permissions, access controls, and data at Box tends to be the rub when training a model around one person's… Full answer

  7. Aaron Levie says 70 percent accuracy on a bank's 100-page loan documents versus 97 percent is a world of difference, and automating that work means hill… Full answer

  8. Aaron Levie says the number one way to get current on AI is a Twitter feed wired into the technology. Full answer

  9. Aaron Levie says any question about the Box business that was ever written down as unstructured data is already 100% in Box, and he thinks the data hygiene is… Full answer

Related notes

0:00

I want to send an emergency alert to like everybody who's like a sophomore or junior in college and just be like follow these 20 accounts on Twitter. Yeah, yeah. And also join Twitter because like this will just help your career. You're either like a year ahead or a year behind

Expand full transcriptCollapse transcript

The episode notes above make this conversation easy to skim.

0:00

I want to send an emergency alert to like everybody who's like a sophomore or junior in college and just be like follow these 20 accounts on Twitter. Yeah, yeah. And also join Twitter because like this will just help your career. You're either like a year ahead or a year behind simply based on your feed. And my feed is like so, so wired in.

0:17

I'll still talk to 20 year olds that are like, yeah, like, you know, I see some articles and I'm like, what do you mean? How do you see articles? Like, I don't even know what that means. Like, do you just like get lucky that somebody emailed you an article? You just have to be wired in. I enjoy it. It's a lot of fun. I play with everything.

0:51

Today, I'm excited to welcome Aaron Levie, founder and CEO of Box. Box is building a collaboration platform and content management platform for the enterprise. That has now taken on really new life with AI. And I'm excited to chat with you today about Box and AI, about your general thoughts on AI because you are such a thought leader and about how founders can reinvent themselves for this AI wave.

1:10

So thanks for joining us. Thanks for having me. I'm a big fan of the podcast. I religiously watch every episode and it's a great work. I am wondering, though, I know this is a different podcast, but when do you talk about root canals? Oh, my gosh. Is that? We're going to call in Doug Leone. Have we figured out how to weave that in yet or not? I mean, I'm happy to talk about different kind of pain and suffering that I've been through.

1:32

We should just do this pod with a live root canal. Let's do it. See how it goes. Actually, it would be like hot ones, but you get a root canal and you have to talk about your strategy while you're in a dentist chair. And Doug is just on the other end of it. Amazing. Amazing. Doug is so pleased with himself, by the way, right now. Okay. Has he seen every tweet? Oh, yeah.

1:52

Okay. Oh, yeah. He's very pleased with himself. Okay, good. Okay. Let's start with, I'm curious your take on this, application companies are the hottest. Neo Labs. Yes. Agree or disagree? You know, two years ago, I think it would have not made that much sense as like, what does that mean? But very clearly, I think this is what's playing out in the market.

2:10

And it's all working out mostly because of open source. But what's pretty amazing is right now, you know, the whole concept of being a sort of LLM wrapper or model wrapper is actually working out. Because what I think people underappreciated was that in the real world, in the enterprise, what you need is some bridge from the model's capability to the actual workflow that the enterprise has.

2:37

And that bridge, you basically, like probably a trillion dollars has been bet on basically one of two outcomes. Either that bridge is very kind of limited or that bridge is actually very vast and needs to be able to take on, you know, a lot of depth within organizations. And that bet basically looks like, are you only long, you know, kind of the model itself and super intelligence? Or are you long this sort of application tier? Or, you know, maybe previously would have been just pure Neo Lab.

3:07

But I think it's very clearly playing out that actually there's a lot of gap between the model and the workflow. And as you bridge that gap, over time, you get to a point where you realize, oh, I should also do the model. And then you have enough data, you have enough sort of domain expertise where that becomes its own, you know, sort of flywheel.

3:25

So I'm very bullish on this idea. And what's cool is it's opening up like multiple layers of, you know, sort of opportunity for startups. Because you could either be the actual applied company itself, i .e. the Neo Lab, or you could be the infrastructure provider to the Neo Lab. And you have like multiple layers of, you know, going and attacking that space.

3:44

But I think a huge update for the market and everybody's kind of views of it. Box Labs, let's go. We already have it. It's a little bit secret, but we pay very close attention. Right now, the main focus is let's make the agent really, really good on any model. But over time, obviously, you would peel off certain use cases, either on a per customer basis or kind of across the whole data set.

4:07

Yeah. I mean, it seems like there's two forces that are happening. One is people don't want the fox to be guarding the hen house. They don't want the seller of the token to be the one that is also metering and gating, you know, what is the best token for each use case. And then the second is, you know, there's actually a lot of work to do on that bridge to cross.

4:22

And there's real research involved in it. And it's pretty bespoke to the exact workflow and the exact end customer that you have. Yeah. I think what I tend to see happen in the Valley is, and for a very good reason, and it's actually why these companies have been so successful, is like everybody is so kind of research -pilled, which is, again, totally awesome.

4:41

Big fan. It leads to all these breakthroughs. But the sort of sense that, okay, the model and the intelligence in the model is kind of the only form factor that matters. And then you go to the real world and you sort of see how intelligence actually gets rolled out in people's workflows. And the model could be the most intelligent, you know, super intelligence in the world, but that workflow still requires you to connect up to other data systems.

5:05

Yeah. Still requires these moments where there's a human in a loop interaction. There's delays in the workflow, and so the sort of agent has to sit idle. There's change management of the actual business process. There's legacy systems that haven't been modernized. So you kind of go through these five or ten things that are way sort of, you know, much more operational, much more blocking and tackling than just the pure super intelligence of the model.

5:27

And the last thing that I think a classic sort of research organization wants to go do is go attack every single one of those things. And this is sort of not a– there's like– this is not a sort of a one -off in history. Like we've always had this relationship between infrastructure and application. Like, you know, obviously AWS or GCP or Azure have created, you know, trillions of dollars in value of kind of market cap of infrastructure.

5:49

But guess what? There's also trillions of dollars of value in software that only exists because of that infrastructure. And if you were to go back, you know, ten years ago and you were to look at what was happening in the data space and you looked at what, you know, GCP– early kind of versions of GCP was building or AWS was building, I guarantee you would not have predicted Snowflake or Databricks existing.

6:08

You would have been like, the infrastructure just already does that. Why would you pay another $10 billion of revenue to all these other products that are just making it so you can work with your data? The same thing is going to be true for intelligence, which is the models will be insanely valuable. But the application of bringing those models into real workflows in banking and life sciences and healthcare and government, that's just going to be a lot of software.

6:28

Now, the sort of challenge for the next, let's say, two to five years is how much do the model providers need to move up that stack and also try and, you know, go and compete at that layer? Or do they either leave open intentionally or accidentally that entire space for the application sort of ecosystem? And actually, to some extent, this is like a big question strategically for them because on one hand, you want to be closer to the customer.

6:51

On the other hand, you also want to be able to have an ecosystem so people trust you. So that's going to be like a really interesting tension over the coming years. How do you think they'll play out? I feel like that's the question we spend every day wrestling with. Yes. You can, you know, I don't want to speak for Sequoia, but you can sense some of the existential dread of a VC right now is like, should I just put another billion dollars into Anthropic? Or, you know, should I attempt to sort of see what the applied layer is going to play out? So nobody's, you know, that envious of your guys' position having to figure that out.

7:21

But obviously, it's even harder for the entrepreneur.

7:24

But, you know, I am, you know, I'm pretty long, obviously, the application layer. Like, I'm also equally very biased. Like, I have a very concentrated bet with very limited diversification on it working out that you still want to buy technology that sort of understands your workflow and can get to the core enterprise data. But I don't know that there's– I just don't see a different event happening than all of history.

7:50

I mean, you know, we only have like 50 or 60 years of computer software history. But basically, when you go to that law firm and you go to that, you know, pharma company and you go to that business process. And AI has not meaningfully sort of changed the need or the shape of what that looks like. And the best sort of manifestation of this is sort of this chatbot versus kind of agentic workflow demarcation.

8:18

So the chatbot can be totally universal and can be totally horizontal. But, you know, all of a sudden you kind of look at that and you say, well, my workflow kind of needs something to kind of ping me at the right time in the process. Or it needs access to a certain kind of data that just, you know, the chatbot can't natively get access to.

8:36

So somebody has to go into that organization and like get it set up. And somebody has to go and provide domain expertise to this model so it really understands our particular business process. And so then unless you really just underwrite, you know, the big labs at honestly, like I'm not exaggerating, like 100 ,000 employees. If you don't underwrite that, then then the diffusion kind of economy is going to be massive.

8:58

Because every single one of those companies, whether it's a 50 -person firm or certainly a multi -100 ,000 -person firm, is going to need an army of people to go in and help them with that transformation, that change management. And so this is, you know, like whether it's the FDE phenomenon or just, again, understanding that domain expertise, that's going to be a very big deal.

9:15

And then you kind of alluded to this point, Fox kind of guarding the head and house. And that might actually be singularly the biggest sort of reason this has to happen, which is even under all sort of like complete benevolence and like nobody's actually doing something in any kind of like sneaky way, it just stands to reason that if I'm going to sort of give a task to an agentic system, I just want that task to be cost optimized, like with accuracy as kind of holding constant.

9:45

And so who can do that? It's the company that doesn't care among, you know, 10 different models which model is performing that task. It's like definitionally, you would want that to be the company that does not have a sort of a preference. And then the force you have going against that is that the model companies can subsidize their models or offer them at different rates.

10:04

I don't know how long that lasts, though, because when these companies become public, I think they will be held to basically the same laws of capitalism that everybody else is. So the subsidization is working very well up to a certain threshold of spend. And we might have exceeded that spend when you're at like tens of billions of dollars of kind of capital.

10:24

And in fact, if anything, it might even be worse because eventually you have to go in and sort of pay for your training runs as well. So like they like the subsidization of tokens is a I think it just has to be a temporary phenomenon. You mean from a gross margin perspective or from like an antitrust perspective? And entirely gross margin.

10:42

Like if I. But they have such high gross margin on their inference right now. But then what are they subsidizing? The like then then it's actually then they're then they're charging actually like, you know, decent rates. Yeah. But they can afford to price the API higher than their own first party products. Right. So that part totally fair.

10:58

You know, there's an interesting kind of, you know, dimension, which is, well, like if API is the high margin thing that is sort of paying for the subsidization. But you've moved all your customers over to the applied product and there's no API revenue. So like there is like an equilibrium you have to strike with this. Totally. And and so then all the while, if you have some sort of either non -economic actors or people just with a totally different game theory in this, you know, meta being one, maybe SpaceX being one, China certainly being a giant one.

11:25

Even Nvidia being one like that changes the calculus as well, which is those four kind of cohorts don't necessarily need to make money on on inference in that and in sort of at least the same margin structure at like what an anthropic or an open AI needs. So they would might be fine to bring down inference to 10 percent margin because they just want to basically, you know, pay for massive compute clusters.

11:48

And then as long as that happens and as long as like there's not insane proprietary sort of, you know, you know, sort of closely held secrets, then no matter what, you're going to have kind of cost per token go down on a like for like basis. All of which means more value accrues to the application layer, which is, I think, like a very long winded way of saying I think there's just value in kind of everybody in the stack.

12:08

I just don't know that that the I the only thing I probably wouldn't bet on is just, OK, one or two labs get 95 percent of the value creation. I think there's just going to be a much more dynamic environment. And honestly, if I were one of the, you know, two or three biggest labs, I think I'd prefer this outcome, too, because back to your antitrust point, like at some point you'll just be nationalized if you're the only thing that that is sort of exists as intelligence.

12:32

So you kind of want a little bit of healthy competition in this ecosystem anyway. Yep, totally. Let's let's transition talking about box. I want to come back to talking about tokens and Jevon's paradox and trying on all these all this stuff. But let's talk about box for a second. I love that topic. Give folks, I mean, I'm guessing a lot of people that listen to this podcast.

12:47

But give give people a brief kind of explanation of the history of box and how you're reinventing yourself. Yeah, we started the company as a way to be able to kind of securely store and share data in the cloud.

13:00

And and it was a very simple idea. But but we we kind of just kind of cracked a nut or struck a nerve. And, you know, for for Doug out there, we we were able to sort of scale up quickly. We pivoted rapidly in the enterprise. And the idea was enterprises would be moving from on premises systems to the cloud and they would need a better way to be able to sort of store, share, collaborate, manage their all this unstructured data, their their corporate documents, their financial documents, their marketing assets, their research materials in the cloud securely.

13:33

So that was the kind of company. And we we had been sort of flirting with AI kind of products and sort of experiences really since like 2015. If you remember, like the first kind of rise of the you know, maybe the the at least in modern times, the the winter that happened in like the 2015 to 2018 period, which is like we think it's going to happen now.

13:53

And it didn't. And that was a period where we were like, OK, these very, you know, sort of early models were showing us signs that, OK, if you if you looked at an image and you could classify the image, that's pretty useful if you're in an enterprise, because now maybe you take all of your image data and sort of label it or maybe you'd OCR something and you'd be able to kind of pull out, you know, kind of, you know, sort of the text in there.

14:16

That's enormously helpful. The problem was, is insanely expensive. And you had to have a model for every single use case required, you know, kind of a hyper trained model for each workflow that you wanted to do. So we kind of shelved it, you know, a few years later, you know, started paying attention to the GPT's. We had some hackathons where people were like, oh, we could do like, you know, type ahead in one of our kind of note taking products.

14:38

And that was that was sort of early kind of versions. We did some some early work in sort of text detection and classification, which helped with security use cases. Then obviously, chat to PT moment sort of hits. And and that was the big sort of, you know, head exploding moment. If for no other reason why, you know, then then they kind of figured out a form factor that opened up everybody's mind to, oh, these could be these interactive systems that you just like ask a question, get an answer back of of sort of, you know, kind of increasing complexity and length.

15:08

So we looked at that very quickly. We jumped all in. We sort of did the whole company pivot, like like everything was was kind of exactly like academically what you should do. We had a team carved out. We we put, you know, the best people on the team. We met every day, looked at the updates and then slowly but surely sort of built out what what today is is kind of our, you know, our AI stack and then basically the box agent.

15:30

And and for us, you can imagine the use cases is, you know, very straightforward. We sit on hundreds of billions of files. Every single one of those files contains critical information for an enterprise that could be their contracts, their research files, their marketing assets, their loan documents, like all of this critical information.

15:48

The problem is they rarely know what's actually inside of it. So they they they unless you literally look at the document and kind of search and find it, you just don't know what's inside of it. So now agents can go and basically be farmed out to go and answer questions about that data. They can pre -process it and extract metadata from those documents and turn it into structured data.

16:07

You can use agents to automate sort of steps and workflows. So we built a platform that basically lets you deploy agents against all of that unstructured data. And that's been the kind of core focus. Are you using agents to create new content? We are. Sure. There's a couple modalities where that shows up. One is we have, again, an online sort of collaborative product that an agent can just like generate any amount of content in it.

16:29

And then we've done most of the, I think, probably more exciting work with OpenAI and Anthropic on just how do you do like advanced document creation, PowerPoint creation. We've decided that their tech is, you know, at this point going to always be frontier. So we have an agent that goes and interacts with those systems to produce, you know, a high quality PowerPoint, et cetera.

16:50

Awesome. Your tagline, your business lives in content. Unleash it with AI. Yes. What are the hero home run use cases for how people are unleashing it today? Yeah. And then if you had to fast forward a few years, what do you think people will be doing with AI in your products in a few years? Yeah. So probably the easiest hero for, again, more of a traditional enterprise to think about is just as simple as you have a million contracts.

17:13

Why don't you find out what's inside them? Or you have a million research documents, be able to go and pull out all of the critical structure data, put that into a database, and then be able to query, analyze, automate workflows around that. So that's kind of the thing that just knocks it out of the park every single time because it's been a longstanding problem that people have never been able to go and sort of apply human sort of labor to because it's just too expensive to read every contract, every research document.

17:39

You know, maybe you could do it if you had like a loan document process, but most other data just never gets read at that scale. And then I think the stuff that we're probably, you know, as much if not more excited by is really the equivalent of what we see with, let's say, coding agents or other complex agents, which is you have long -running agents that are just executing your entire kind of workflow or process.

18:02

And this would be in the form of you go to a bank and you're onboarding at a bank and they've basically like automated every step that is possible to automate and then sort of jumps out to a person in the steps in the process for extra review or extra verification. But now instead of that sort of one or two week back and forth, it just happens in like an hour.

18:21

Like that's the dream state of most of these enterprise workflows is what if we can onboard a client faster? What if we could discover kind of critical data inside of our research much more quickly? What if we can alert to a security event much more quickly? So to do that, you need these sort of background agents or workflows that are sort of pre -established for those processes.

18:40

Totally. It's the year of the long -running agent. It is. It is. Yeah. I'm curious, like you made the analogy to cloud code. It seems to me that in the coding domain, using AI is like not only accepted, it's embraced. Yes. In the content domain, which is I think where a lot of content in box sits. Yep. But using AI to produce content at least, it's just like there's almost this allergic reaction to it.

19:03

Like all the Pangram stuff on Twitter. There's the– it's like this concept of work slop. I'm curious what you think about work slop and like will this still be a thing in a few years? So I'm going to sort of separate the box corporate hat and just now kind of maybe riff as a consumer of– gosh, I wish there was a better term– but work slop as– inside of an enterprise context.

19:26

I get bored, Dexter, and entirely written by AI these days and it kills me. So here's the difference I think on the acceptability. So there's probably like more symbolism to this actually topic than just like the slop element but like actually like diffusion of AI in general sort of almost ties to this. Code like other than the top engineers that we hang out with that are like– they have deep taste in the code.

19:55

Like like– you know, and the judgment is incredible and like it is them as much an art as it is a science. Yep. So take that group aside. For most of the world, code is a utility. Yep. It is just trying to accomplish something. You're just trying to automate something. You're just trying to put a sort of interface up there that somebody presses a button and moves to the next step.

20:14

So for most of the world, the value creation of code has been to automate things and to be able to have it as a utility. So at the end of the day, like– and we'll probably still use the term slop for a while because there's taste in kind of front -end design and there's taste in sort of systems and you don't want to have vulnerabilities in your code.

20:37

So that's going to exist for a while. But at the end of the day, if you can tell an agent like please go and generate my entire back -end system or my front -end system, like it's just like it's not only acceptable, it's preferable because it's just like that was the thing that was blocking us from moving forward. So we need to go do that.

20:52

At least the way society functions and the way the world works and our brains work at the moment, maybe this changes. When you get a presentation from somebody, there's still this association which is like I'm trying to decide if I can trust that person to go and execute on that thing or deliver that result or understand that topic.

21:40

And I don't know– I mean it just might be a thing that as a society we have to kind of keep cranking through over the next kind of three to five years and end up at the other side. Like I hate to use like these like totally busted analogies but obviously you don't care when you see somebody's financial model. You're like, yeah, that was generated clearly by like a macro or– that was like Matt, you did not personally go and compute all of that.

22:05

But you're showing it to me and we're talking about it. So why can't the same exist for a strategy deck or whatnot? But I think right now we're going through this evolution of like what is the person's role? What is the content a proxy for? Is it supposed to be a proxy for how much that person knows? Is it a proxy for what we think that they can go and execute on? I think we're just in this very messy period where we have to kind of figure that out.

22:29

Totally. Did you read the Stan Druckenmiller, Wall Street Journal article? I read the– Read the discussion about it. Yeah, the reaction to it. But actually I didn't read it. But was it like very sloppy? I didn't think it was sloppy. I loved it. And so to me it was just a nice counterexample of I have this like allergic reaction. How many it's not X, it's Ys were in there? I don't think there were any.

22:50

Okay, okay. But it does show up as 100% AI in Pangram. Okay. And dashes? I think there are dashes. Okay, you can't do that.

23:03

Something that's clearly written by AI and I just have this allergic reaction. Yeah. Whereas with the Stan piece I didn't. Yes. I'm not sure how much of that was just, you know, it's Stan therefore I trust in Stan versus– Yeah, no, but it is psychologically kind of like weird because I'll read these like, you know, X articles. And I'm like now like doing 2X the amount of work to read these.

23:20

Yeah. I'm reading it one for the substance. I'm also reading it two for the calculation of like did the person write it or am I just literally reading like a clod prompt? And then like I'm like– I'm literally like my mental processing is now like should I now– does that upweight or like lower my sort of judgment of the person or the post? And I think we're– yeah, we're in for some weird times because of this.

23:40

I would hate to be a college professor. I would just– I would totally quit because you're just like– you're like I don't know anymore what you did. Like what's going on? It does seem like the calculator is the closest analogy though. Yeah, except it's just like that was like more like finite in terms– and you still had to piece together so many more things.

23:57

Yeah. Like we– like the– and some of these analogies are breaking down of it's just a task because it's like, well, at some point this thing is doing at least 10 tasks at once.

24:08

Let's talk about harnesses. How does the box agents work? Great transition to harnesses.

24:14

Speaking of calculators, let's talk about harnesses. This is back to the kind of Neolab kind of applied layer.

24:22

There's a bunch of things that we know about our file system, permission structures, our search engine that certainly, and by all means, we actually would love all the labs to train on our understanding of this because it would only make external agents as good as possible. That are a using box. Yeah. We always talk to labs like, hey, we'll give you as much data as you want about kind of how the system works.

24:46

But let's just say like that aside, we have a lot of depth of understanding of what do people do in box? How do they search box? How do they decide when they look through 10 files, which is the one to go pick? What is their internal kind of calculus or heuristic on sort of figuring out the most relevant document to look at? So we know all that.

25:04

And we basically have built an agentic harness that attempts to kind of understand that set of domain understanding about our system. It obviously has access to our search system, our file system. It has a bunch of kind of mechanisms for just pulling out just the text of a document, just pulling out chunks from the document, doing embeddings on the document on a fly.

25:27

So there's a set of kind of tools that it can use. And effectively, it's a harness for asking questions of a large data set. So in my box account, I have– I don't even know the latest number, but on the order of tens of millions of files just because it's like everything that has ever kind of accumulated over 20 years. But I can now ask any question of all that data set using the box agent, and it goes around.

25:50

It does multiple searches in one. It re -ranks it. It then very quickly sort of pulls out the most relevant information. Then it will, in some cases, read the full document. It does all the steps. And then we compare that against like, well, what if we just gave– you know, Claude our API or gave OpenAI our API, and we see like meaningfully better results on accuracy and latency because, again, we kind of know exactly how to tune it for our workflows.

26:13

So that's effectively the harness that we built out. And then what evals matter the most to you? So I have a couple like just funny personal ones that I just keep track of of like my own use cases. But we do– we have, I don't know, hundreds of different tests that we do on every single model. We actually have two evals at the moment. One is we put out a thing called the complex work eval, which is a set of domain -specific work in life sciences, financial services, public sector, tech, et cetera.

26:40

And it's kind of exactly what you'd think of as a document -centric eval. So given these five documents and this set of problems, like what would your answers be? And we test every single model against those with our agent. And then we have a holdback eval, which is– actually, the first one is holdback also. But the second one is just like then our box instance and how box employees use their data.

27:02

And then we eval every model again on that. So we're able to kind of roughly keep track of all of the incremental progress. Like we see when things move by half a point in terms of model improvement. And then we roll out sort of default models based on different kind of cost and accuracy thresholds. And then we let customers also choose any model they want from effectively our model garden.

27:23

What's your current view of the race and like where all the horses are in terms of model performance on your use case? They more or less closely correlate code with one exception, which is actually in some of our use cases, Gemini is disproportionately better than what you would see from coding. And it might be sort of just better tool use given kind of the Gemini ecosystem and what they need to build for.

27:50

It solves kind of a strong set of sort of general knowledge work use cases as well. But I think mostly correlating to kind of code. So Fable 5 .1 was clearly kind of state of the art and the best model that we've seen. There's obviously rumors about other models. And so we'll see how the kind of race kind of continues on this front. But basically by and large, like when you look at GDPVal, Mercore has their Apex eval.

28:19

These things will all generally follow the coding models. And so I think it's we're just neck and neck on like Grok, Muse, the Fable class and, you know, GPT 5 .6 slash whatever the building next. Like these are just like it's a total race right now. And your customers typically express a preference on which model they want to use or do they just use your default? So they, you know, kind of by volume, they use our default because it's just easy and it works extremely well.

28:47

And it's tuned for there's a few ways our kind of agent manifests. So like the way that you'd most commonly experience it as an end user is you would just be searching and kind of, you know, asking questions of your data. But by volume, the volume of tokens tends to go through more of our workflow agents or data extraction. That's where actually you have customers actually doing evals.

29:08

And they're basically saying, OK, I want to really make sure that at this cost profile I can get 98% accuracy on data extraction. And that's a place where like we'll have an FDE that goes in and helps you, you know, helps understand your data environment, tests against five different models. And then you're just basically at the mercy of the eval.

29:25

Yeah. What are you saying in terms of the adoption of open weight models in your customer base?

29:30

So probably higher than people think. Lower than what enterprises actually want. And much, much, much, much, much lower than what it will be in five years. So so like some mix of that would be the kind of message like it's primarily cost that's driving that decision. I, you know, I have to probably attribute 30 plus percent to just kind of the sexiness of like I want to try GLM.

29:55

Yeah, I think I think there's that. Like I've heard I've heard, you know, CIOs of Fortune 500 companies say we're playing with open source here. And I look at that and like, well, I know for a fact like like Gemini or Muse would have been just fine at that particular cost profile that you're trying to do. Or probably even like, you know, you know, five, six Luna or Terra or whatever, whichever, you know, one had the had the crazy discounting they just did.

30:16

Like it probably would have been totally fine. But you want to be able to be like, OK, I'm a little hedged. Like it's cool. Like like we're that phase still. Over time, I think it stands to reason that that you'll see meaningful different costs because you'll be able to peel off workloads that that just only makes sense at at sort of, you know, grinding down to the cost of inference, in which case open open weights will have this sort of economic advantage.

30:39

Right now, there's, you know, this challenge of like sometimes it's more token inefficient. You know, sometimes like randomly, like I've heard stories like randomly, it'll just like speak Chinese, like mid mid chain. So you're like, OK, well, that'll be weird for a bank. So it's funny. So like we need to like probably work on some of those things.

30:55

But long term, I think it has to be, you know, the case that you're going to you're going to peel off those workloads. You know, one of the more interesting posts, I think, on this that I totally subscribe to is Jesse at Decagon. You probably read that post of like of like this paradox of like we're going to have like you're going to see clothes just, you know, continue, you know, go exponential.

31:12

But what's going to happen is each use case that kind of matures, you can peel off to open source. And once you have kind of stability in that use case, it starts to make sense to veer it toward an open weights model. Well, assuming one of two things is true, one, that that it's actually literally cheaper or two, having some post training gets you X, X percent more performance.

31:32

And and so I think you will just be in a reality where we will we will. And this is going to be very confusing, probably for like the press more than people in the valley, because you'll be like, wait a second. Like the revenue of anthropic opening, et cetera, are like off the charts. But somehow open weights is like also growing exponentially, you know, like, well, how is this like how is the pie is growing so fast? Yeah.

31:52

And it's like the pie is growing so fast. Yeah. But what's happening is is actually there's there's an interesting duality. It's it's not even just like like rising tide lifts all boats. It's like, no, no, We either use Fable or 5 .6 for orchestration, and then we farm out all these long -tail tasks to a cheaper model, or the opposite is true.

32:12

You have some kind of orchestration agent that by default does the cheaper stuff, but occasionally sees something that is just way too hard, and then it pops it out to one of these heavier models. And so you might have blended 50% spend on each, but 10 times the amount of tokens on the open weights model. And so everybody's kind of winning, but there's an interplay between why they're winning.

32:35

I'm curious about how you think about memory and customization or personalization and where that's going to go. Because it seems like today the dominant architecture is kind of like rag -based systems still. Yeah. You can get fancy on the rag, but it's still context lookup where the weights themselves aren't fundamentally changing. Yeah. It seems like, I mean, if I listen to my friends at the labs, like continual learning, this idea of like the model's weight should adapt as it gets to know you.

32:58

Yeah. We had Engram on the podcast. I don't know if you know Dan. I just got introduced to him. I listened to the podcast. Oh, amazing. So I would have loved to have been the fourth person in the room. Amazing. Yeah. I think they're working with customers to help basically bake in some of the context into the weights themselves. Yeah. What direction do you think this is going to go? You know, you're catching me at a time right before I'm actually doing my call with Dan.

33:21

So I wish I could have talked to him first, and then I'll have like a way more eloquent answer. I'm like extremely fascinated by the approach. Like I have no reason for not wanting it to work and exist. We live in a world at Box where we see the high degree of complexity on permissions and access controls and data that tends to be the rub on a lot of these types of approaches.

33:49

And I'm going to put Engram aside because I'm sure they've already thought this through. So I'm going to talk more generic philosophically. I think sometimes you will talk to a researcher that sort of imagines the world working the way they work, which is like I'm a researcher. I have access to everything. And so if I had a model that was trained just on my world, this would be amazing.

34:09

And then you're like, let me introduce you to a lawyer. And the lawyer like has this tiny little, you know, access point of just like the five projects they're working on because somebody right, you know, one door over is working on the competitive project to to another company in the space that they can't have any sort of overlap with with what they see or what they know.

34:31

And there can't be a single document that passes between those two walls and they have to be these kind of hard barriers. So, you know, so like, sure, like you could still you could still now train a model just for that one user. But like what happens if every single day they get added or removed from something that that adds important context to sort of what they need to understand? And, you know, again, like I think there's going to be probably breakthroughs in continual learning that sort of all resolve this.

34:54

But like this is why previously it was just like, you know, there was no way you could pull this off five years ago because it would be insanely expensive, impossible to kind of wrap your head around around how those access controls are supposed to work. But, you know, obviously with like as the cost curve goes down, as open weights, you know, get cheaper, smaller, faster, better.

35:12

I think this becomes super interesting. One thing on the podcast that I found very fascinating and I just need like a T -chart, honestly, is just like, you know, like what is the decision point of what goes in context and what goes in the weights? Like you have to be a little bit thoughtful about like where is the the massive performance gain that you get by by baking in the weights.

35:31

And there's probably some like incredible like like calculation of like like when the rate of change of the data is not, you know, so far. But the upside of the of the weights, you know, you know, dramatically change the accuracy of the model. Like, you know, you'd have to kind of land on some sort of rubric like that. I mean, if you could wave a magic wand, it almost seems like I think Karpathy has said this in some prior.

35:53

Like if you could almost remove all the memorized information from the models and just have have it encapsulate the specific reasoning capabilities, like the ethos of how we do things, for example, at Sequoia. Yes. And then you have all the, you know, actual content. Yes. And a lookup system that almost feels like the if you could wave a magic wand, that's what the system would look like.

36:13

And so that one's super interesting. The I think the the question will be like, how much are enterprises different at that level versus it's actually their their literal IP that is what makes them different. Like how many different types of styles of execution are there in the world? Yeah. Versus no, it's just like the depth of knowledge about that particular legal case.

36:33

And how do I apply it to this other project I'm working on? That's where so much of the value sits. So so but again, if you can just like wait till my Zoom call with Dan and then I'll really know the answer. But like I'm a fan because no matter what, there's going to be like like I've jumped right into like the individual, you know, but like no matter what, like at a firm level, there's probably ways to take this approach.

36:53

Like I'm a big sort of fan of what trajectory or applied computer doing or prime intellect because because I think there's like there's there's no question that if you're Eli Lilly, you want a model for how you do drug discovery. And that probably does need to go like farther or deeper or more sort of specific than what you're getting off the shelf.

37:11

And there's not a lot of sort of, you know, kind of church and state problems for for drug discovery workflows. They probably want as much of that information available to as many people as possible. So I think it's going to be like domain specific and, you know, you're just going to have different outcomes based on which vertical or, you know, type of use case and where the firewalls need to be in that process.

37:29

Makes sense. OK, so you hinted at the beginning that there's a box labs. What what type of work is box labs doing? So it's the equivalent of box labs. Like I don't know if we've used capital L yet, but but basically, you know, it's our it's our applied sort of A .I. team. And what research areas are most interesting to your team right now? Yeah.

37:46

So so the the biggest areas that that the most sort of research kind of, you know, of the continuum of engineers, there's some cluster that is sort of more on the research bent. And of that cluster, the things that we spend time on still, again, is at the kind of applied layer. But but it's a lot around how do you how do you take agents and make them, you know, another 10 points of accuracy improvement given X problem? So what is the what you know, how do you build a map of the of the problem set, you know, with a given set of data to sort of best execute on that task? So we spend a lot of time on on that style of work.

38:21

We have a team, for instance, working on how do you do effectively at the agent level, at the harness level, you know, some form of kind of auto research on on sort of hill climbing on accuracy of answering questions or sets of problems on on given a kind of a set of client data. So if you're a bank, you have a bunch of loan documents coming in.

38:41

And these are like 100 page documents, like whether you're getting 70 percent accuracy with an off the shelf model or like 97 percent is like basically obviously a world of difference. And can you actually go and automate that process? So somehow you have to hill climb from the base model to the 97 percent. And there's a lot of work going into the system to basically pull that off.

39:02

Maybe zooming out, what do you think of as and this can be a box question or a non box specific question, the role of systems of record in a world with agents. And I'm sure you saw some of the Twitter discourse on like, you know, every software company is trying to sell me their own agent right now. I don't want another agent from them. I want I want their system record to work well with my agent.

39:20

And so how do you think about that? Hashtag Claude Force. So I mean,

39:27

literally, it's like one of those things where where like the first three minutes, you're like, man, that seems funny. And then four minutes later, especially when you see the stock, you're like, ah, brilliant move. Like, like, this is great. We're doing this. And then when you I think somebody somebody actually said this the best they're like when they heard Matthew McConaughey say it out loud.

39:45

That was like really the that sealed the deal. That was the aha moment. That was the aha moment. It's like, man, he can sell software like it's actually incredible. Like his voice is so good for selling systems of record and agents. If you're in like like our sort of, you know, contemporary group of like you built a SaaS platform and you have some set of data and workflow that that, you know, your customers kind of operate in.

40:11

There's there's effectively two things you just have to do. And I think anybody attempting to do one over the other is just going to lose. You have to build an agent that is insanely great at your product. Like that agent has to be you have to provably be like 10 or 20 points better than an off the shelf agent using your system. Not because it's hobbled the other side.

40:32

It's just like you are so eval maxed and you're like so tuned to your particular workflow that you can improve your system. You have to have that. And you probably because you understand your domain, unless you're like totally asleep at the wheel, you probably have use cases that no one has sort of thought to build products around because you talk to customers every day and you see what they run into.

40:51

And you're like, oh, we could just like have our agent go do that for you. Like I've had at least a dozen. I mean, so I probably talked to a couple hundred customers a year in variety capacities. I've had at least two dozen times where the customer has a use case that is like a breakthrough moment for me of like, shit, that would be actually totally insane.

41:11

Like, what example? Unfortunately, since you put me on the spot, I don't know if my example will pay off the level of excitement I just had because it'll be on the spot. No, no. It's just like I think the thing I was thinking of is just like I think it's going to be a womp womp for the podcast. But there was there was basically a customer had this idea of they wanted an agent in the background trying to sort of figure out when documents sort of met their governance policies.

41:38

And like does something need to go into some kind of archive or something into some kind of legal hold or whatnot. Oh, cool. See, exactly. See, that's exactly the voice. That was exactly the voice I was worried about. Yes. Yeah, no, you couldn't even pull it off. Okay. Okay. So but in our world, this is awesome because you're like, oh. Yeah.

41:56

Like no, because think about it. Every company has a head of governance. I meant Cole sincerely. Okay. I believe you. I mean, listen, you do enterprise. So like I think that it was at least half serious. Imagine you're an enterprise. You have a head of compliance and a head of governance. Okay. They can only be like overseeing the whole sort of enterprise.

42:15

They've never been able to be everywhere at once. Now imagine if they could sit next to the employee and basically be able to be like, oh, you're about to go do something that breaks our governance policy. So so like the idea was like, oh, what if there was just an ongoing agent that just like automatically was just like, now that's going to break your governance policy instead of the user having to like try and predict or understand the stuff.

42:36

So anyway, those are the kind of things where if you have an agent within your product, you're going to be able to identify sort of sooner and better than the rest of the market and or just like do things that maybe would be impossible to do off platform. On the other hand, it's just like, obviously you have to go headless. You literally have to make sure that your APIs are exposed to Claude and ChattoBt and in all the different platforms.

42:58

And you have to make sure that you have a either a direct way into deterministic APIs so that those agents can use your APIs, make calls via MCP or whatever, or at least make your agent be headless and be exposed in those systems. And then the only reason maybe this is like even remotely a hard debate is you have to just make sure as a system of record that you can find a way where commercially it sort of makes sense on the other side and is sort of valuable and interesting.

43:25

And the reason why I think a lot of people got that wrong that weren't in these companies was just underestimating the amount of sort of new use cases that are just total upside. There are like complete white space opportunities for these systems of record. So like in the Salesforce example, I use Salesforce more today, probably by an order of magnitude than I ever have because I MCP into it via Claude or ChattoBt.

43:46

And so I just am always asking questions about the data inside of our CRM system. And do you think that means the systems of record become more toll booth businesses then to make sure that they're capturing the opportunity? I don't love that term because like no one's had a good experience at a toll booth. I love toll booth. Yeah, exactly.

44:02

You love it more than governance agents. So I would say that because they sort of have a depth of purpose of organizing the workflow, managing the data, securing the data, providing guardrails, then it's really just– yeah, you have to have some kind of volume -oriented business model on that other side. And I just think there's– like if you're solving real problems for customers, it'll just like make money.

44:27

I've like– like this is so cheesy but like I've told like LinkedIn product managers, I'd probably pay 10x more for LinkedIn if I just could MCP into it. If I just had a way of just like always understanding like, okay, the CIO is doing this thing and I need to reach out or whatever, I'll like take my money. So there's– these systems actually have a tremendous amount of value based on the data that they have.

44:50

And customers will absolutely sort of find some way to reward you for that value creation if you're doing a job. Super interesting. Maybe related, let's talk about product UI. Yep. Big, you know, generic chatbot– chatbox agent. Is that going to be the dominant UI for how people are using AI in the future, especially when it comes to the application layer? This is why I think the applied layer has so much room to run is because probably the sort of universal chat system that you ask a question to, you get an answer back or it does sort of some work in the background.

45:26

I think it's going to be– obviously like that's going to be a mainstay. That UI will always exist. It will be incredibly powerful. The horizontal products will have it. The vertical products will have it. Everybody will have it. It's just like your product has a search box. Obviously it does. Yeah. So that's always going to be here for these sort of one -off asks of an agent or go find this thing or answer this question or produce something for me on demand.

45:48

But most of the enterprise is sort of made up of these processes and workflows that are kind of just happening behind the scenes. Sometimes they're happening with computers and computers are running these things or sometimes they're happening with other people that are doing these things. Or sometimes they should be happening with people but you could never afford to have them happen with people so they just didn't happen.

46:05

And so that sort of is a slightly different kind of metaphor than a chatbot where you ask a question and it comes back with an answer. That's like, okay, I kind of want agents in the background to do things for me, read every contract, look at every log, triage every security incident. And then instead of me chatting, maybe I'll chat as a means of doing kind of a catch -up.

46:30

I want a dashboard. I want a workflow. I want a queue. I want a task list. So then the challenge becomes, well, does the horizontal product kind of take on every one of those components and manifest every one of those experiences in one? In which case I think you'll start to be like, man, that thing is like really like a– that's pretty heavy.

46:52

And then we'll start to like be like, oh, this is no longer this simple, easy, delightful thing anymore. So then the vertical players actually like– like they actually sort of understand the process and can manifest all the right buttons and tabs and the names of the things for that particular workflow. So I think it puts– I think as you have agents that are doing more work in the background, doing more async work that is just like I farmed out a bunch of agents to review things as they happen or whatnot.

47:17

That leans more toward the applied companies that can understand those workflows, that can understand those processes. I think you're going to have these in every field. You're going to– certainly we already know that– how they're going to look in legal with Harvey Lagorre, et cetera. We are seeing them start to emerge in areas like security.

47:35

We've seen them start to emerge in the long -running kind of coding agents with cognition and factory. So I think that that will be one of the bigger kind of applied AI sort of use cases. And ultimately like in five years from now, I would bet like 90% of all tokens in the enterprise are things that a user never kicked off and they just see a result.

47:56

They just– they see a task show up and they have to go review it. And it's just like– it's just happening. Yeah. Yeah. Makes sense. Okay. Let's talk about AI diffusion. Coding agents, it was like boom, January 1st, 2026 happened and like the fastest diffusion of anything into the economy we've ever seen has happened. The diffusion of the rest of the AI magic into the rest of our jobs seems like it's been a lot slower.

48:20

Yeah. What are your thoughts on that and where are the areas where you think we're going to see faster diffusion and how is that going to happen? Yeah. Yeah. So you always have to kind of compare and contrast coding versus everything else to really understand the dissimilarity. So in coding– and this is back to the kind of utility point on slop.

48:38

Like the utility of code is almost 100% represented by the amount of text that you can generate. Like obviously an insane amount of value went into the text but like knowledge and expertise and meetings and everything. But like ultimately like the text is the thing that produces the program that is actually the thing that you're trying to do.

49:01

You know, if you could have the world's greatest programmer like– and never had to sleep, never had to eat. They could intuit what to build and they could just sit on a computer all day long. Your value creation would be 100% correlated with how many hours they could sit at that computer and type code. Like lines of code, ideally good code is the most thing that will be correlated to whether you produced software that people wanted.

49:25

So it's all text. The models are hyper -trained on these. Everybody in AI labs treat coding as a competitive benchmark to constantly try and exceed. They get to do their own evals on it every single day because they are the ones coding the models themselves. And it's the most technical audience of all time where when they deploy an agentic system and they run into either a bug or a problem or like some MCP server comes back with like connection invalid, they know how to triage the problem.

49:54

They don't call IT. They're just like, oh, yeah, no, I didn't open up that port. Sorry, I'll go fix it. So that's like five things. Oh, and maybe like the sixth. It's just like a very, very high -paying vertical that it's just like automatically valuable if you could get 10% or 20% productivity gain, let alone 5x productivity gain. So take those five or six things that coding has as sort of beneficial properties to sort of automation, then compare that to, you know, you know, every other form of knowledge work.

50:25

And you'd probably have like a histogram. And I don't know if anybody's published. Maybe you can like the similarity to coding and like, like what are the domains that like, like start to sort of sort of, you know, look closer, like, like less and less like coding as you as you kind of scale out. And it's like, OK, well, lo and behold, legal is kind of interesting because because like there's a lot of value creation to somebody sitting at a computer reviewing legal documents, writing legal documents, like processing large amounts of information.

50:51

OK, so that's kind of blowing up. And then you kind of go down the list. Now let's take something like like a sales rep. OK, so much farther down the list in terms of sort of likeness. The sales reps value creation is basically convincing an external customer to buy software or technology or a caterpillar truck, you know, from them. That is the value creation to the to the economy of the sales rep.

51:14

And so let's say we brought the world's best automation to them. Like, first of all, again, they'd have to like figure out how to technically wire it up. They'd have to make sure they give it all their data, all these kind of things. But no matter what, the they're still rate limited and constrained by like, did the customer respond to them? Do they want to meet? Can they meet next Tuesday or can they meet today? Like, does the customer have budget? All of these other things.

51:36

So that's maybe the entire continuum right there is like one is like of knowledge work. Like, obviously, this is not even touching the sort of working with with Adam. So that's a continuum, which is on one end, you have somebody rate limited by so many external factors. On the other end, you have somebody who could sit at a computer all day long and just type type text.

51:54

And that is your ability to basically automate things is that continuum. So for the real world, we have to basically bring intelligence to these workflows in ways that are that sort of somewhat feel like the shape of their workflow, somewhat feel like the shape of their work and then find a way to deliver the change management, deliver the implementation, get data into a format and into an environment that actually works with these systems like asterisks.

52:19

Like one of the other big things is like if you go to most engineers in 2026, like maybe like minus two months ago, given the given the latest phenomenon. But like like the codes in GitHub, you just like connected to GitHub. Like like remember, there's this period where like you're when you launched a coding agent, like there was no like sign up or register.

52:37

It was just like, give us your GitHub. That doesn't exist in knowledge work. There's no like give us your GitHub for knowledge work. There's like give us your box. Well, box customers have a much easier time with all of this. Unfortunately, like we're only 1 .3 billion in revenue run rate. So like like that means there's a lot of people not using box.

52:53

And so what are they using? Their data is in on -premises systems, legacy file shares, legacy infrastructure, you know, enterprise environments that don't talk to agents particularly well. So just think about that sort of distinction between, you know, implementing coding agents versus everything else. Even something, again, you'll sort of fall asleep about is like access controls in the enterprise are totally different.

53:17

I actually totally forgot that point about coding. In coding, you get access to basically most of the stuff ever relevant for your job. In knowledge work, you're like, hey, Sally, can you open up that sort of file share for me? Can you open up that, you know, sort of project? Correct. Because I didn't get access to it. How do you make sure the agent has access to those set of things? All of that work has to get done.

53:37

So the thing I think we have to prepare for is two things. One, Silicon Valley has to prepare for diffusion taking a lot longer than they think. Or then I'll say we, but I really think they because because I like I know how long it'll take. And then the second thing is, is the good news is this is sort of all correlated to the applied layer value creation.

53:56

Like the because the companies that will just have the patience, the sort of the full sort of domain expertise, the just the sheer work ethic, because it's not like everybody's just from the floodgates. Like you have to go in and just, you know, pound pavement and get out there. That will be the applied layer. So I think this actually represents, you know, a trillion dollars of applied layer AI value is is actually how do you go get the technology to the lawyer or to the sales rep or to the life sciences researcher or to the person that runs the customer support team? That's all sort of opportunity right now that exists.

54:32

Awesome. I'm going to close by asking some advice for the founders. Maybe let's start with founder advice and company building advice on the founder side. It seems like you are in every AI cap table, you know, every cool new company like, you know, you know, and Graham, you know, how did you kind of get yourself in the middle of the AI conversation? There's probably two two parts.

54:52

One was just very well primed for it. Like working with unstructured data for 20 years, you just like can instantly see the benefit of agents on that. So like I honestly like took longer than I would have wanted that we we got to have this conversation because we tried to have this conversation, you know, eight, nine, ten years ago. And now it's finally happening.

55:09

So so first of all, just super well primed our product sort of shape and what people do with our product already lends itself extremely well to agents. So like obviously we had to bet the company on that. And and then lo and behold, we had the positive feedback loop of customers actually saying, yeah, that would actually be very powerful if I could go read every document and answer any questions.

55:28

So that that's the first and certainly by, you know, biggest by kind of a factor of ten. And then the other is just like I'm extremely fascinated by the technology and it's just fun. It's like where to learn about it, your podcast podcast podcast.

55:45

The I mean, probably unfortunately for brain cells, like it's probably 95 percent Twitter. I have a routine where like at the end of each night I just go through the feed and it's just like I like I look like some sad meme. Probably I'm just like I'm just scrolling and scrolling and scrolling and like attempting to like triangulate all the information.

56:05

It's amazing because it is it is the global town square for it is. And I want to send an emergency alert to like everybody who's like a sophomore or junior in college and just be like, follow these 20 accounts on Twitter. Yeah. And and also join Twitter because like this will just help your career. You're either like a year ahead or a year behind simply based on your feed.

56:23

And my feed is like so, so wired in. This is the number one advice I give to people when they're asking how to get current on AI. It's like follow follow these hundred accounts. It's like but I mean, virtually first step join Twitter. Yeah. Like you'll still talk to 20 year olds that are like, yeah, like, you know, I see some articles and I'm like, what do you mean? How do you see articles? Like, I don't even know what that means.

56:43

Like, do you just like get lucky that somebody emailed you an article? Like, just join Twitter. Like, what are you talking about? So so so I would you know, you just have to be wired in. I enjoy it. It's a lot of fun. I play with everything. What's what's your favorite new product? Please say instinct. So, OK, full full disclaimer. I've not done the instinct invite code yet simply because I have a backlog of like three other personal assistant products.

57:10

You guys I think I know I know everybody's read. I'm very excited. I'm on I'm not an investor. So you're not. OK, so this is like totally genuine. OK, so I 100 percent will have I don't know when this is going to run, but I'm sure I will have play with it by the time it runs. I'm in pre -release in a couple right now that I'm spending some time with.

57:27

I think the personal assistant stuff is super exciting, at least in some of my use cases, it's still showing some of the limits of like browser use as an example. Like, man, we still have some work to do there. Like probably the entire internet needs just like a CLI for their product. So we still have, like there's still some blocking and tackling at the infrastructure level for these things to be totally awesome.

57:46

And then I kind of look like your average, probably, you know, AI -pilled knowledge worker on, you know, every day I'm asking one of five different AI systems, you know, 20 to 30 questions on just like doing research, looking for talent, looking for what is a competitor doing? What's happening in this market? How do you expand there? So to kind of that.

58:08

What about on the company building side? How do you, you know, 20 -year -old company at this point, what's your advice for other people that are trying to reinvent their companies to make sure they have, you know, max adoption of AI, not just at the individual level, but also at the kind of company level? How do you make your business legible for AI? Yeah.

58:25

Again, some of this we have as a byproduct of how we've always thought about information systems in the company. So like, no exaggeration, if you have a question that you would like to ask about the business that has ever been documented in a form of unstructured data, so a meeting note, a project plan, a roadmap, a presentation, a financial document, a financial planning session, it's 100% in box.

58:52

So like we benefit from a data architecture that already is insanely sort of tuned for, because this is how we've run the company, we didn't let anybody use anything else. And so the data is very easy for us to be able to work with at scale. And then of course we have Salesforce and all the other kind of core canonical systems. We've been able to kind of have, I think, pretty good data hygiene.

59:12

So agents running on top of that makes it a little bit easier to be kind of AI first and how we operate. Maybe a couple best practices or things that we've seen, you know, first of all, trying to figure out where the highest leverage impact workflows are going to be and, you know, trying to kind of target those. So, you know, our CIO is very AI -pilled.

59:29

We have a little bit of like a center of excellence on AI. We've hired some internal AI FDEs to kind of help with these processes. Do you have leaderboards? We don't token max. We do actually have like, I mean, like, I guess like literally we have a list of people buy a number of tokens, but it's usually to like go and inspect like, okay, do we think that's useful or is there a learning there that we should take back to some other function? So like probably our top, you know, in the top three, all AI users at Box, like probably one of them is wasting, you know, half the tokens and then two of them are like, oh shit, whatever they're doing, we need to go like do like an

1:00:03

internal training session for everybody else. So, you know, we had this thing like two weeks ago where I was like, can you just get everybody in a room, you know, on this particular team and just like, like show them how this one person is using AI. And then like, you know, six hours later, they were like in a room and the guy was doing like a full demo of what he was doing.

1:00:23

Shout out to Mick. And so like, that's the kind of stuff that we're trying to do, which is just like, how do we show everybody what it looks like to work in this way? But, you know, for as fast as we're moving and we are shipping at some parts of the stack two or three times more sort of, you know, actual customer facing products. So like, I don't care about how much code, but like, did we actually deliver more functionality that customers asking for? Some parts of the stack, we're doing that.

1:00:48

Then you'll go talk to a friend in Anthropic and they're just like, and you're like, oh my God, we still have, we still have ways to go. Like, like I, you know, the particular meta kind of constantly changes, which is like, you know, two years ago, you'd be like, you're just using a plugin in your IDE. And you're like, man, that's, we're not going to be that.

1:01:04

Like we're not ready to fully, you know, be AI first. And then you're like, okay, everybody roll out, roll out cursor. And then you're, everybody rolls out cursor. And then you're like, and then that finally happens. And then you're like, you know, you're walking around, you're like, you're not only working from Slack, just at mentioning bots doing your code.

1:01:20

Like, what are you doing? Like, like, it's like, we're constantly just changing. What are the, what are the workflow paradigms on this? Do you guys have like a Slack coworker agent, for lack of a better term? We have a few that, that have that shape. I'm pretty excited about like, like Claude Tagg as like a form factor. You know, you still have to again, kind of get the, the team construct right and the data right.

1:01:40

But we have a variety of ways that people, you know, kind of work with agents in Slack. I don't know if it's as, as sort of Slack pilled as Benioff would like us to be, or as like anthropic or opening eyes, but like we're, we're heading in that direction. Yeah. I think that history books will be written about, you know, the art of business in this time.

1:01:56

Like, I would love to read the new, the art of war with like everything that is happening here. Cause I think it's pretty extraordinary stuff we're seeing. What do you think it takes to win an AI versus pre AI? And like, what does it feel like to be a founder right now versus when you started books? I am both jealous of, and also not jealous of, like, like the young founders you meet.

1:02:16

Cause you're just like, like, oh, to be young again. And, and the whole world is your oyster and you can go in, you know, any direction and the leverage you have, you'll meet with them. And you're like, oh my God. Like I saw a product a week and a half ago and they, they did a demo of the product. And I was like,

1:02:33

this would have been a 40 person project five years ago. Like, you know, especially when we were starting out easily 40 person project. And it was, it was two people. And you're just like, how do you have so many tabs that work? And they all seem to have stuff behind the tabs that all seem very functional. Like this is not fake. Um, and, and so I'm very jealous of that.

1:02:51

Like, it's like incredible because you could just like, you can start your company from scratch with that as the design principle. Now we will, we will get there because we're just going to muscle through it. There's a couple of things that we just can't do, which is like, we're not like, we're very uncomfortable at the idea of like, you sort of, you know, removing the code review and like some of these things to get talked about because our customers can't possibly entrust us with their data security and compliance if we don't take that seriously.

1:03:13

So we're always going to have a little bit of a discount on the productivity because of, of where we are in the stack, what we, what we do as a business. But so jealous of, of being able to kind of be fresh in that. And then on the other hand, like it's like, man, like at the same time for every great idea, it's like instantly five competitors.

1:03:29

We didn't have that, you know, kind of couple of years where we just could like, we could just grind on our product and our, and our sort of experience. And it wasn't like every three days you were like, oh, like Sequoia funded this thing and Benchmark funded this thing. And like, we, you know, we weren't going kind of nuts like with that.

1:03:47

Now we, we had our own version of that. So like at the time I probably was going, going nuts, but it was like, like in retrospect, it was like not worthy of going nuts. Now it's like, oh man, this is like a real race in every one of these markets. Totally. So I think I'm probably pretty consensus on this, which is in a world where AI builds things so much faster, then probably the shift moves to whoever can actually get it to the customer is in the best position.

1:04:11

So, you know, it's, it's fun talking to founders that are pretty kind of pilled on that. You know, Scott or Matan are like, they, they get the mandate. They're just like, this thing is going to be an enterprise diffusion play. And, and so you have to get it to the enterprise. Um, and, and anybody who kind of mistakes the mandate right now, you're just going to lose.

1:04:31

It's just like, it's game over. Sorry. Like there is quite literally trillion to trillions up for grab at the applied layer and the companies that, that know how to build the teams and get to the enterprise will be the ones that win. It's just like, obviously guaranteed. Well said, Aaron, this is a very fun conversation. Thank you so much for joining.

1:04:49

Thanks for having me. Thank you.

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.