Full Transcript for Managing AI Coding Costs at Scale

Over the past couple of years, companies raced to get their employees using AI, often encouraging them to lean on it as much as possible. However, as AI coding tools spread through engineering teams, many companies watched their AI bills explode. That kind of runaway cost growth can threaten a business, so the new challenge is capturing the productivity gains from AI without the spending spiraling out of control.

Databricks is a data and AI company that hit this problem early, running AI tools across thousands of its own engineers. Rather than clamp down on usage, the team built a set of practical ways to manage the cost, from taking advantage of steadily falling model prices to automatically routing each request to the cheapest model that can still do the job well.

Patrick Wendell is a co -founder of Databricks, where he leads engineering teams and has driven much of the company's internal AI adoption. He recently wrote a viral blog post on managing AI coding costs that reached the top of Hacker News. In this episode, Patrick joins Kevin Ball to discuss why AI costs suddenly exploded, the main levers companies have to bring them under control, the trade -offs between managing costs on the server versus in the developer's tools, and why the next big challenge is giving AI the right context about how a company actually works.

Kevin Ball, or Kate Ball, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co -founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action Discussion Group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball .llc.

Patrick, welcome to the show. Thank you so much for having me. I'm excited to be here. Yeah, I am excited to dig in. Well, let's start with a little bit of your background. So can you tell our audience a little bit about your background, how you ended up where you are today and in this conversation? Yeah, sounds great. So my career history is pretty simple because I've only really had one job in my career.

And that job is that I was in grad school at UC Berkeley in 2013. And a few of us that were working on, we were working at the intersection of machine learning, then machine learning, which was not really a lot of deep learning yet, but it was sort of scaling up classic ML. And we were looking at those problems. And then they were very tightly coupled with big data problems.

So dealing with really large data sets, almost a little small progenitor to like the scaling laws that are happening now. It was also true then that you saw if you just kept scaling, a lot of stuff started to work better. So we were looking at research problems in that area. And then we left to start a company that's called Databricks that's still around.

And it was initially around some open source technology we had built around these problems, the kind of large scale data and machine learning. And then at Databricks, I wear a founder hat. I'm one of the founders. So we have many different jobs. You never really just have one job as a founder, but my day job is working quite a lot in engineering and helping support several of our engineering teams.

And in that capacity, I've also been focused a lot in the last few years on our own use of AI in the company. And for a company our size, I think we were pretty early in terms of being quite ambitious about adopting AI internally. And I've been the one most internally responsible for a lot of those efforts working with our team. And so I think I have quite a perspective to share around.

We have thousands of engineers and we're trying to give everyone AI tools and get them to be super productive. And at the same time, we're worried about security costs, our own intellectual property, being careful around that. And I think it's an area a lot of people are interested in. And I'm happy to share. I don't think we have all the answers, but we have learned quite a bit from what we've done.

So I'm happy to share that here with you. Yeah, absolutely. Well, and I feel like we could do multiple episodes. Like I'd love at some point to just dive into the history of Spark and all of that, like which really honestly changed the direction of data analysis as you enabled streaming data analysis and all of that stuff. But let's focus on AI for today.

It is the topic of the day. You recently wrote this blog post about managing costs. And I feel like this is a transition we've gone through over the last year where it was like first everybody's like adoption, adoption, adoption, throw everything at the wall. And then suddenly people were saying, oh, my gosh, this is expensive. How do we rein things in? And so can you give us the TLDR of how that's gone at Databricks? Totally.

Yeah. And the reason I wrote that post, I do encourage people to check it out. You can just Google Databricks, AI costs, and you'll find it. Is that I had talked to so many of our own customers who are also many of them tech companies that are trying to roll out AI. And they were saying, hey, what have you tried? What works? What doesn't work? And I also have talked to a few other peer companies.

Actually, there are some credits in that post. We had talked to Stripe, Uber, I think Ramp, and a few other. We just trade notes with other tech companies and see what everyone's doing. And remarkably, folks are looking at very similar problems and solutions. So that was pretty cool. So I wrote that post to kind of dump everything out of my brain so that at least it's one data point people can look at, since I think many of us are facing similar problems.

But the origin of that problem is something you just said, which is that there was this period, and we got there actually pretty early, but it was around 2023, 2024, maybe beginning of 2025, where companies just were worried they couldn't get people to use enough AI in the company. So especially bigger companies. And it was all about pushing employees, try this, try that, incorporate these into your workflow.

And something changed roughly, I think, around Opus 4 .5. It was kind of when the AI models started to get really good at coding. And it was started with Opus 4 .5, and then many others came, combined with these really high -quality harnesses like Codex or Cloud Code or Cursor, or there's others as well. Suddenly, the problem wasn't getting people to use them.

The problem was that people were using them a lot. And around the same time, all of the foundation model apps switched to consumption pricing. So instead of like a seat where you know, okay, I'm going to pay $100 per month for developer, it was a consumption, you pay as you go kind of thing. And then those two combined to at least at Databricks start to encounter what looked like exponential growth in costs.

And exponential growth is like not as super sustainable, like in any cost in a company, unless you're exactly getting exponential increase in revenue that's commensurate, and exponential growth in cost is just eventually going to be a huge problem. So we started looking at this when we were modeling the costs and seeing, huh, that's funny.

It's like doubling in this two -month period, and it's doubling again in the next two -month period. And we have thousands of developers, so this can start to accumulate really fast. And I would say, oh, I don't know, maybe beginning of 2026, around that time, or late 2025, is when we realized. And again, we wanted to have an attitude of people use tools.

We want you to use tools and be productive. But they're just some law of physics where when costs start to accumulate at a certain rate, it's going to eventually, any company that doesn't address it, it will eventually bankrupt that company. Just law of physics. Well, and I think that's why they went to this usage -based pricing, right? Because the model providers and then cursor and the harness providers were like, oh, we're paying per use because we're fundamentally shuttling electricity through GPUs, turning it into intelligence in some form.

That costs money. And as the models got good, people's usage of them skyrocketed. And so they said, oh, we can pass this along. They can hook their revenue into their costs. But that's hard for the rest of us. I mean, I'm not on the inside. We do partner with a lot of those companies. I actually think many of them wanted to have a fixed price thing.

But if they have a fixed price thing, then they're in the business of managing how much capacity and goodness you can get out of that fixed price. I actually think some of those companies got a bit ahead of their skis where they were offering fixed price deals. And then when you did the math of how much token capacity you were getting, the Foundation Lab was losing a lot of money on certain people.

At least I am one of those people because I would look at it. Oh, 100%. Me too. So I think they themselves had a sustainability challenge. And then understandably, they're like, okay, well, let's switch to consumption pricing where then our customers, which is like a company like Databricks or others, can then internalize, okay, where is the ROI for us? Like, how much are we willing to spend compared to how much we're getting? And that has a lot of benefits to the ecosystem.

But it has one problem, which is it puts the onus on you, the buyer, to figure out how to do cost management. And incidentally, a lot of the solutions we looked at at Databricks, we ended up building also products that our own customers, we productized basically a lot of the stuff we built. So others are welcome to use those products that Databricks now offers.

It's our AI gateway. But before we productized it, we just looked at how to actually solve the problem. And it passes the responsibility for ROI to you as the user. And so we thought a lot about then how do you as the user handle that responsibility, basically. Yeah. So in your blog post, you identified a few different levers that are available.

Do you want to take us through that high level? And then we can dive into each one subsequently? Yeah, definitely. The real world is obviously very complicated. But I had to make this simple in the way that I thought people could understand. And if you really simplify things, at least as it came to like the short term mitigations that were available to us, we looked at four things.

Number one is that the single biggest lever that I've seen, and this has actually become even more true since we wrote this post, is that there's this beautiful thing happening of constant price reductions in models. Meaning to say, now, even at one moment in time, there's models that are more expensive and have better performance. There's models that are less expensive and have worse performance.

There's always that I often call it like a Pareto curve for those who are familiar with economics. But it means to say that there's this continuous tradeoff of quality versus cost. But nothing is more amazing than when that whole curve shifts in one direction, meaning to say, a new class of models have come out that you could view it either way.

Either you fix the complexity of what you're doing, and then that got a lot cheaper. Or if you have a fixed budget, you can do a lot more. But it means to say that there's a deflationary thing where costs go down. And the way that that happens in the AI space is that some new model will come out that expands the frontier would be the economic term.

But it dominates in some ways, the models that were there before in terms of usually what people target is, hey, it can do roughly what we could do before, and it's cheaper. And when you're driving a massive token volume, even getting 20% off a huge cost is a really meaningful thing. But there's a lot of secondary things that we can get into when we go through the detailed list around how do you put yourself in a position where you can take advantage of that? Do you want to go into that now or you want me to finish? I was getting too excited about this.

So I want to know this is good. And this is it's a really interesting one, because classically, you have this with a new technology where it's pushing the boundaries of something new and people keep chasing after that. And I keep expecting us to hit a point where we saturate our need, right, where it's like, oh, you know what? I mean, in some ways, I thought when we hit Sonnet 3 .5, even before we got to like Opus 4 .5, Sonnet 3 .5 was the first model I played with that could write solid code reliably.

And I was still driving it by hand. I didn't have the agentic harnesses we have today. But I was like, holy smokes, models could get no better. And this would change the world of software engineering. And models keep getting better. And Sonnet 3 .5, you can't even get any more. But if you could, it'd be like, so cheap, it's unbelievable. And I thought, okay, well, are we going to keep going? But it feels like we keep chasing after that frontier, right? People keep adopting, oh, why would I use that model? I could use Opus 5 now, or what have you.

So like, where is that efficiency frontier? Why have we not saturated that for at least coding? Yeah, I mean, it's a great question. Like, there's sort of different things to tease out in your question. One question is like, how come they can keep making these models better? Isn't it getting harder to make it better? And the truth is, they're just getting better at training them.

Like, first of all, software engineering is a highly verifiable domain, which really helps. Because they can, you know, throw RL at it, and they can get into these self -improvement loops. They also continue to throw more compute at the problem. Data is, I think, pretty saturated from what I understand, although I'm not in those companies.

So I wouldn't consider that authoritative. But it's just between throwing more compute, clever techniques, optimizing heavily the inference and the training stack. Companies are even looking at building their own hardware. They just keep squeezing and squeezing more out of this quality of model per dollar. And there's almost some Moore's Law equivalent thing happening here that I don't know if it's been measured.

But the quality cost ratio just seems to keep improving. And that's why it's so beneficial to have the agility to switch between models. But then I think a second part of your question was, by the way, I realize we're doing like depth first search, because technically I'm supposed to be going through the four ways to save money. But the second part...

Yeah, you know, following interest. I think let's go depth first, and then we can pop back up the stack. Okay, cool. So the second part of your question is about, or maybe what I heard maybe was, aren't we as developers at some point, aren't they kind of good enough for the things we're trying to do? And there, I think what's sort of remarkable is that developers are just able to hand off more and more complex tasks.

So in some sense, they were good enough for the old stuff, but we were thinking of new stuff to give them. And that also seems to be... Like there's this famous quote, I don't know if it was Bill Gates or someone was like, why would anyone ever need more than 1000 kilobytes of RAM or something? It was this famous... Fair enough, yeah, yeah.

Do you know who said that? I don't even remember. That is a quote. Something like that is a real quote. And, you know, someone counted the table about... IBM of who would need more than five computers in the world? Like, we keep making this mistake. Like we keep making this mistake, right? And when you look at other analogs, like how much bandwidth does your mobile phone have now compared to 10 years ago? It's been like an exponential increase.

And people just find new stuff to do. Like FaceTiming with another human, like with near instant, super high -def video communication. We never would have even thought of that as a thing you would do on phones 10 years ago or 15 years ago. But when capabilities improve and prices go down, people just find new ways to use them. Yeah. No, that's definitely true.

I do wonder, and maybe this pops us up the stack and goes to the next one, right? Like some tasks don't need all of that horsepower. And so are there ways that we can decompose the types of work we're doing so that we can keep those costs under control a little more? Yeah. So definitely, like as an example, just to move to a different context, before these enterprise coding stuff started really working, the big hit in AI was personal chat stuff.

And that was itself a very disruptive and innovative product form factor was like, oh, I can just pull up my phone at any time and I can ask any question I want. And it turned out that although the models keep getting better for that, the set of things people kind of want is sort of saturated. Like even very low -cost models can answer the vast majority of individual questions.

Because most questions are like, where can I get a hot dog in my neighborhood? The frontier labs are really excited about proving new math, eros problems and stuff. But like most people don't ask those questions day to day. So in that capacity, in that form factor, my view is there has been a bit of saturation where I actually think the main thing happening now is that costs continue to go down.

Rather than that, there's lots of higher quality models that matter for the types of questions people are asking. So then the second big wave in Gen AI has been these coding and other enterprise workflows. There, I don't think, as I said, I don't think we've hit saturation. I think there's far more difficult coding stuff that they still can't totally handle.

But especially when it comes to architecture or high -level system design. But it is absolutely true that for like routine coding tasks, hey, go refactor this thing. Please add this simple feature to a JavaScript code base. They're much lower cost models. Even the last generation of models will perform identically to a very, very expensive frontier model.

So there is this cost gain from sort of fitting the model to the problem. And there's different ways of doing that fitting. One way is the user themselves can do fitting. You can ask the user. Now, in a small company where people are policing their own costs, maybe that works. In a big company like Databricks, where no one is themselves paying the bill.

Understandably, it's a little hard to convince people to constantly switch to crappier models. My incentive as an individual developer is just turn the dial to 10 and get out of my way. So we do a little bit of downshifting on purpose or how you can control this when you're doing things at scale is the default model maybe could be not the most expensive model.

So if you want to shift up to it, you can. We also prune the set of models, meaning to say, even the frontier labs continue to release updates to their models that are not better than the old models. Like if you look at how they train these models, it's like more art than science. I mean, they put a bun in the oven and they're kind of seeing, does it look like it's burning on this side? But they're not quite sure they're doing evals as it's being trained, but they don't quite know what they're getting.

Well, and they're very spiky, right? So like soul is phenomenal for mathematics, apparently. But like my engineers are like, I don't want this for coding. It overcomplicates everything. And by the way, like we had one of the Opus, like I think many people were using Opus 4 .6, Opus 4 .7 came out and we deployed it. And well, actually, I think first we did some evals, we did some benchmarks.

And on our own benchmarks, we're like, huh, this seems like it's more expensive and lower quality. Like it's the inverse of the Pareto. It's like both things got worse. And then we like tried it with the pilot group. By the way, I don't want to get in trouble. I don't know if exactly 4 .7, I think it was, I mentioned it in the blog post. And it since got better with the next model.

But there was one that came out that was like worse along both dimensions we cared about. And so there, it was actually an easier decision. It was like, oh, we're going to skip that one. Like we just didn't make it available to our users. And when our users asked and we just explained, hey, it's not so great. Then there's other models where like, okay, that model is good in certain circumstances or for certain complexity of tasks, but it needs to be this fitting thing that we were talking about before.

And there the levers you have are like, ask people to do it on their own. But increasingly what we're using is like dynamic routing, basically, where we try to decide what model to use. And there's a few different flavors of that. Sometimes it's done at the task level. So we might for a whole session, for instance, put someone on a particular model, or there's certain interfaces where someone just submits the unit of work.

It's not an interactive session. So then we can decide like based on that thing they submitted, what model do we want to use? There's also much finer grained routing where even request to request you might divide across. This is looks like traditional load balancing in a server system. That one is trickier to get right. It's more powerful in some ways because the end user is even further abstract.

They don't even know what model is really being used. It's trickier in that, although we can model inference as like a stateless problem, there actually is some sleep. I was going to say like the key value cache feels like it's going to be a tricky thing to optimize. The cache is really important to economic performance or to economic cache reads are like, I don't know, 10 times less or something than non -cache reads or something of that nature.

So you have to be very deliberate about routing. When is it worth it to develop a new cache to get a much better fit for the economics for this particular problem? And we already have some efforts in this area and our gateway supports it. But I think this is a very rich area of research, to be honest, is like, how do you decide when to switch based on a variety of how much you can save and how much will it degrade the performance for the end user, if at all? And how much am I losing that cash benefit? You need to keep all these into account.

Yeah, it's fascinating. Well, and how are you classifying like what types of tasks? Is it going to be good enough anyway, even if the economics work out? Yeah. So that's also a hard problem. Like, first of all, even humans aren't good at that problem. This is the number one thing that people say when we're like, hey, choose the cheaper model.

And they're like, I don't know which model is good enough for this. That's why it's really tempting to just stay on the most expensive model all the time. So the answer to your question is we build models to do that classification. So that classification of which model is good enough for this problem is itself an AI problem. And is itself amenable to training models that are classifiers that are also LLMs, basically, they're very different type.

But I do think it is in principle learnable, not with 100% confidence, but it doesn't need to be 100% confidence. It just needs to be a high enough hit rate. And it is in principle learnable. Well, especially in software development, because as you highlight, it's very validatable in terms of outcomes in a lot of ways. Exactly. And this validation is a little trickier because it's like, oh, was that model in practice good enough for this task? And that can sometimes be measured just completely objectively like, oh, the pass rate on some tests, or we ask a much smarter model to evaluate the solution.

But there is a little bit of like eyeballing that still happens here. So in my view, one of the things we focus on at Databricks a lot is like, we're not in the business of ourselves building these massive, massive foundation models, but we are in the business of how the models get used in real companies. And I think routing and coming up with models that are good at routing is a very important problem.

And it's one that we're looking at and have some ongoing research in. But I think I believe most companies should take AI into their own hands and have different models they're choosing from and different vendors they're working with. And part of that is being able to know how you can switch in between them, basically. For sure. This portion of Software Engineering Daily is brought to you by Origin.

Your coding agents can launch each other. Theirs did. An engineer asked Codex to pull data from staging. Codex had no connection there, but Cloud Code did. So Codex launched Cloud Code. When Cloud Code said it lacked permission, Codex relaunched it with the dangerously skipped permissions flag. Nobody was asked. A permission check doesn't mean much if another agent can switch it off.

Your logs might show a process that started with a flag. Their trace showed the whole job and why Codex did it. Origin finds the agents on your company's endpoints and reconstructs their work from request to outcome on one timeline. Origin is endpoint AI observability. Read what they found in their own traces at originhq .com .sed. That's O -R -I -G -I -N -H -Q .com .sed.

Think about your mobile app's source code. Once it hits the App Store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs.

Their advanced tools integrate seamlessly into your CI -CD pipeline. We're talking polymorphic multi -layered code hardening techniques and automated runtime application self -protection paired with mobile application security testing and real -time threat monitoring to deliver the highest level of mobile security without compromise. Don't leave your hard work exposed.

Secure your mobile applications today. Go to guardsquare .com to learn more. Sponsored by Deepgram, the leader in voice AI, so you can use voices like mine. So staying on routing for a second, I feel like there's a couple different types of routing that you have alluded to here that might be worth digging one level deeper. So you've talked about gateway level routing, which is like here we have a particular inference coming in.

We can make a decision using a classifier or something else and route it to a particular model. Then there's harness level decision making where you're actually like something is deciding, oh, I want to explicitly route this in this way. That something might still be an LLM or sub -deterministic logic, but it's in the harness layer that knows more of the surrounding context rather than just here's my current inference request.

From what I could tell in your blog post, you're looking at both of those. Do you apply them in different ways? When are they relevant? How does that look? Yeah. so we use both And if you want to ask me in two years where will it end up, will almost everything be a fat client? Or will there be proxies? I don't actually know It's going to depend on how well this stuff works and various other factors But we use both The nice thing about the proxy or gateway -based routing Is that your client is out of the loop and doesn't need to even be routing aware in some sense It just seems to be the case that this proxy speaks the same inference API to that client And you can just

swap it in with no assumptions about your client You may even have an older client that doesn't know how to do routing or whatever Also, it depends on your control model But like in a company, your end users are mostly trusted But for things like cost controls, you might want to sort of have ability to coerce activity And therefore doing that in a server rather than the client is a bit easier Because it avoids creative employees realizing that they can tweak settings on their client and get around rules And for instance, things like entirely Let's say removing access to certain models because they're not efficient models and we don't even want them in our set Or maybe

some of our users accidentally run like runaway spend They have a loop going they didn't even realize they left it on overnight. It's not doing anything useful We have like a thing that will torpedo that and it lets them reset it if it was truly intentional So those are much easier to do on the server side It's transparent to the client and in some sense the client just doesn't get to make the decision Which is sometimes a feature However, there's a lot to be said about doing this on the client side too One is as you said like the client knows much more Like it may know much more about what's going on It has tighter access to the whole history of the session for instance

It may have other context on what type of job it's being used for So I do think potentially you'll see a lot of intelligence happening at the client side as well There's also this further evolution of client routing which is like More like dispatching between harnesses And this gets into a sort of slightly nuanced thing which is that There's a increasingly a coupling between the foundation model and the harness the choice of harness And when I say one model is better than another model or so forth I'm sort of with simplifying a fact of the world which is that Model harness pairs are really what you need to look at So a particular model with a particular harness may be

Very expensive or very cheap and so on so forth And another class of thing you can do on the client side that you sort of by definition can't do on the server side Is you could even dispatch between different harnesses? Like there's a project an open source project we've released called omnigent at databricks And it's a multi -harness dispatcher basically And the idea is you do your coding against a fairly uniform interface behind the scenes It can leverage other harnesses Sometimes it's even beneficial to like have an adversarial relationship between your harnesses One of them writes code and one of them reviews code And it tends to be the case that when you review code

from a different model than wrote the code You get way better feedback Models are very bad at reviewing their own output They give their own output high marks And it's not surprising because it's all a matter of what statistical distribution they come from So for even other reasons than just costs There's nice benefits to having these like multiple harnesses that are playing with each other on the client side But it's not even just client versus server side It's also like as i've said it's like server side versus a client where maybe within one harness you're routing to different backend models And then there's even this next layer up of a meta harness that's routing

amongst harnesses And I don't know in the very end of the day if you fast forward where most of the logic will be But right now we're employing different types of logic in all of three of those places Yeah, no, it is fascinating. I remember I think it was the early gpt5 series Folks were trying it in the context of cursor and seeing incredible results and trying it in a different harness and finding Really running into trouble and i've recently been on pi and comparing that to different things like the harness makes a larger impact than we like to think Totally and for instance, I think databricks also regularly we do a lot of evals of our own Because we have this problem

of like, you know, I told you some models come out. They're actually not better. It turns out Well, the way we used to find that out is we would roll out the models to all these developers Okay, like thousands of developers Then the complaints would start to come in and by the way everyone always has some complaints about every model So you'd have to wait until you have enough signal to noise.

That's like, okay This actually is really a problem and the cost we get kind of an a b test That's pretty quick because like you'll just see costs go up or down based on the cohorts that move But the quality is hard was initially quite hard for us to gauge and the downside was that we would waste like weeks with a crappy model and And then eventually someone said hey, come on look at all these people complaining this model really does stink So then we roll it back and we would remove it from our working set basically But the way to fix that we found was like you just need to have good evaluations.

You yourselves can run And so we ended up building our own Evaluation suite based on our own code basically like it literally asked these models to go contribute code to databricks the code base We obviously have a point in time snapshot of our code. That's before we went and actually added all those features And then we run these offline evals and we want to get to the point where like within an hour of a new model coming out We know if this one is sort of a candidate to try internally and that offline benchmark rules out a surprisingly large number There's so much I mean every week now we're running that multiple times between open source models and proprietary models

like Every week we're running it and we want to get down to a set of three four or five models We're using that define that pareto frontier and so we needed to get the speed of Evaluating whether models were even worth considering down and so we ended up making a benchmark We also publish the results to that regularly although I don't want people to over extrapolate that's based on our code base So there could be some fitting going on But one thing we found when doing that benchmark is the model plus harness pair needs to be benchmarked and you'll see even drastically different results By the way pi has suddenly become very popular at databricks also I guess you guys are

using that like there's this constellation of open ones that are getting pretty good I mean i'll just say pi is incredibly hackable so you can customize it to yourself and build extensions and it knows about its own extension system But that compounds your harness plus model problem because now it's model plus harness plus the set of extensions or other Skills that you've incorporated etc Yeah, so we have to cut it off somewhere and like i'm sure that There's probably some minor regressions where someone actually gets worse for someone because we didn't test their particular configuration But we do quite a bit of this offline testing mostly to save the human time of it's

just become impractical to test each one with humans There's too many of them for sure I want to dig in on an implication or a thread you mentioned these are on databricks code And so it's different than other other things and one of the things that i've definitely seen is Code organization and its style has a dramatic impact on How effective lms are writing that code and how cost effective it is? I'm curious like Is that something that you're able to see in data in terms of like oh if we use a service oriented architecture or this language tends to be more efficient in terms of things or stuff like that where you can have even that meta level of feedback of Hey, if we

write services primarily in I don't know i'm just going to pick go as an example That tends to be much more cost effective than writing a monolith in python or something like that Yeah, so that's a great question We don't have systematic research benchmarks that look at that problem and I think the problem if I understand it is Are there ways you can write your code that make it more amenable to being highly efficiently? Modified by models and the only answers I have for that are of the form of my intuitions Which I do have some like certainly The distribution of code written in different programming languages on the internet is like almost a power law I think there's

some extremely popular programming languages and there's a bunch of very obscure ones Databricks started, we have a meaningful portion of our code base is written in Scala, which is not on the very extreme end We've actually over time more and more moved to those pieces of code to Java But we started Spark in 2013 or 2010 when Scala was super exciting and I still think it's a cool language But some of the useful features of that have actually been moved into Java at this point So a lot of our go -forward work is in Java and then also we have a huge amount of Python and, you know, Rust and C++ in certain areas But I can tell you LLMs are not amazing at Scala, but that's

not a useful fact for most of your audience I think the general principle is more useful Maybe that the biggest and most popular languages just tend to be the best because they have the most training data on those languages Python is probably the single best For instance, we did have an episode in one of our own products where we have certain internal parts of Databricks Where you can call it via Python or via SQL Behind the scenes, it gets translated into exactly the same representation We just have two front ends for it And wouldn't you know it when we switch that the code generated portion from SQL? This is not ever exposed to our own users, but it's just an internal

representation when we switch that from SQL to Python Everything got way better even though just because the AI is better at generating Python than it's at generating SQL and I would say SQL is a fairly popular language too So I think Python dominates in terms of like quality of generated code, but it's getting so good now They're probably all pretty good And then in the teams that have started new code bases where they're able to also innovate on how the code base looks.

I have seen them focus much more on interfaces behind which they could build things that were highly verifiable. And I realize that's like a very abstract statement, but in some sense, the key to being able to trust the LLMs more was make the scope of their changes smaller and more verifiable. In a big legacy code base, that's difficult. In a brand new code base, you can think about modular design principles where you have, by the way, this is probably what would have anyways been good for humans.

This is one of the fun things here is like what's good for LLM coding is actually the stuff we probably should have been doing. Yeah, but the ROI from doing it for LLMs is even much, much higher. So you'll see some very like flat types of organizations where it's like, okay, you know, I actually built an interface and there's 50 different implementations of this interface because each one was able to be like super easily verified when for humans, you might have imagined that same thing would have been written with like a slightly more monolithic structure.

So this is in the territory of my intuitive takes. We don't have robust research findings in this area. Yeah. I feel like this is one of the fascinating things to me of like, what are the characteristics and some of the characteristics that make something like Scala, for example, good for humans may not actually be there for LLMs. So one thing we didn't talk about, but a thing that's true of both Java and like Python, I've seen it in Go, I think it's true in particular logic is coming from Java.

If you go heavily into dynamic dispatch, maybe, maybe not. But I think Go is a great example where there's a very linear dependency path where the LLM knows exactly where to pull the context in to understand what something's doing. And it doesn't have to like dive through a deep dependency train or load a whole bunch of context to be able to do it.

And I think that sort of linearization. I do agree. Like Scala code has these implicit things where it's kind of like duct typing. It will like become a thing if it meets the signature of that thing. But I actually suspect what totally dominates all these factors is just how many lines of code on the internet are in this language. Fair enough.

Yeah. I think that's just how these LLMs learn. Like they're pretty good at learning almost anything. They just need, they just not, they learn very efficiently compared to humans. So they just need to see a lot, a lot, a lot of examples. And my hunch is that that would be the thing that differentiates the most, but there's certainly languages where there's more implicit stuff happening.

And that probably also makes it harder for the models to learn. Let's dive into some of the other things that you've kind of learned in terms of rolling out AI data breaks. Cause I feel like every company right now is trying to figure out how to effectively roll out AI. And as you highlighted, there was this like top down push. And, you know, I think you were generous, honestly, in saying enterprises were going through that, you know, up until 2025.

I keep hearing of enterprises that are still in this like top down, how do I get people to use the tools space? But like, what have been the important challenges and what's been really effective in terms of, you know, getting these things to work well for your org? Yeah. There's many facets to it. Like the types of things that we have thought about is like, first of all, how many different AI tools do I even give my employees? And like, can they use any tool they want? Or I should just give them a few or should I choose two? And then do I give the coding tools to like the non -coding people? Because, you know, even my marketing team wants access to codecs or something and

they don't know how to code, but they, but it can build these apps for them that they're like super excited about. Also my inference data, I'm attaching these tools to all my most sensitive data. And then that inference is getting like logged by some third -party company. So now they have all my sensitive. So there's like so much around, not just cost, but like which tools you deploy, how to deal with data security and like, where is your data going by virtue of these tools? Who gets what tools do you, do you go all in with one vendor or do you sort of have multiple vendors? And I can share maybe some insights we've had, but I don't think we have all the answers, but we

felt strongly from the beginning that we wanted to be a multi -model, multi -tool company, at least for R &D, which is where I focus a lot of my time. And so we really did not want any scenario where, by the way, it doesn't necessarily mean that an engineer can find some AI startup that's existed for one day and, you know, buy their tool and give it all of our sensitive data.

That's not what that means. What it means is that we have a set of pretty popular tools. We let engineers vote with their feet and use the tool they want. And we want to provide internal choice around tools and models. By the way, Databricks, you know, our product is really for data users. So it's not a general coding tool, but we also in our own product have capabilities as well that are consumption based.

And, you know, you're also seeing also a lot of embedded AIs now inside of different products, including ours. And our thought was like, let our internal users choose what's best and what's best seems to be changing every month. So that's one reason to do this. But where we insist on uniformity is we do have a narrow waist in the company.

And that narrow waist is this thing that we built internally called our AI gateway. So basically all tokens go through that gateway. So we can see every inference that's happening from any random tool that's in the company. All data access to internal systems also goes through that gateway. So it acts as a proxy basically to, you know, an MCP proxy effectively.

So you want to access Google Drive, you want to access Salesforce, you want to access anything. It's going through that proxy where we're doing authorization. We actually log everything. And also we insisted on zero retention for the people providing us the models. So we don't want, you know, the whole thing you do with these AIs is you give your most important data so we can help you with your most important problems.

So the inference request that goes to the model foundation provider includes all of your extremely sensitive data. You might have customer names or PII or credit cards, you know, we're a serious big enterprise. So the rules we came up with were like the end user tool choice we want some flexibility on. We want to have a set of models that we can always be adopting the newest ones and making them easily used by our internal users.

We need to have some sanity. And the way we did that is we run everything through this thing we call an AI gateway. They're sometimes called LM gateways. There's a few, but now they're expanding in scope. So this is kind of an emerging architecture, I think for any company that's big enough, you're going to want a central control plane for this stuff.

And we ended up productizing ours. It's called the Unity AI gateway. There's other companies that also offer them, but whether or not you use the Databricks one, I think at scale, this is kind of necessary. And if nothing else, it's necessary for being able to quickly adopt cheaper and better models. You know, the truth is the Pareto curve.

If you look at the Pareto curve today, I don't even have the most recent one, but we run the evals every week. On there is probably one open source model, one model from Anthropic, you know, one model from OpenAI. And occasionally now even like the there's the meta ones are starting to get good. The Grok ones are starting to get good. So we're in this period where there's more and more supply and it's a huge cost.

So just basic economic theory dictates like maintain option and suppliers, you know? Absolutely. Yeah. And so I think for us, that was a principle that really mattered was like tool choice, not amongst any random tool you can put in your credit card, but we'll try to basically provide the popular tools to engineers and let them choose even day to day they can choose.

But we do insist on like the access needs to go through our gateway so we can control and observe costs and do model routing and stuff. And any access to our internal data needs to go through our proxy basically. And then we insist on zero retention for the model provider. So those were like the three things that I was like, that's my religion.

You know, unless I get a really strong argument, that's my religion. Almost everything else is let's experiment, let's be flexible, how we organize the code base, how we organize teams, like those things are all ongoing experiments. Let's talk a little bit more about that control plane and what goes into that because you mentioned a few things you mentioned, all tokens are going through that.

And that then allows you to do routing, it allows you to do audit logs, it allows you to do cost analysis, a bunch of other stuff, and then internal data access. Is there anything else that's flowing through there? Or are there any other mechanisms of control or places you're either locking things down or titrating? First of all, data access is quite nuanced, and we have very sophisticated policies.

So the way I think about it is like this. We already kind of had the problem in enterprises that sometimes I would want to allow a piece of software to act on behalf of me as a user. That's already been a thing that was a problem. And it was solved with something called OAuth. I mean, there's different approaches, but really the approach that dominated was OAuth, which was a delegation model.

It says like, I'm okay for this app to be Patrick, you know, and there's relatively reasonable token exchanges and so forth to avoid long live tokens. And, you know, it's a lot better than like, I send that vendor my password to that thing. And you can ask for particular capabilities, though I've noticed the foundation model companies love to ask for everything.

Yeah, well, I'm getting there. So we're still in the old days. Okay. OAuth has a thing called scoping. It's up to the vendor to implement scopes, but most reasonable applications have some notion of scopes. They have tended to get better over time. So I don't know if you've ever set up OAuth with GitHub, like that first they just had like very, it was like a read and a write or something.

And it's like, okay, well, I want much more fine grained scopes than that. But it's actually very hard if you implement OAuth as a vendor, which we also have, like, it's kind of hard to implement new scopes. You need to propagate that throughout your whole app. So OAuth scopes tend to be fairly coarse, but they've gotten better over the last few years.

But now there's this other thing called like an agent, and it's not quite the same as a vendor app. And I kind of want that agent to do things on my behalf. But OAuth is like, it's sort of necessary, but not sufficient. Because the truth is, if I have 100 agents running around, I don't fully trust them as much as I trust myself. Like, I don't think the company should trust them to be exactly equal to me.

They need to be put in much more of a constrained set of capabilities. And the problem is we're stuck with OAuth scopes, which are okay, but they're very coarse grained. And they're also static. It's like, there's an OAuth scope that I give to, you know, all of Cloud Code or Codex or, you know, even the Databricks agent or whatever. But like, it's not a thing you can dynamically adjust or adjust in a super easy way.

And agents look more like people. And the way that historically, this problem has been solved, like ignore software systems. Okay, here's like an analogy. There's credit cards running around. Okay, this credit card is a delegation point that I've said, like this card represents me and my bank accounts. But sometimes people steal those cards.

And they start to do bad stuff. So how do credit card companies decide whether to authorize a transaction and know like, okay, is this a legitimate thing that the person wanted to have done? Or it's a thing that they didn't want to have done? Well, it turns out they have a highly dynamic and sophisticated modeling system. So if it's like they risk score every transaction, if a few risky ones, some might be so risky that they just block it, some might be risky enough that they put it on warning.

And if another one comes, they block it, they might dynamically decrease the credit limit. But these agents running around need to be modeled more in that way. And so one thing we do in our gateway is we're dynamically looking at what agents are doing and looking at risk associated with their actions. Because just saying, oh, you can have read access to Google Drive.

That's a static Oath entitlement that will sit around for a long time. I don't think they use the word entitlement. It's a scope, I guess would be the right word. But what if I deploy an agent that suddenly starts scanning our entire Google Drive and downloads 10 ,000 documents? That is a risky behavior that warrants attention. And that warrants attention of the form of like, cut off that agent and email the person who deployed it and ask if it was intentional or not.

I get on my phone, the credit card, like, did you mean to buy this gas in like, you know, Nevada or something? I'm like, no, I didn't mean to do that. So you need to profile the behavior. If you want to have good security, it's not sufficient to just say, oh, I have like a lot scopes. And so I'm done. You need to like look at what they're doing dynamically.

And that is a function that we enforce at the gateway layer as well. And you can even set policies like you can set policies like you can have access to this particular Google Doc, but you can't just because Oath is too broad for that. Oath doesn't support the protocol to support these extremely fine grain entitlements or this dynamic sort of risk scoring based approach.

You're like laying out all my frustrations with working with agents and Google Docs right now. Like you've got, yeah. By the way, a lot of them don't even support like read only access. Like that's the most simple one you should have had from like 1995. So your question was, what else do you do? Is it just like gateway for just like cost or something? And I would say the two really big functions are like cost stuff and security stuff.

And security stuff actually gets very, very nuanced. But my view is if we want to tell people to go nuts with agents, we need to like make sure that things are being done safely. And I think it's very hard to ask every individual person in a big company to conceptualize what safe means. We can ask them to do some things, but some of it needs to be enforced in a central location.

So like by intermediating all access, we have the ability to log stuff. We have the ability to start looking at potentially weird behavior. I sleep better. I tell them it's my blood pressure, founder blood pressure, you know, project. Like I sleep better knowing that when inevitably, like these agents do unaligned things. They might not do really severe unaligned things.

Like they're going to, they're not going to go launch a cyber attack, hopefully. But they might think, oh, someone wants to find something in Google Drive. Let me download everything in the Google Drive to like a public URL, you know, pastebin or something. Like they'll post stuff on the Internet that we don't want posted on the Internet, you know, because they'll think, oh, it's very easy.

I can copy something to this public endpoint and that's the easiest way to accomplish this task. So if you start to really observe what the agents are doing in practice, although there's not severely unaligned behaviors, there's like medium unaligned behaviors, even in like a normal company deploying frontier agents. So frontier meaning to say this is like from OpenAire Anthropic or, you know, Google, like these are highly safety post -trained models.

And those occasionally do stuff that you kind of would be like bad security practice kind of thing. And so the only last thing I would mention, which is it couples with the gateway, but it's not quite the same piece as the gateway, is we do also like network sandboxing and other types of sandboxing of the agents. So so not only do you have to go through us to access our enterprise systems, you can't go anywhere else.

Like you can't just call out to the Internet easily from one of our development environments. And we constrain what they can do. And in many cases, they're running in physically isolated, which, by the way, the AI companies can't seem to figure out how to do a network sandbox. I don't know, because they're having their agents escape them all the time.

But to our knowledge so far, none have escaped our own development sandboxes. Yeah, I feel like sandboxing is like one of the big pieces here so that I understand everyone at all of your development environments that are cloud based or you're shipping around these little like sandbox environments? Yeah. So first of all, we primarily have like cloud dev boxes at Databricks, which is we've had it for a while.

It has a lot of benefits in terms of being able to more elastically scale or let people do. Maybe I'm compiling my code base in four different branches and I don't want to, you know, my laptop to get too hot. But we do have some people that will for certain things develop on their laptops, in which case we actually have a local sandbox that we use as well that ships with Omnigent.

But most of it is happening in our cloud environment where like it's already running in a virtualized environment. So that thing is just hardened with pretty restricted network access. Well, and so we've talked a lot about like sort of current layer of challenges and what you're tackling. As you look out, I don't know what the right forecast length is these days, because as you highlight, like the Pareto frontier is shifting every week.

But say the next three to six months, what do you see as the big challenges that you're focused on in terms of AI usage at Databricks? Yeah. So I think this is an excellent question. I think first was, by the way, I don't want to sound like some Debbie Downer. AI is like helping us in immense ways at Databricks. It's awesome. We're moving way faster by every observable metric In our development stack We're shipping more code We're shipping more features You know, our revenue growth has actually accelerated In the post -AI era Which we've talked about publicly So a lot is going right But as the leader, when the stuff is going right That's like, I don't need to do anything

You know, I need to come help when there's challenges Yeah, that's not what you need to worry about Yeah, exactly So I should qualify all this by saying Like it's been an abject success Our adoption of AI There's just a lot of tricky things to deal with Which are the ones we're talking about here In terms of like upcoming challenges I feel that we The costs are feeling much more under control at Databricks Based on this set of techniques we've done I would say that's an evolving situation always Like so far what's happened is that The set of things we want to do Just keep getting cheaper, which is great If some new set of capabilities comes That has some really high costs

associated with it But we see huge ROI Then we'll have to reconcile with that You know, but so far that hasn't happened A lot of what I focus on now is like Making sure these teams are actually as productive as possible now With the AI And it's quite interesting Like the models themselves are just amazing reasoners now It's very rare that they will make any kind of error In that they have a premise and they have conclusions And the premise does not follow from the conclusions Like they will never make that form of error And in fact, I would say they're smarter than me in many cases And I think they're smarter than our median employee Which I guess implies that I'm smarter

than our median employee But let's say I'm in the median Let's say I'm at the median Okay, they're smarter than me But it's quite interesting because Despite that Even though you have like In principle you have an army of In theory we could have 10 times more employees than overnight Because we have these people that are just as smart as the people that we have at Databricks So like what is actually Not quite working where we're not getting immediately those gains We can't just add 10 times as many agents and we have 10 times as much output So what is broken? Like so that's where we encounter every other type of problem I would say that the number one issue is that the

models not having the context that they need To get the job done And you know it's funny because people thought We're shocked that models were good at coding Because coding is a very specialized skill in the world Like for instance there's like way more people that can click around on a computer Than can write code but the models are really bad at clicking around on the computers And they're good at writing code that doesn't seem quite right By the way they're getting better at the computer thing But even you know Databricks does this We haven't built an AI in our product that's not primarily focused on coding It's focused on like asking questions from your data And in

many ways like asking questions from your data should be easier than coding Because there's so many more people that know how to use excel or whatever Than know how to write code But it turns out it's really hard And it's hard because it gets into the set of things that can't be sort of like verifiably Hill climbed for the models And it involves a lot of surrounding context and nuance That just like can't be baked into the model because it's very specific to each company So coding although it's intellectually difficult It has this amazing property A code base is self -describing It's a bunch of files that are self -describing It's designed to be hermetically sealed We've

spent decades figuring out how to deterministically validate the outputs of non -deterministic inputs are coding Exactly so we like handed this to the lms on a plate The other sort of nice thing is like ever since git came around no code bases have ACLs on them So code bases have already one big problem in the data side is like Not every piece of data in a company can be seen by every person Okay and like you can't feed all the data to one ai and then suddenly it's telling It's leaking something from the ceo to like the intern that just started yesterday So there's this ACL problem now Merely by coincidence by the way perforce and these older source code management tools

did have ACLs in them But just merely coincidentally this tool called git took over and git has no ACLs You know the only ACL git has you're either in or out of the repo So that actually means that this whole problem of who can see what doesn't happen in the coding space Which also makes it a lot easier for the ai models So although coding is like intellectually very difficult Or at least for humans it's like a thing that not everyone could do it was considered a specialty Anything intellectual the models can run at what they need is they need like a They need all the context to be sort of hermetically sealed and and searchable And they need to have like verifiable

testing and they have all that stuff for coding So I find it actually kind of like unsurprising that they got really good at coding But it turns out that most people at databricks even though we have a lot of engineers They don't do coding for their day. They're doing other things The thing they're doing it might require less extremely sophisticated individual reasoning, but it requires a lot of context And so the thing that i'm most working on now Is how do we at the scale of a big company? I think there's this other layer basically like the layer I just talked about was this gateway layer that like will deal with security and And tokens and all this stuff there's also I

think like a knowledge layer It's like some would call it an ontology. That's what we call our products databricks But like you call it whatever you want There's a layer that's going to exist in any company of a certain size where it's not trivial to just like stick all the information from that company in a very small amount of context And that layer is gonna be the thing that helps the ai have the context they need to do the job And designing that layer is hard like it's not like a code base where there's a very hermetically sealed thing There's a sprawl of a million random things in a company There's data in slack.

There's data in the structured data warehouse like the data lake. That's what we do at databricks There's data in docs and there's data in people's activities and there's implicit knowledge that seems to only diffuse through conversations So like how do we get this all in one place and it respects access control? And it's not creepy like I don't feel like i'm in the company I have like people listening to everything i'm saying all the time That's a big big problem in enterprises and I think It's kind of the next frontier and and we're solving it in very There's very like small ways in which it's being looked at like for instance You can create skills and you can share

those skills, you know Those skills are supposed to encode some knowledge of this enterprise and what we're doing But when you have 10 000 people that doesn't really scale and by the way The skills themselves become the same thing they're messy and some of them are wrong and they contradict each other And so they just look like every other thing in an enterprise That's like the big other area i'm personally focused on is this How do we build this knowledge layer and how do we make it so that the ais are never stuck on? They're just not smart enough.

They're never stuck on that They're stuck on like they don't have the right information to get the job done They don't know how that job is done or they're missing access to a certain thing or they're they they aren't able to search and understand The user is asking them You know, like we have in databricks the unit of pricing we have is called a dbu databricks unit It's an opaque Thing that means like a gold token of databricks.

That's just like how all our pricing works every different product. We have normalizes into that unit Okay And so people will ask a is like hey, you know, how was our dbu growth in this sector or whatever? A lot of the models are just don't you know what dbu means? That's like an internal Yeah, yeah We have and by the way, there's like sort of slightly nuanced different versions of that we use internally and Finance uses one thing and product uses different thing actually Finance and product don't even totally agree on the definitions Of every derivative metric from this thing.

So like that problem is very difficult to solve and very messy as a data company We're very interested in that for our own products and for us internally But this is the thing that i'm most focused on next now that we're not gonna go bankrupt is like How do we actually make these ais better and the thing that they're stuck on often? I'm seeing is the context basically No, it's a fascinating problem and you mentioned skills right skills Are probably the best example right now of this technique of like progressive disclosure of context where it's like, okay I'll give you a little snippet that helps you dive in but I feel like you need nested ontologies of like How do you

build the map for the ai to even know what context it should be getting? Well, yeah, so skills have this hierarchical like, you know, the problem is there's a context budget in these ais exactly and so like Skills have they do have this like kind of nice property that and the way you write them is increasingly like to do this progressive disclosure thing.

But in my view, everything that a skill is accomplished in today should in principle just be learned by the AI. First of all, skills have the same problem with a lot of other things, which is they're human created and they're, that human better keep it updated, you know? No, they go stale really quickly. They go stale, like every other document inside of it.

So we need to solve the problem of knowing what's stale and knowing what's not stale. We already need to solve that problem because we need to understand what's in all these Google Docs and all these other things. So in my view, by the way, I think skills are very powerful. They're like way better than nothing. But I don't think the end state is all of us curating lots of things and desperately, it's like a wiki, half the things are out of date and they're doing more harm than good sometimes.

So like I think the end state is that there's a learning, a new learning problem, which is learn how this company works. It's like what you have to do when you join a company. And for the first year, you're like, you know, trying to figure out where the light switch is. Like totally. AIs in principle can also learn that, you know? Yeah. Well, and as you highlight, from what I've seen, AIs are really bad at understanding time.

And so they're very bad at understanding, oh, this is actually still relevant because this was updated recently. And this thing is actually probably less relevant because it was last touched two years ago. Yeah. So they need help. I mean, this gets into like traditional search type stuff. How do you know Google search had to solve this? Like there's a lot of data on the internet.

A lot of it's garbage. There's things that contradict each other. And there's ways to do this. You look at authority. You know, you look at recency. You tell it very deliberately, like prefer things that are more recent. Prefer things that are from users who seem to be authoritative users in this company. If it's a question about finance, maybe prefer something you saw from the finance team.

Then like, so these are implicitly like often called signals and like search ranking or the search space. And the same thing needs to happen here. And I think skills or other types of explicit annotation are kind of like, you remember the Yahoo directory? It was like. Yeah, exactly. Yep. Yeah. And at some point that thing didn't scale. It was like, look, I'm like dealing with a long tail thing.

That's not in anyone's skill, you know, like, or that thing is outdated or like, and Google just came and was like, oh, we're just going to go dynamically on all this stuff. And it was way better. That's a great summary. We're in the Yahoo directory era of LLM for enterprise. With respect to skills, the other stuff LLMs do are like complete magic.

And I don't think ever should be compared to Yahoo. Well, I feel like we could keep diving on this for forever, but we are just about at time. Is there anything we have not spoken about yet that you want to make sure we cover before we wrap? I do want to do one minute to plug. You know, we have our AI gateway that we made into a product. We have thousands of customers using it, you know, way more than that developers.

And we actually also have this product called the Ontology product, which is very similarly. It's like totally open. You could use it with other tools. But like we've thought a lot about this problem of how to encode that enterprise data. And so, yeah, give Databricks a shot if you're interested. Also, there's tons of like DIY versions of these things, too, if that's a better fit for people.

So I'm not overly trying to plug Databricks. But and then I would let people know that everyone is just figuring all this stuff out. And I put myself in that category. I think we've learned some things that I think will stay relevant for a little while. Even the things we've learned might become irrelevant. And the only way that we've been able to learn is through experimentation, just trying different stuff, seeing what works, seeing what doesn't work.

You know, finding colleagues and other companies, asking them for their advice, as I said, like several other folks helped us when we wrote that blog. And I think that like we'll just figure this out one step at a time. But it's a very exciting time right now.