Back to Managing AI Coding Costs at Scale.
Why Do Teams Stay on the Expensive AI Model?
Patrick Wendell says people are bad at judging which AI model is good enough for a task. The number one reply to a request to choose the cheaper model is that nobody knows which model is good enough, so it is tempting to stay on the most expensive model all the time.
Databricks builds models to do the classification. Deciding which model is good enough is itself an AI problem, and the classifiers are large language models of a very different type. The judgment is learnable in principle, especially in software development, where outcomes can be checked. It does not need 100% confidence. A high enough hit rate is enough. A check can be fully objective, such as the pass rate on some tests, or a much smarter model can evaluate the solution. Some eyeballing still happens.
Databricks is not in the business of building massive foundation models. The work is how models get used in real companies, and models that are good at routing are an important problem with ongoing research. Most companies should take AI into their own hands, keep different models and different vendors, and know how to switch between them.
Managing AI Coding Costs at Scale ·
From the transcript
Yeah, it's fascinating. Well, and how are you classifying like what types of tasks? Is it going to be good enough anyway, even if the economics work out? Yeah. So that's also a hard problem. Like, first of all, even humans aren't good at that problem. This is the number one thing that people say when we're like, hey, choose the cheaper model.
And they're like, I don't know which model is good enough for this. That's why it's really tempting to just stay on the most expensive model all the time. So the answer to your question is we build models to do that classification. So that classification of which model is good enough for this problem is itself an AI problem. And is itself amenable to training models that are classifiers that are also LLMs, basically, they're very different type.