Smash Notes

Back to Managing AI Coding Costs at Scale.

Why Do Teams Stay on the Expensive AI Model?

Patrick Wendell says people are bad at judging which AI model is good enough for a task. The number one reply to a request to choose the cheaper model is that nobody knows which model is good enough, so it is tempting to stay on the most expensive model all the time.

Databricks builds models to do the classification. Deciding which model is good enough is itself an AI problem, and the classifiers are large language models of a very different type. The judgment is learnable in principle, especially in software development, where outcomes can be checked. It does not need 100% confidence. A high enough hit rate is enough. A check can be fully objective, such as the pass rate on some tests, or a much smarter model can evaluate the solution. Some eyeballing still happens.

Databricks is not in the business of building massive foundation models. The work is how models get used in real companies, and models that are good at routing are an important problem with ongoing research. Most companies should take AI into their own hands, keep different models and different vendors, and know how to switch between them.

Transcript

Managing AI Coding Costs at Scale ·

From the transcript

Yeah, it's fascinating. Well, and how are you classifying like what types of tasks? Is it going to be good enough anyway, even if the economics work out? Yeah. So that's also a hard problem. Like, first of all, even humans aren't good at that problem. This is the number one thing that people say when we're like, hey, choose the cheaper model.

And they're like, I don't know which model is good enough for this. That's why it's really tempting to just stay on the most expensive model all the time. So the answer to your question is we build models to do that classification. So that classification of which model is good enough for this problem is itself an AI problem. And is itself amenable to training models that are classifiers that are also LLMs, basically, they're very different type.

See in transcript

Software Engineering Daily

More from this episode

  1. How Many Jobs Has Patrick Wendell Had?Patrick Wendell has had one job in his career.
  2. Do Cheaper Models Match Frontier Models on Routine Coding?Patrick Wendell says much lower-cost models, even the last generation, perform identically to a very expensive frontier model on routine coding tasks like a…
  3. Why Does Databricks Sandbox Its Coding Agents?Patrick Wendell sandboxes coding agents at Databricks so a model that takes an unsafe shortcut cannot easily call the open internet from a development…

More answers like this

The other side

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.