Do Cheaper Models Match Frontier Models on Routine Coding?

Patrick Wendell says much lower-cost models, even the last generation, perform identically to a very expensive frontier model on routine coding tasks like a refactor or a simple feature in a JavaScript code base.

Before enterprise coding tools started really working, the big hit in AI was personal chat: pull up a phone and ask any question. Models keep getting better, but what people want is saturated. Even very low-cost models can answer the vast majority of individual questions, because most are ordinary, like where to get a hot dog in the neighborhood. Frontier labs chase new math, and most people do not ask hard math questions day to day. In personal chat, the main change now is that costs continue to go down, rather than higher-quality models mattering for the questions people ask.

Coding and other enterprise workflows are the second big wave, and saturation has not hit. Far harder coding work still cannot be totally handled, especially architecture and high-level system design. Fitting the model to the problem creates a cost gain. In a small company where people police their own costs, asking each person to switch models maybe works. At Databricks, no one is personally paying the bill, so it is hard to convince people to constantly switch to weaker models. An individual developer's incentive is to turn the dial to 10 and get out of the way. At scale, the control is a little downshifting on purpose, and the default model maybe could be not the most expensive model.