Why Is a New AI Model Shape Cheaper Than Reliability?
Diogo Almeida says a new model shape is cheap and reliability is the part that matters. He still wants novel shapes and plans to keep shipping them, but people in machine learning are over-focused on the shape of the task. When he thought the work would take a week, he meant a week to make the shape, and that part did take a week. Other teams are now making similar shapes. The shape does not matter next to the model being useful.
He expects many more shapes than the two already out, even if Jev ships them alone. Too many shapes without utility burn goodwill, because the releases over-promise and under-deliver. He does not want creative energy spent on things that clearly do not work. He is against benchmarks for the same reason. Developers who cannot test every release get swept up, and a benchmark does not show how the model actually behaves.
Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·
From the transcript
I would say when we first launched this, I was really excited for novel shapes to emerge. Really, really excited. And I still think that that will be the case. I think new shapes are freaking awesome. We plan to keep shipping them. I'm a little bit worried that, again, what we talked about, maybe this was before we got on the record, that ML people especially have kind of lost the actual message I was trying to say.
And I'm worried that people are now over-focused on the shapes of tasks rather than the reliability. The shape of task is incredibly cheap. It's super easy to do that. When I thought this would take a week, I thought it would take a week to make the shape of the task, which it did. And this is what's happening now with other people making similar models with similar shapes.