How Far Along Is Jev's Image Model?
Diogo Almeida says Jev's image model is maybe 10 percent of the way the text model is. He would be truly embarrassed to call it production ready, because so much jaggedness is left, and it still lacks his seal of approval.
Text tasks will be automated before image tasks. Investment went really heavily into text tasks, with a tiny amount of work on image tasks by comparison. He is jazzed the image model does as well as it does. The lead in images is not as large as the lead in text, but he senses an overall lead in images and could be totally wrong. He calls the image model completely insufficient. It is not quite as bad as other people just shipping a shape of something. In the grand scheme the image model is maybe 10 percent of the way the text model is, and it needs many more rounds.
Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·
From the transcript
And my philosophy has always been, automate the easy work before the hard work. Will Jev models be used to automate software engineering? maybe one day, but software engineering, maybe I'm arrogant, seems a lot harder than really easy tasks that LMs fail at. And in order to be calibrated to the world, we should study why they fail at those easy tasks and really honestly go in-depth into studying them, not to cheat them, which I think is quite common, but actually dig in and figure out what is missing and why.
And sometimes it's not the model. Sometimes it's the prompt. Sometimes it's every other stuff. It's a collaboration between software and model. But it's through that that I think we learn the truth. I have stories about how we got to even be anti-benchmaxing more. But this is a thing that LLMs try to avoid because they will look bad on benchmarks, as far as I can tell.