Smash Notes

Back to Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It, with Diogo Almeida.

How Far Along Is Jev's Image Model?

Diogo Almeida says Jev's image model is maybe 10 percent of the way the text model is. He would be truly embarrassed to call it production ready, because so much jaggedness is left, and it still lacks his seal of approval.

Text tasks will be automated before image tasks. Investment went really heavily into text tasks, with a tiny amount of work on image tasks by comparison. He is jazzed the image model does as well as it does. The lead in images is not as large as the lead in text, but he senses an overall lead in images and could be totally wrong. He calls the image model completely insufficient. It is not quite as bad as other people just shipping a shape of something. In the grand scheme the image model is maybe 10 percent of the way the text model is, and it needs many more rounds.

Grokipedia·Diogo Almeida

Transcript

Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·

From the transcript

And my philosophy has always been, automate the easy work before the hard work. Will Jev models be used to automate software engineering? maybe one day, but software engineering, maybe I'm arrogant, seems a lot harder than really easy tasks that LMs fail at. And in order to be calibrated to the world, we should study why they fail at those easy tasks and really honestly go in-depth into studying them, not to cheat them, which I think is quite common, but actually dig in and figure out what is missing and why.

And sometimes it's not the model. Sometimes it's the prompt. Sometimes it's every other stuff. It's a collaboration between software and model. But it's through that that I think we learn the truth. I have stories about how we got to even be anti-benchmaxing more. But this is a thing that LLMs try to avoid because they will look bad on benchmarks, as far as I can tell.

See in transcript

Unsupervised Learning with Jacob Effron

More from this episode

  1. Was Diogo Almeida Mission Driven at OpenAI?Diogo Almeida would have called himself one of the lowest percentile mission-driven people at OpenAI.
  2. Why Can AI Solve Hard Math but Not Accounting?Diogo Almeida says the models people need are System 1 models, and the honest move is to say where they fail.
  3. Why Is a New AI Model Shape Cheaper Than Reliability?Diogo Almeida says a new model shape is cheap and reliability is the part that matters.
  4. Why Is Key-value Cache Compaction Hard to Solve?Diogo Almeida says compaction fits a write-once, append-only key-value cache, because a full cache gets restarted, but compaction is compression and…
  5. Why Isn't AI Automating the Easy Work Yet?Diogo Almeida says models have been smart enough for easy automation for years, and that intelligence still is not being put to work.
  6. Does Jevons Paradox Already Hold for AI?Diogo Almeida says people at Jev truly believe in Jevons Paradox, which is why the company carries the name, and the paradox does not hold yet.
  7. Why Does Diogo Almeida Call Financial Engineering the Labs' Strength?Diogo Almeida's sense is that the labs' greatest strength is financial engineering, not models, product, or distribution.
  8. How Much Do Coding Agents Speed Up Engineers?Diogo Almeida estimates coding agents speed up engineering by 20% to 40% on average, with error bars for long-term effects like skill degradation and tech debt.
  9. Why Is True AI Agency Two Breakthroughs Away?Diogo Almeida guesses that true AI agency is something like two gigantic breakthroughs away.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.