Smash Notes

Back to Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It, with Diogo Almeida.

Why Is a New AI Model Shape Cheaper Than Reliability?

Diogo Almeida says a new model shape is cheap and reliability is the part that matters. He still wants novel shapes and plans to keep shipping them, but people in machine learning are over-focused on the shape of the task. When he thought the work would take a week, he meant a week to make the shape, and that part did take a week. Other teams are now making similar shapes. The shape does not matter next to the model being useful.

He expects many more shapes than the two already out, even if Jev ships them alone. Too many shapes without utility burn goodwill, because the releases over-promise and under-deliver. He does not want creative energy spent on things that clearly do not work. He is against benchmarks for the same reason. Developers who cannot test every release get swept up, and a benchmark does not show how the model actually behaves.

Transcript

Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·

From the transcript

I would say when we first launched this, I was really excited for novel shapes to emerge. Really, really excited. And I still think that that will be the case. I think new shapes are freaking awesome. We plan to keep shipping them. I'm a little bit worried that, again, what we talked about, maybe this was before we got on the record, that ML people especially have kind of lost the actual message I was trying to say.

And I'm worried that people are now over-focused on the shapes of tasks rather than the reliability. The shape of task is incredibly cheap. It's super easy to do that. When I thought this would take a week, I thought it would take a week to make the shape of the task, which it did. And this is what's happening now with other people making similar models with similar shapes.

See in transcript

Unsupervised Learning with Jacob Effron

More from this episode

  1. Was Diogo Almeida Mission Driven at OpenAI?Diogo Almeida would have called himself one of the lowest percentile mission-driven people at OpenAI.
  2. Why Can AI Solve Hard Math but Not Accounting?Diogo Almeida says the models people need are System 1 models, and the honest move is to say where they fail.
  3. Why Is Key-value Cache Compaction Hard to Solve?Diogo Almeida says compaction fits a write-once, append-only key-value cache, because a full cache gets restarted, but compaction is compression and…
  4. How Far Along Is Jev's Image Model?Diogo Almeida says Jev's image model is maybe 10 percent of the way the text model is.
  5. Why Isn't AI Automating the Easy Work Yet?Diogo Almeida says models have been smart enough for easy automation for years, and that intelligence still is not being put to work.
  6. Does Jevons Paradox Already Hold for AI?Diogo Almeida says people at Jev truly believe in Jevons Paradox, which is why the company carries the name, and the paradox does not hold yet.
  7. Why Does Diogo Almeida Call Financial Engineering the Labs' Strength?Diogo Almeida's sense is that the labs' greatest strength is financial engineering, not models, product, or distribution.
  8. How Much Do Coding Agents Speed Up Engineers?Diogo Almeida estimates coding agents speed up engineering by 20% to 40% on average, with error bars for long-term effects like skill degradation and tech debt.
  9. Why Is True AI Agency Two Breakthroughs Away?Diogo Almeida guesses that true AI agency is something like two gigantic breakthroughs away.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.