Smash Notes

Back to Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It, with Diogo Almeida.

Why Isn't AI Automating the Easy Work Yet?

Diogo Almeida says models have been smart enough for easy automation for years, and that intelligence still is not being put to work. A document he wrote at OpenAI was titled "we could have had AGI a year ago," after OpenAI's definition of doing most of the world's economically valuable work. He reads the spirit of that line as work by volume. Models from the GPT-3.5 era, and possibly the GPT-3 era, could have met it. They were already past a median level on all sorts of tasks.

That should have meant tons of automation, at least on the easy work. GPQA, the hard multiple-choice test he describes as PhD-level, was passing that bar in 2023. He calls it tragic that intelligence at that level was not put to work, and says the raw material has been around for a long time.

Transcript

Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·

From the transcript

it would have been harder, but doable. Back when I was more naive, oh man, I sound like an old dude. When I was more naive and I thought that this would take a week of work, What convinced me, I love writing documents. I think it's a great way to think. I wrote a document at OpenAI that was titled, we could have had AGI a year ago, at least by OpenAI's definition of AGI, which I don't think that people really listen to the letter of the definition, but I think it's actually a really great letter.

I think that models starting from GPT-3.5 era, possibly even GPT-3 era, could have met the definition of AGI at the time, which is still the same, like doing most of the world's economically valuable work by volume. It doesn't say by volume, but I assume the spirit of that is by volume. Because the models are so unbelievably smart. Like GPT-3.5 level models surpassed median level intelligence at all sorts of things.

See in transcript

Unsupervised Learning with Jacob Effron

More from this episode

  1. Was Diogo Almeida Mission Driven at OpenAI?Diogo Almeida would have called himself one of the lowest percentile mission-driven people at OpenAI.
  2. Why Can AI Solve Hard Math but Not Accounting?Diogo Almeida says the models people need are System 1 models, and the honest move is to say where they fail.
  3. Why Is a New AI Model Shape Cheaper Than Reliability?Diogo Almeida says a new model shape is cheap and reliability is the part that matters.
  4. Why Is Key-value Cache Compaction Hard to Solve?Diogo Almeida says compaction fits a write-once, append-only key-value cache, because a full cache gets restarted, but compaction is compression and…
  5. How Far Along Is Jev's Image Model?Diogo Almeida says Jev's image model is maybe 10 percent of the way the text model is.
  6. Does Jevons Paradox Already Hold for AI?Diogo Almeida says people at Jev truly believe in Jevons Paradox, which is why the company carries the name, and the paradox does not hold yet.
  7. Why Does Diogo Almeida Call Financial Engineering the Labs' Strength?Diogo Almeida's sense is that the labs' greatest strength is financial engineering, not models, product, or distribution.
  8. How Much Do Coding Agents Speed Up Engineers?Diogo Almeida estimates coding agents speed up engineering by 20% to 40% on average, with error bars for long-term effects like skill degradation and tech debt.
  9. Why Is True AI Agency Two Breakthroughs Away?Diogo Almeida guesses that true AI agency is something like two gigantic breakthroughs away.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.