Smash Notes

Back to Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It, with Diogo Almeida.

How Much Do Coding Agents Speed Up Engineers?

Diogo Almeida estimates coding agents speed up engineering by 20% to 40% on average, with error bars for long-term effects like skill degradation and tech debt. He treats the average gain as a decent but small boost.

The number depends on the use case and on the skill of the user. Side effects are the bad part, so the gain also depends on whether side effects can be isolated in a way that stays legible. In some use cases the speed-up is kind of like an infinite percent, because the work would never get done otherwise. In other use cases the agents are as bad as coding the thing by hand.

A decent but small average boost is the consensus he has seen in the field right now.

Grokipedia·Diogo Almeida

Transcript

Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·

From the transcript

OK, I will say I have error bars on this, because I think that there's long-term effects like skill degradation and tech debt and stuff like this. So my estimates are in the 20% to 40% speed-up range. But again, it really depends on the use case. And it also depends on the skill of the user and if you can, because the side effects are the bad thing, if you can isolate the side effects in a way that those are very legible.

So depending on the use case, for some, it's kind of like an infinite percent speed up. You would do things you would never, you can do things you can never do otherwise. For others, they are like just so bad that they're as bad as like coding the thing by hand. I would guess on average, it's like a decent but small boost. And I believe that is the consensus that I've seen in the field right now.

See in transcript

Unsupervised Learning with Jacob Effron

More from this episode

  1. Was Diogo Almeida Mission Driven at OpenAI?Diogo Almeida would have called himself one of the lowest percentile mission-driven people at OpenAI.
  2. Why Can AI Solve Hard Math but Not Accounting?Diogo Almeida says the models people need are System 1 models, and the honest move is to say where they fail.
  3. Why Is a New AI Model Shape Cheaper Than Reliability?Diogo Almeida says a new model shape is cheap and reliability is the part that matters.
  4. Why Is Key-value Cache Compaction Hard to Solve?Diogo Almeida says compaction fits a write-once, append-only key-value cache, because a full cache gets restarted, but compaction is compression and…
  5. How Far Along Is Jev's Image Model?Diogo Almeida says Jev's image model is maybe 10 percent of the way the text model is.
  6. Why Isn't AI Automating the Easy Work Yet?Diogo Almeida says models have been smart enough for easy automation for years, and that intelligence still is not being put to work.
  7. Does Jevons Paradox Already Hold for AI?Diogo Almeida says people at Jev truly believe in Jevons Paradox, which is why the company carries the name, and the paradox does not hold yet.
  8. Why Does Diogo Almeida Call Financial Engineering the Labs' Strength?Diogo Almeida's sense is that the labs' greatest strength is financial engineering, not models, product, or distribution.
  9. Why Is True AI Agency Two Breakthroughs Away?Diogo Almeida guesses that true AI agency is something like two gigantic breakthroughs away.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.