Smash Notes

Back to Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It, with Diogo Almeida.

Why Is Key-value Cache Compaction Hard to Solve?

Diogo Almeida says compaction fits a write-once, append-only key-value cache, because a full cache gets restarted, but compaction is compression and compression is not fully solvable. Compressing without priors is very hard. Compress for the problem just solved, then ask a different problem that needed information already thrown away, and the agent is dealt a losing hand. He does not know the alternative. He calls it science fiction: notice a new task, let code compute the expected value of cache reads and cache writes, rebuild a cache for that task, and keep the old cache hot.

Working the same code base means rereading the same files and relearning the same abstractions. Markdown files written by the agent are only a partial fix, because keeping them in sync is a pain. History from older models, opened only when the model decides a tool is necessary, is a bit of a hack. A very cheap map-reduce over that history, as one of the first steps, plus sharing it with coworkers on the same code base, is a doable software problem. The snag is a key-value cache that stays unfriendly when the intelligence has to be append-only.

Transcript

Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·

From the transcript

I think that would be cool. Another example here is compaction. Compaction makes a lot of sense for a write-once KV cache that is append-only, because, like, oh, it's full. We need to restart it. But compaction is a form of compression, and compression is just not totally solvable. Like, it's very hard to compress without priors for compression.

But what happens if you compress based on what problem you were solving before, and now you want to actually ask a problem that is different from that one, and it would use information that was compressed away? That agent is like Delta, very much losing hand. But what if... how the state gets compressed was actually semantic. And it could detect, hey, I'm doing a new task.

See in transcript

Unsupervised Learning with Jacob Effron

More from this episode

  1. Was Diogo Almeida Mission Driven at OpenAI?Diogo Almeida would have called himself one of the lowest percentile mission-driven people at OpenAI.
  2. Why Can AI Solve Hard Math but Not Accounting?Diogo Almeida says the models people need are System 1 models, and the honest move is to say where they fail.
  3. Why Is a New AI Model Shape Cheaper Than Reliability?Diogo Almeida says a new model shape is cheap and reliability is the part that matters.
  4. How Far Along Is Jev's Image Model?Diogo Almeida says Jev's image model is maybe 10 percent of the way the text model is.
  5. Why Isn't AI Automating the Easy Work Yet?Diogo Almeida says models have been smart enough for easy automation for years, and that intelligence still is not being put to work.
  6. Does Jevons Paradox Already Hold for AI?Diogo Almeida says people at Jev truly believe in Jevons Paradox, which is why the company carries the name, and the paradox does not hold yet.
  7. Why Does Diogo Almeida Call Financial Engineering the Labs' Strength?Diogo Almeida's sense is that the labs' greatest strength is financial engineering, not models, product, or distribution.
  8. How Much Do Coding Agents Speed Up Engineers?Diogo Almeida estimates coding agents speed up engineering by 20% to 40% on average, with error bars for long-term effects like skill degradation and tech debt.
  9. Why Is True AI Agency Two Breakthroughs Away?Diogo Almeida guesses that true AI agency is something like two gigantic breakthroughs away.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.