Why Is Key-value Cache Compaction Hard to Solve?
Diogo Almeida says compaction fits a write-once, append-only key-value cache, because a full cache gets restarted, but compaction is compression and compression is not fully solvable. Compressing without priors is very hard. Compress for the problem just solved, then ask a different problem that needed information already thrown away, and the agent is dealt a losing hand. He does not know the alternative. He calls it science fiction: notice a new task, let code compute the expected value of cache reads and cache writes, rebuild a cache for that task, and keep the old cache hot.
Working the same code base means rereading the same files and relearning the same abstractions. Markdown files written by the agent are only a partial fix, because keeping them in sync is a pain. History from older models, opened only when the model decides a tool is necessary, is a bit of a hack. A very cheap map-reduce over that history, as one of the first steps, plus sharing it with coworkers on the same code base, is a doable software problem. The snag is a key-value cache that stays unfriendly when the intelligence has to be append-only.
Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It ·
From the transcript
I think that would be cool. Another example here is compaction. Compaction makes a lot of sense for a write-once KV cache that is append-only, because, like, oh, it's full. We need to restart it. But compaction is a form of compression, and compression is just not totally solvable. Like, it's very hard to compress without priors for compression.
But what happens if you compress based on what problem you were solving before, and now you want to actually ask a problem that is different from that one, and it would use information that was compressed away? That agent is like Delta, very much losing hand. But what if... how the state gets compressed was actually semantic. And it could detect, hey, I'm doing a new task.