Smash Notes

Back to The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham, with Adit Abraham & Jennifer Li.

Do Frontier Models Silently Drop Facts From Long Documents?

Adit Abraham says frontier models were more precise than many specialized document parsers on a long-extraction benchmark: when they pulled a fact, it was more likely to be correct. The failure was recall. They silently dropped information, and that miss is hard to catch once the pipeline is in production.

A single model call is the wrong shape for a long document. The file has to be broken into parts. The Deep Extract harness is not tied to one model. It gives the model tools to review its own output and to apply checks, for example that the lines on an invoice add up to the total, so it knows when to look again. He says that review loop is what got the extraction to near-perfect accuracy on that task.

Transcript

The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham ·

From the transcript

capable large models in the last mile? Maybe you can talk through like what are like typical components of a pipeline and how does that maybe on the price and latency compare to these like one shot type of approach? There's actually a really interesting benchmark that Micro One put out. It was independently run and we got to partner with them on it where it was focused on long extraction tests.

And the reason why I think it's interesting is we tested all the Frontier models at the time. We tested Reducto and some specialized document parsers. And we found actually that Frontier models were better than a lot of specialized document parsers at precision. Like given that they were extracting something, they were more likely to be correct.

See in transcript

a16z

More from this episode

  1. How Does Reducto Read a Messy Document?Adit Abraham breaks document reading into a few epochs, ending with Reducto R1, one unified model that reads a page from end to end.
  2. Why Do Document Mistakes Compound Across a Dozen Files?Adit Abraham says a loan packet is more than reading the page.
  3. Why Did Coding Tools Reach a Billion in Revenue First?Adit Abraham says coding tools reached a billion dollars in annual revenue while professional-services AI has not, even though healthcare, insurance, legal…
  4. Why Won't the Big Model Labs Solve Document Parsing?Jennifer Li asks why the big model labs, already focused on knowledge work and on vision, will not simply solve document parsing.
  5. What Should You Check First in a Document Benchmark?Adit Abraham says the first thing to check is whether the distribution of data matches the work you want done.
  6. Can Agents Do Healthcare Work Without Reliable Document Parsing?Jennifer Li asks how accuracy changes whether customers will trust an agent with their documents.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.