Smash Notes

Back to The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham, with Adit Abraham & Jennifer Li.

Why Do Document Mistakes Compound Across a Dozen Files?

Adit Abraham says a loan packet is more than reading the page. The job splits the files, extracts individual fields, and may edit the application at the end, and each of those steps has to be accurate. Across a dozen files, 90 percent on the individual pages is not acceptable, because those mistakes compound.

Jennifer Li says a lot of traditional OCR still lands around 70 to 80 percent, short of the high 90s. For parsing itself, Adit says the vast majority of use cases can already reach the high 90s, aside from corner cases such as handwriting he cannot read. The newer work is a harness around the model. Deep Extract is where he finds an agent can beat humans at structured extraction on many tests, because a human-made answer key drops rows or fills in only part of the page. The harness extracts, then reviews and corrects its own output.

Transcript

The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham ·

From the transcript

What are you shooting for? Like, how do you think about that trade-off of still having human in the loop versus you can actually use agent to replace a whole sort of document processing pipeline? Yeah, and this actually goes to the work that we're doing beyond parsing. I think parsing is like this core building block of how do you even understand what was on the page? And for the vast majority of use cases, we can actually be in the high 90s today.

There are corner cases, like there's doctor's handwriting that I can't read, and so I don't even know if we did a good job. But if you think of an end-to-end task, like you're taking in a loan packet, it's not just parsing, you're splitting the contents, you're extracting individual fields, maybe you're editing a loan application form at the end.

See in transcript

a16z

More from this episode

  1. How Does Reducto Read a Messy Document?Adit Abraham breaks document reading into a few epochs, ending with Reducto R1, one unified model that reads a page from end to end.
  2. Why Did Coding Tools Reach a Billion in Revenue First?Adit Abraham says coding tools reached a billion dollars in annual revenue while professional-services AI has not, even though healthcare, insurance, legal…
  3. Why Won't the Big Model Labs Solve Document Parsing?Jennifer Li asks why the big model labs, already focused on knowledge work and on vision, will not simply solve document parsing.
  4. Do Frontier Models Silently Drop Facts From Long Documents?Adit Abraham says frontier models were more precise than many specialized document parsers on a long-extraction benchmark: when they pulled a fact, it was…
  5. What Should You Check First in a Document Benchmark?Adit Abraham says the first thing to check is whether the distribution of data matches the work you want done.
  6. Can Agents Do Healthcare Work Without Reliable Document Parsing?Jennifer Li asks how accuracy changes whether customers will trust an agent with their documents.

People

More answers like this

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.