Smash Notes

Back to The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham, with Adit Abraham & Jennifer Li.

What Should You Check First in a Document Benchmark?

Adit Abraham says the first thing to check is whether the distribution of data matches the work you want done. Simple documents often need only a simple pipeline, and a generic benchmark is the wrong yardstick. Strict scoring rules are a mistake, because the same information can be written faithfully two different ways and should not be penalized.

Benchmarks in the AI industry have become marketing, so everything looks state-of-the-art and it is hard to know what is good in your own context. A public benchmark they tested shows an egregious version. Take the same documents, tilt each page one degree, and add a little Gaussian blur. The provider that published the benchmark gets way worse, while everyone else stays consistent. The average person is not going to run that test. A language model used as a judge is quite strong today when the rubric is thoughtful. The better path is testing on your own data so you know what works best for you.

People often already have some sort of golden data set, and how they made it creates all sorts of issues. In situations they found, a human-generated set dropped information. Sometimes the set is language-model generated, and for cost reasons maybe they used Sonnet instead of Opus, and Sonnet made some sorts of mistakes.

Grokipedia·Sonnet

Transcript

The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham ·

From the transcript

Like, how do you think... I guess, what does a useful eval for a benchmark look like when your customers are evaluating Redacto versus maybe open source models or like other frontier models to similar tasks? What do they care mostly about? Yeah. And what do you think they should care mostly about? The number one thing that I think they should care about is distribution of data and understanding if it's representative for what they want to do.

Like if you're dealing with really simple documents, odds are a simple pipeline is sufficient. And you don't want to look at a generic benchmark for that. But number two, I think there's a lot of places where things go wrong of like how you even decide what to measure, where we made a post about this recently, like it's generally a mistake to have really strict rules on your benchmarking, for example, because you could represent the same information faithfully two different ways and you don't want to penalize that.

See in transcript

a16z

More from this episode

  1. How Does Reducto Read a Messy Document?Adit Abraham breaks document reading into a few epochs, ending with Reducto R1, one unified model that reads a page from end to end.
  2. Why Do Document Mistakes Compound Across a Dozen Files?Adit Abraham says a loan packet is more than reading the page.
  3. Why Did Coding Tools Reach a Billion in Revenue First?Adit Abraham says coding tools reached a billion dollars in annual revenue while professional-services AI has not, even though healthcare, insurance, legal…
  4. Why Won't the Big Model Labs Solve Document Parsing?Jennifer Li asks why the big model labs, already focused on knowledge work and on vision, will not simply solve document parsing.
  5. Do Frontier Models Silently Drop Facts From Long Documents?Adit Abraham says frontier models were more precise than many specialized document parsers on a long-extraction benchmark: when they pulled a fact, it was…
  6. Can Agents Do Healthcare Work Without Reliable Document Parsing?Jennifer Li asks how accuracy changes whether customers will trust an agent with their documents.

People

More answers like this

The other side

Related episodes

Plays from this note through the rest of the episode.

Weekly highlights

One letter a week.

The notes worth hearing, with the minute they were said.