What Should You Check First in a Document Benchmark?
Adit Abraham says the first thing to check is whether the distribution of data matches the work you want done. Simple documents often need only a simple pipeline, and a generic benchmark is the wrong yardstick. Strict scoring rules are a mistake, because the same information can be written faithfully two different ways and should not be penalized.
Benchmarks in the AI industry have become marketing, so everything looks state-of-the-art and it is hard to know what is good in your own context. A public benchmark they tested shows an egregious version. Take the same documents, tilt each page one degree, and add a little Gaussian blur. The provider that published the benchmark gets way worse, while everyone else stays consistent. The average person is not going to run that test. A language model used as a judge is quite strong today when the rubric is thoughtful. The better path is testing on your own data so you know what works best for you.
People often already have some sort of golden data set, and how they made it creates all sorts of issues. In situations they found, a human-generated set dropped information. Sometimes the set is language-model generated, and for cost reasons maybe they used Sonnet instead of Opus, and Sonnet made some sorts of mistakes.
The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham ·
From the transcript
Like, how do you think... I guess, what does a useful eval for a benchmark look like when your customers are evaluating Redacto versus maybe open source models or like other frontier models to similar tasks? What do they care mostly about? Yeah. And what do you think they should care mostly about? The number one thing that I think they should care about is distribution of data and understanding if it's representative for what they want to do.
Like if you're dealing with really simple documents, odds are a simple pipeline is sufficient. And you don't want to look at a generic benchmark for that. But number two, I think there's a lot of places where things go wrong of like how you even decide what to measure, where we made a post about this recently, like it's generally a mistake to have really strict rules on your benchmarking, for example, because you could represent the same information faithfully two different ways and you don't want to penalize that.