Do Frontier Models Silently Drop Facts From Long Documents?
Adit Abraham says frontier models were more precise than many specialized document parsers on a long-extraction benchmark: when they pulled a fact, it was more likely to be correct. The failure was recall. They silently dropped information, and that miss is hard to catch once the pipeline is in production.
A single model call is the wrong shape for a long document. The file has to be broken into parts. The Deep Extract harness is not tied to one model. It gives the model tools to review its own output and to apply checks, for example that the lines on an invoice add up to the total, so it knows when to look again. He says that review loop is what got the extraction to near-perfect accuracy on that task.
The Missing Infrastructure for AI Agents: Documents | Reducto’s Adit Abraham ·
From the transcript
capable large models in the last mile? Maybe you can talk through like what are like typical components of a pipeline and how does that maybe on the price and latency compare to these like one shot type of approach? There's actually a really interesting benchmark that Micro One put out. It was independently run and we got to partner with them on it where it was focused on long extraction tests.
And the reason why I think it's interesting is we tested all the Frontier models at the time. We tested Reducto and some specialized document parsers. And we found actually that Frontier models were better than a lot of specialized document parsers at precision. Like given that they were extracting something, they were more likely to be correct.