How Does Parallel Build a Web Index Over Time?

Parag Agrawal says Parallel launched a search agent first, not a search product, and grew a larger, more sophisticated web index over time while solving problems for more customers, without spending all of the money on infrastructure before serving a customer. The insight convinced him the problem was viable to take on.

Deep research leaves about a minute of patience, and some products sometimes take 10 minutes of research. With a map for which pages to fetch first, the wait is long enough to crawl a lot of pages after a query arrives. An index often works as a latency optimization. Giving up instant speed meant the search agents competed with outsourcing work to humans who curate data. To him, humans sitting on search engines seemed far easier to compete with than a search engine on day zero. A search agent that did real work on web data let Parallel build the index piece by piece.

In the early days, people built search agents on Parallel for insurance underwriting and claims processing. Sales teams used them for sales data enrichment. Finance teams that used to send data collection overnight to humans before a modeling exercise started using Parallel instead. Parallel went after places where outsourced human work already sat on top of web data, collected evals, ran agents, and learned what search for agents should look like from empirical use cases instead of theoretical evals.