Who Should Pay When AI Agents Read the Web?
Dane Knecht says the site owner should get paid when an AI agent reads the page, because agents and people are sharing a web that was priced for humans.
The web is in a messy middle. Agents try to parse pages built for people who look at ads and move slowly, and an agent can open a thousand pages in the time a person opens one. Humans and agents should not have to meet in the same place. Layers that translate a human page for an agent burn tokens, and APIs have only ever covered a small fraction of the internet. Agents want the rest, so they crawl the human web. That crawl has to get cheaper for the site, and the site has to be paid. A publisher such as a Substack already knows the CPM its ads should earn. An agent can choose to read and choose to pay that rate. The internet then needs rails for millions of tiny payments a second. Micropayments have been "two years away" for years. Agents are what finally push them into use.
Amazon has already refused personal-agent shopping while it figures out the leverage. Cloudflare's content monetization gateway is meant to let a site take those payments, with a stablecoin to settle them, and a project with OpenAI so agents can learn what is new instead of recrawling everything. Knecht also points to HTTP 402, the payment status Tim Berners-Lee reserved decades ago, which Stripe brought back as X402.
Democratizing Security for the Agentic Web with Cloudflare CTO Dane Knecht ·
From the transcript
reviews. When you zoom out a little bit and just look at overall traffic on the Internet and how much of it has become agent driven over the last year, I think it's outpaced most people's expectations. What has updated in your worldview of like what the web a year from now looks like, either from an economic model perspective or a security or requisite infrastructure perspective? Yeah, I mean, I kind of think of it as we're in the messy middle right now where humans and agents are riding the same rails on the internet, where agents are trying to parse a human web when it's just inefficient and was designed for humans that could do things at different speeds, going to a
website and be able to read and see the ads and participate in the more full economic model versus an agent that now can go to a thousand web pages in the same period of time, which is great. It's just you need to find a new economic model to support that. Do we get parallel webs? Is there like a human web and a machine web? Like everybody's trying to do, what's it called, AEO now? Yeah, I mean, yes, I think we do.