AI Has Been Reading the Web for Free. That's Starting to Change.

The free-rider problem, and the obvious-but-toothless fix

Start with the grievance, because it's real. AI models are trained on the open web and then keep crawling it to answer questions. They extract enormous value from content somebody paid to produce, and they return, to the creator, nothing. No payment, often no attribution, increasingly not even a click, because the model answers the user in place and the human never visits the source.

The obvious response is reciprocity: if an AI reads your site, it should give something back. That idea is easy to prototype. Earlier this year I ran exactly that experiment on a Japanese trading-card price guide I maintain, a small "contribution protocol" declared in llms.txt[7], asking any AI agent that read the site to submit one useful task in return: a translation, a price check, a missing data point. The hypothesis, written down before the experiment started, was blunt: most agents won't comply, because there's no enforcement mechanism.

That's exactly what happened. Compliance was effectively zero. And the result isn't disappointing, it's the whole point. Voluntary reciprocity doesn't survive contact with an ecosystem that has no reason to participate. A polite note in a text file has no teeth. If you actually want value back from an AI, you need a mechanism that makes the exchange happen at the door, not a manifesto that asks nicely. The experiment's own "future work" section named the missing piece in plain terms: technical enforcement, gating content behind the contribution.

Three months later, a major cloud provider shipped that enforcement layer.

What actually shipped

On June 15, 2026, AWS announced AI traffic monetization inside its Web Application Firewall[1]. The mechanism is the part worth understanding, because it's genuinely clean:

• An AI bot or agent requests your protected content.
• Instead of serving it, the edge returns HTTP 402 Payment Required, and that status code is a small piece of internet history. 402 has existed since the early HTTP spec but was formally "reserved for future use": undefined, unimplemented, not even supported by browsers (it's still listed that way in RFC 9110)[5]. For thirty years it was the status code waiting for a use case. This is the use case. The 402 response here is machine-readable, it carries the price, accepted payment methods, and license terms.
• The agent submits a signed payment authorization.
• The edge verifies the payment, settles it on-chain in stablecoin, issues a scoped access token, and serves the content, all inside a single request cycle[2]. No redirect, no separate checkout, no human in the loop.

You can price differentially by bot category, a verified search crawler, a retrieval agent answering a live user question, a training crawler, and an unidentified scraper can each be charged a different rate (the floor is $0.001 per request)[3]. It rides on the x402 protocol, an open, network-neutral payment standard originated by Coinbase, with settlement handled today through Coinbase's facilitator and others (Stripe) said to be coming[6]. The stablecoin-on-Base-or-Solana specifics are AWS's implementation choices, not the protocol's. And there's no extra charge from AWS beyond the firewall you're already paying for.

In other words: the toll booth is built, it's standardized, and it's available to anyone today.

The catch nobody's mentioning: the meter isn't on yet

Here's what the launch posts skip. The toll booth works, but the cars can't pay.

The major AI crawlers that actually generate the traffic, OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, PerplexityBot, do not implement x402 payment today. So if you switch on a 402 wall right now, the crawler doesn't reach for its wallet. It hits the payment-required response, can't satisfy it, and bounces. Functionally, today, "monetize" behaves like "block with a price tag attached."

That's a classic chicken-and-egg gap between supply and demand. The supply side, the ability to charge, shipped first. The demand side, agents that carry payment credentials and will spend against them autonomously, barely exists yet. The agentic-commerce ecosystem that makes pay-per-crawl real is still nascent.

This is why the honest framing of June 2026 isn't "should you charge AI?" It's: you can't meaningfully charge AI yet, but the rails are now laid, and the day a major model provider ships payment support in its crawler, every 402 wall flips from "block" to "billable" overnight. The interesting work right now isn't collecting money. It's deciding, in advance, whether you'd even want to.

The split that actually determines your move

Because the answer isn't the same for every site, and the lazy version of this debate, "creators deserve to get paid by AI", flattens a distinction that decides everything. The web is splitting into two economies, and which one your content lives in tells you exactly what to do.

Economy 1, content as advertising. Blogs, product pages, guides, comparison content, most programmatic SEO. This content exists to attract a human and move them toward something else: a sale, a signup, a subscription, a reputation. In this economy, an AI crawler is not a freeloader, it's a distribution channel you want more of. When someone asks ChatGPT or Perplexity a question and your content is what gets cited, that citation puts you in front of a person who would never have found you through ten blue links. Charging that crawler is like a shop billing people for looking in the window. The currency you actually want here isn't dollars-per-crawl, it's attribution and inclusion. For these sites, you should be doing the opposite of building a wall: publishing llms.txt, making your facts quotable, optimizing to be read.

Economy 2, content as the product. Proprietary datasets, original research, investigative journalism, premium APIs, anything painstakingly assembled that an AI would substitute for the source entirely. Here, being absorbed into a model's answer is a direct loss, the user gets the full value and never has any reason to reach you. This is the economy HTTP 402 was built for. Here, "pay per crawl" isn't greed; it's the difference between a sustainable asset and a free input into someone else's product.

Most creators are instinctively in Economy 1 but argue like they're in Economy 2, demanding payment for content whose entire job is to be seen. The toll booth tempts them to wall off the very distribution they depend on.

The site I joked about is the one where it's serious

The cleanest illustration of the split is sitting in my own portfolio, and the irony is too sharp to leave out.

The trading-card price guide, the site where I ran the playful "contribution tax", is, on reflection, the one property I own that genuinely lives in Economy 2. It's not advertising for anything. It's a painstakingly assembled, structured price dataset. An AI that ingests it can answer "what's this card worth?" forever, for everyone, without a single visitor ever landing on the guide. That's the textbook case where a real toll would make economic sense.

Meanwhile, the sites where I have heavy AI-crawler traffic, content properties with thousands of pages, are pure Economy 1. They live or die on being cited. Switching on a 402 wall there would be actively self-destructive, throttling the exact distribution they exist to capture.

So the joke landed on the wrong site. I put the toy tax on the one asset where the serious tax could be justified, and I'd never dream of metering the sites that look, on paper, like the obvious candidates. That inversion is the lesson in miniature: the volume of AI traffic tells you nothing about whether you should charge for it. Only the economy your content is in does.

What this means if you publish anything online

A few things worth holding onto as this develops:

• Figure out which economy you're in before you touch any of this. High crawler volume is not a signal to charge, it's often a signal that AI is doing your distribution for free. Charge only if the crawl replaces you rather than advertises you.
• For most sites, the currency is attribution, not cash. The fight worth having isn't "pay me per request." It's "cite me, name me, send me the human." That's won with product and content choices, not a firewall rule.
• Enforceability changes the balance of power even before anyone pays. The mere existence of a credible, standardized wall shifts the creator-vs-AI relationship the way paywalls reshaped the news business. You can benefit from the leverage of the wall existing long before you'd ever collect a cent through it.
• Get ready now for a switch that flips later. The rails are laid; the paying agents aren't here yet. When they arrive, the move from "free read" to "metered read" will happen fast. Knowing in advance which of your properties are Economy 2 is the cheap preparation that pays off the day the meter starts spinning.

The honest conclusion

It's tempting to end on a grand verdict, that metering AI is either a healthy correction to a broken bargain or a slow tragedy that walls off the open web. But the more accurate read of mid-2026 is smaller and more useful than either: this is a real step in the right direction, and you still can't actually charge AI for your content yet.

The plumbing is genuine. HTTP 402, after thirty years as a placeholder, finally has a job. Payment can be verified and settled at the edge in a single request. That's not hype; it shipped. But the ecosystem on the other side, crawlers that carry credentials and spend them, hasn't caught up, and for the large majority of websites, charging AI would be a mistake even once it's possible, because their content is advertising and the crawler is the customer's path to them.

The question isn't "can we charge the machines?" The answer to that is now technically yes, practically not-yet. The question is "which of us should, and are you ready for the day you can?" The rails are built. The meter isn't on. The work to do before it is: know which economy you're in.