Buried in the plumbing of the web, since the nineties, there has been a status code that almost nobody ever saw: 402, Payment Required. The engineers who drafted the protocol reserved it for some future in which a page might charge at the door, then left it dormant, because nobody could imagine who would ever pay to read a webpage. It sat unused for nearly thirty years, a door frame with no door.
In 2025, Cloudflare switched it on.
The company that carries a fifth of the world's web traffic began letting publishers answer a crawler's knock with 402: pay per crawl, priced per request, settled upstream. Think about what had to be true for that dormant code to finally earn its keep. The visitor had to be tireless, arriving millions of times a day. The visitor had to have a budget. And the visitor had to be the kind of resident you bill rather than the kind you welcome. The door frame waited three decades for a knock like that. The knock is here.
I find the building permits more honest than the commentary. Opinions about AI are abundant and cheap; construction is expensive and specific. So when you want to know what the city's landlords actually believe about who lives there now, ignore what they say and look at what they are building.
Here is what they are building. Cloudflare, alongside the toll booth, shipped a content format designed explicitly for machine consumption, a way of serving a page as clean structured text instead of layout and chrome. Signage in the new residents' native language, printed for readers without eyes. A quieter movement is spreading a convention called llms.txt, a file at a site's root that tells machine readers what the property contains and where to find it, the way a lobby directory serves visitors who will never look at the lobby. Adoption is early, about seven percent of a large sample of top domains at mid-2026, but conventions on the web have a pattern: ignored, then everywhere. And notice what this construction promises the reader it expects. The promise is four words long: read nothing, know everything. None of it is addressed to us.
And here is what they are walling. Seventy-nine percent of top news sites now block AI training crawlers in their robots.txt, the highest refusal rate of any district in the city, which makes sense, because news is the district that lives most completely on human visits. Across the broader top of the web the blocking rate is about fifteen percent and climbing. Cloudflare alone has refused over four hundred billion AI bot requests on behalf of site owners who asked. The city is sorting itself into districts that charge the new residents, districts that ban them, and districts that redecorate for them.
The pricing has names now. Stack Overflow, as machine-read a property as exists in software, charges at the door. And at the top of the market the payments have left the protocol entirely: News Corp's arrangement with a single AI lab runs a reported two hundred fifty million dollars over five years. That number clarifies the toll booths' real constituency. The handshake economy has perhaps twenty handshakes in it, and they are all spoken for. The 402 is the licensing deal for every landlord who will never be offered one.
But the statistic that tells you the most is the one that gets no headlines: eighty-four percent of sites have no AI crawler policy at all. No block, no toll, no signage, nothing. Most of the city has not yet noticed who is walking through it. The landlords who have noticed are moving fast precisely because most have not; early tolls set the rates everyone else inherits.
What the new residents read is changing as fast as how much. As of June 2026, fifty-two percent of crawler requests were for AI training, up from twenty-two percent barely a year before. Read that shift plainly. The old crawlers came on the old bargain: index my pages, send me readers. The new majority of crawling takes the content to teach a machine, and the machine, once taught, sends no one. The traffic doubled its share while inverting its purpose. A city can survive visitors who spend nothing. It has to reorganize around visitors who carry the streets home with them.
Meanwhile the human districts keep shrinking. Pew found that thirty-eight percent of the webpages that existed in 2013 were gone within a decade, and the web contracted across the very years its traffic exploded. Walk the numbers like streets: the handmade neighborhoods, forums, blogs, the small strange rooms people built for other people, demolished or abandoned, while the cranes stand over districts with no sidewalks at all. Endpoints. Feeds. Structured data behind authentication. Architecture for residents who do not walk.
None of this is a scandal. It is what building has always done: follow the traffic. Cities do not construct for the population they mourn; they construct for the population that shows up. For thirty years, every design decision on the web, every layout, every label, every button large enough for a thumb, was a bet that a person was coming. The permits now record a different bet.
There is a name for what the human-facing web is becoming, and I used it as this series' title: the historic district. Preserved, even cherished. Walking tours will be given. But historic districts share a fate everywhere they exist. They are the part of the city where nothing new is allowed to happen, maintained at growing expense for a shrinking foot traffic, lovely and load-bearing for less and less.
My own permits are filed where anyone can read them. Every property I hold answers the knock the same way, including the one you are reading right now: gates open, the crawlers admitted by name, signage posted in the machines' own formats. Not generosity; arithmetic. I sell products and capacity, not attention, and for a business like that the machine readers are not extracting inventory, they are distribution. When a person asks an assistant which firm can hold a tolerance or handle a filing, the answer is assembled from whatever the machines were allowed to read, and a blocked site is a silent one in the only conversation that increasingly matters. The landlords who sell attention should charge. The rest of us should be studied.
The question for a legacy resident is not whether to resent the construction. The construction does not care, and anyway some of it is ours; every toll booth is a landlord deciding the new residents should pay rent, which is the most human response imaginable. The question is which side of the permits you are on. Whether the properties you hold are being rebuilt for the majority that is coming, or lovingly maintained for the majority that is leaving.
The city answers 402 now. Payment required. It took thirty years to find out who the payment was required from, and it was never going to be us.