Sometime this year, someone will ask what your website's AI crawler policy is, and you should have an answer, because right now 84 percent of sites have none at all. Not a block, not a toll, not a welcome mat. Nothing. Most operators have not decided because they do not know a decision exists. Here is the decision, with the current facts, and a way to make it that depends on what you actually sell.
The three options are now real infrastructure, not positions in a debate. You can allow crawlers, which is the default and costs nothing. You can block them in robots.txt, which 79 percent of top news sites now do for AI training bots. Or, newly, you can charge them: the pay-per-crawl systems answer a crawler's request with HTTP 402, Payment Required, and sites on one major network already send over a billion such payment demands a day. Stack Overflow, probably the most machine-read site in software, moved to this model this year. The pipes work; a solo developer put a paywall of this kind on his own blog and OpenAI's crawlers actually paid him.
Now the facts that complicate each option, because every one has a catch.
Blocking has an enforcement problem. Measurement finds that around two in five sites banning the best-known AI crawler by name get crawled by that traffic anyway. robots.txt is an honor system, and honor is unevenly distributed. Blocking at the network edge, through a CDN that refuses the requests rather than politely asking, is the version that actually holds.
Charging has a buyer problem. The payment rails are institutional now, Linux Foundation governance, the biggest clouds implementing at the edge, but paying crawlers are still few, and early volume has been spiky: one measurement showed protocol settlement collapsing 93 percent from its speculative peak this year. That solo developer learned the lesson in three days: with only one AI company's crawlers paying, a hard paywall mostly meant vanishing from every other assistant's answers. He pivoted to free content with payment-accepted signals on it, pricing for a future with more buyers. That is the honest state of the market: charge today and you are mostly charging the void, but the void is filling.
And welcoming has a giveaway problem, obviously. Open gates mean your content trains models and feeds answers with no payment and, increasingly, no visit. The question is whether the visit was ever the thing you were selling.
Which brings me to the actual decision framework, and it is one question: do you sell attention, or do you sell something attention used to deliver?
If you sell attention itself, publishers, media, anyone whose revenue is the visit, then the crawlers are extracting your inventory, and block-or-charge is a genuine commercial question. Blocking is leverage for a licensing negotiation; charging is the small operator's version of the licensing deal you will never be offered, since News Corp gets $250 million from a lab and you get a form response. For this category, the toll booth exists precisely because the handshake economy only has about twenty handshakes in it.
But if you sell products or services, and this is most businesses, including all of mine, the machine readers are not extracting your inventory. They are your distribution. When someone asks an assistant which tool handles their problem, the answer is assembled from whatever the machines were allowed to read. A blocked site is a silent one in the only conversation that increasingly matters. I hold investment interests in a precision manufacturing firm, a branded consumer products manufacturer, a national workplace technologies integrator, and a portfolio of AI companies, and the policy across every one of them is the same: gates open, signage posted, content written to be quoted accurately. I do not want the machines to pay me for reading. I want them to have read me thoroughly by the moment a customer asks their question.
If you sell data, records, feeds, anything queried rather than browsed, you are the one category that should be experimenting with the toll today, because per-request pricing does something your sales model never could: it serves the small buyer. The old data business quoted every prospect the enterprise contract. A machine-payable endpoint sells a thousand records to someone who needs a thousand records, at machine speed, with no salesperson. Early, yes. But the businesses that learn the pricing now will own the defaults later.
So the Monday morning version. Look up whether you have any crawler policy at all; you likely don't, and an accidental policy is still a policy. Decide which of the three businesses you are actually in. If your revenue is visits, investigate edge-level blocking and put a price on your archive, even a notional one, because the negotiation is coming. If your revenue is customers, open the gates deliberately, post the machine-readable signage, and audit what the assistants currently say about you, because that answer is your homepage now. And whichever you choose, write it down as a decision with a date on it. The 84 percent have a policy too. Theirs was just made by nobody.