On August 6, The Economist ran “The tragedy of the commons, AI edition”, about Britain’s employment tribunals. By the article’s account, AI has pushed the cost of drafting a claim to near zero: anyone can generate a professional-looking legal filing in ten minutes. So claims surged, the backlog keeps stretching, and a case filed today may not be heard until 2030. Free legal help should be good news; the result is that everyone waits in a longer queue. The piece is paywalled and I couldn’t read the full text; this summary comes from publicly quoted passages and the Hacker News thread.

A tribunal is a textbook commons: limited processing capacity, first come first served, and every extra filing spreads its cost across everyone in line. The Economist applies the frame to the legal system, but its real home is the open web itself. Over the past year and a half, “AI is consuming the digital commons” has stopped needing a metaphor. The operations reports are full of numbers.

Four invoices

The Wikimedia Foundation published a set of figures in April 2025: since January 2024, bandwidth for multimedia content had grown 50%, with the increase coming almost entirely from crawlers gathering training data. The structure is more telling than the total. Bots account for only about 35% of pageviews, yet generate at least 65% of the most expensive traffic hitting the core datacenters. The reason is caching. Human readers cluster on popular articles, so frontend caches absorb most of their requests; crawlers sweep the whole corpus, including cold pages nobody reads, and every one of those requests punches through to the origin. In the Foundation’s own words: “Our content is free, our infrastructure is not.”

Drew DeVault, founder of the code hosting platform SourceHut, presented the second invoice in a March 2025 post titled “Please stop externalizing your costs directly into my face”: in those months, 20 to 100 percent of his working week went into fighting LLM crawlers, and the site suffered dozens of brief outages per week. The crawlers go straight for the computationally expensive endpoints: git blame, every page of every repository’s commit log. Blocking barely works. Requests arrive from tens of thousands of residential IPs that look like ordinary home broadband users, User-Agents are spoofed to match everyday browsers, and each IP sends one request in any window you try to measure. In the same period, KDE’s GitLab was crawled hard enough that developers were locked out for a time (LibreNews roundup).

Daniel Stenberg, maintainer of curl, delivered the third in July 2025: of the security reports curl received on HackerOne that year, roughly 20% were AI-generated slop, fabricated vulnerabilities and hallucinated function calls, all fluently written. The share of reports describing real vulnerabilities fell to 5%. Each report ties up three to four maintainers for half an hour to three hours apiece. Senior maintainers’ review attention was scarce before this; now it is being poured, in batches, into fake reports.

The fourth is the plainest. By Gergely Orosz’s analysis of public Stack Exchange data, Stack Overflow’s monthly volume of new questions in May 2025 fell back to 2009 levels, the site’s first full year of existence.

Which layer the tragedy happens on

The textbook tragedy of the commons has a precondition: the resource is rival. Your bite leaves less for me. Digital content fails exactly that test: copy a Wikipedia article ten thousand times and the original doesn’t lose a word. On that reading alone, “tragedy of the commons, AI edition” looks like a category error.

Put the four invoices side by side, though, and the picture resolves: what’s being consumed is never the stock of content but the resources that let content regenerate. There are three of them.

The first is money. Bandwidth and servers cost real cash, and Wikimedia’s donations and SourceHut’s subscription fees were meant to be spent on human readers.

The second is attention. The curl case is worse than the bandwidth cases, because fake reports don’t just consume review time, they adulterate. Dumping garbage on a pasture and overgrazing it are two different harms. AI is doing both at once.

The third matters most: the contribution loop. The open web ran for thirty years on an implicit deal, crawling in exchange for traffic. Search engines crawl your pages and, in return, send searchers back to your site; the readers bring ad revenue, donations, new editors and new answerers, and the commons restocks itself. Cloudflare proposed a blunt metric for how much of that deal survives: the crawl-to-refer ratio, how many pages a platform crawls for each visitor it sends back. In Radar data from late June 2025, Anthropic’s ratio was roughly 70,900 to 1. By Cloudflare’s own conversion, a content creator trying to earn the referral traffic old Google once sent has it 750 times harder with OpenAI and 30,000 times harder with Anthropic. Even Google’s own referrals are now nearly 10 times harder to earn than they used to be, because more and more searches end on the results page itself: first the answer box, now AI summaries.

When an AI assistant speaks the answer directly to the user, the user stops visiting the answer’s source. For Q&A communities that’s a double hit: the readers stop coming, and so do the askers. No new questions, no new answers, and the corpus stops updating. One step further and you arrive at the standard structure of a commons tragedy: each AI company, viewed alone, is behaving rationally, since more scraping makes a better model; taken together, they are eating the pipeline that trains them. When next year’s frameworks and APIs ship, where will the next generation of models find human-written answers?

The same frame explains why voluntary norms couldn’t hold the line. The web’s traditional way of saying “don’t crawl me” is robots.txt, a plain-text file at the site root with no enforcement behind it, a gentleman’s agreement dating to 1994 and Martijn Koster; even the standards document, RFC 9309, states plainly that its rules “are not a form of access authorization.” (I took this mechanism apart a few days ago in my post on Claude share links ending up in Google’s index.) It mostly worked in the search era because search engines needed a long-term relationship with websites; good behavior was self-interest. The training-data game is different: one crawl is enough, no relationship required. If you show restraint and your competitor doesn’t, the model that falls behind is yours. Hence the arms race of residential IP pools and spoofed User-Agents. Garrett Hardin’s 1968 paper in Science opens with “a pasture open to all”; the open web wrote “open to all” into its design.

The fences are going up

The tragedy of the commons is not destiny. Elinor Ostrom won the 2009 Nobel in economics for showing as much: the real commons she studied, irrigation systems, fisheries, mountain pastures, endured for generations when boundaries were clear, monitoring was feasible, and violations carried costs. The open web is retrofitting all three in a hurry. You can see three kinds of fence right now.

The technical fence. Anubis, released in January 2025 by developer Xe Iaso, puts a proof-of-work gate in front of a site: a browser has to solve a small hash puzzle before it gets the page. A single human barely notices; a scrape measured in millions of requests pays a real compute bill. GNOME’s GitLab, WINE, and UNESCO are among the deployments (The Register). It isn’t an endgame: Codeberg reported in August 2025 that crawlers had learned to solve the challenges, and the arms race continues.

The market fence. As of July 1, 2025, Cloudflare made blocking AI crawlers the default for newly onboarded sites and launched pay-per-crawl: if you want to scrape, you negotiate a price first. Crawling flipped from permitted by default to denied by default, a deliberate reversal of “open by default” at the infrastructure layer.

The institutional fence. Wikimedia wrote crawler reduction into its 2025–2026 annual plan, targeting a 20% cut in request volume and 30% in bandwidth, while steering commercial users toward the paid Wikimedia Enterprise API: the data still flows, through the front door, at cost.

All three fences point toward the same ending. England’s actual pasture commons mostly ended in the enclosure movements of the 18th and 19th centuries: the grass was saved, the commons was not. The digital commons is walking that road now, and the costs of the fences fall very unevenly. The big AI labs can afford the tolls, and Cloudflare and Wikimedia can collect them. Squeezed in the middle are the small sites and volunteer projects with no channel to negotiate, and the legitimate crawling that was never about training: academic research, web archiving, accessibility tools. Fences don’t ask about intent.

What to do with this. If you maintain public infrastructure, “crawlers are adversarial” should be your default assumption: rate-limit the computationally expensive endpoints (search, diff, history) separately, treat Anubis or an equivalent as standard kit the way HTTPS is, and don’t count on robots.txt. If you work on data or policy at an AI company, the crawl-to-refer ratio will likely move from blog charts to negotiating tables and regulators’ dashboards; getting yours down before your peers do is cheap insurance. If you’re neither, just a reader, you’ll feel the fences first: more login walls, more “please verify you are human.”

The grass isn’t dead. It’s being fenced. Fences can save the grass; what they can’t save is the original meaning of “commons”: anyone gets in, no questions asked, no fee charged. The corpus the open web accumulated over thirty years raised this generation of AI. Once the fences close, the next person who wants to start from nothing and read the whole web, whether a student, a researcher, or a founder, will find that web is no longer there.

References