Publishing something publicly used to involve a familiar bargain.
People could read it. Search engines could index it. Other websites could link to it. The creator gave up some control in exchange for discoverability.
AI crawling has complicated that bargain.
A 2024 survey of 459 artists found broad concern about generative-AI systems using artists’ work without explicit consent, including concerns about disclosure, ownership, compensation, and who profits from the resulting models. See Foregrounding Artist Opinions.
Those concerns are no longer merely philosophical. Website infrastructure now includes specific controls for AI crawlers.
Cloudflare’s current AI Crawl Control lets site owners monitor AI services and allow or block individual crawlers. In 2025 Cloudflare said more than one million customers had already enabled its one-click AI crawler blocking option before the company expanded its permission-based controls. See Cloudflare’s 2025 announcement.
Public to humans does not have to mean public to every machine
The web has always contained access rules.
Robots.txt can request that automated crawlers avoid certain paths. Rate limits can slow automated requests. Firewalls can block particular bots. Login walls can restrict content to members.
What has changed is the reason some publishers are reaching for those tools.
Cloudflare’s own 2025 explanation framed the problem as a choice faced by creators and publishers: allow AI crawlers to collect material with little referral traffic, or block them and potentially lose useful relationships. Its later tooling introduced finer options such as allow, block, or charge. See Introducing AI Crawl Control.
That is a more precise response than simply taking the whole website offline.
But not every creator has sophisticated infrastructure.
Some may choose the simpler solution: post less publicly, place high-value work behind a login, share previews instead of originals, or move discussion into member spaces.
Blocking the scraper can also block the archaeologist
Access restrictions aimed at automated collection have side effects.
A login wall that keeps training crawlers out may also keep ordinary search engines, archivists, researchers, and casual readers out. A creator who shares finished work only inside a private community gains control but leaves less public evidence behind.
That tradeoff matters for Dead Internet Theory.
A quieter public web can be partly defensive. Creators may still be making exactly as much art, writing, photography, music, and commentary as before while exposing less of the full material to anonymous visitors and automated collectors.
The work did not disappear.
The front door acquired a bouncer.