A website’s robots.txt file is usually written for one audience: search-engine crawlers. It lists paths that Google and Bing should skip. For most of the web’s history, archives quietly consulted the same file too, and that file — not crawling effort — has shaped what survives.
How crawler rules gate access and capture
robots.txt is a convention from the mid-1990s, not a law. It announces which parts of a site a bot should not fetch, and indexing bots follow it because it is in their interest to be let back in. Web archives followed it too, and they treated it two ways at once. The file decided whether pages were captured when a crawler passed by, and it decided whether already-stored captures could be shown to the public.
That second role is odd, because robots.txt is written against current conditions. The Internet Archive’s own team documented the result: a site goes out of business, the domain is re-registered to a parking service that ships a deny-all robots.txt, and the whole history of the site stops being viewable in the Wayback Machine. Nobody deleted the pages. A new file simply hid them.
Separate the policy from the coverage
The Archive changed course in 2016–17, but not as absolutely as this history is sometimes retold. It first stopped consulting robots.txt for U.S. government and military sites. In April 2017, Wayback Machine director Mark Graham said the Archive wanted to rely less on robots.txt more broadly because current rules could hide older captures after a domain changed hands. The post described a direction, not a universal declaration that robots exclusions could never affect future collection.
The old replay behavior created a particularly confusing failure mode: a newly restrictive robots.txt could make historical captures disappear from public view even though the older pages had already been stored. That is why a past “blocked by robots.txt” notice could describe access policy rather than literal absence of captured bytes.
What a missing capture can and cannot prove
An empty Wayback entry proves little. It cannot establish that a page never existed, was removed deliberately, or was censored. Usually it means something mundane: nothing linked to the page, so no crawler found it; the crawler passed while the site excluded it; the page sat behind a login or JavaScript that left nothing to store; or stored copies exist but are being filtered from view.
Recovery has hard limits. If the underlying captures survived and the block was only a file, removing the rule has restored full histories — documented as far back as 2006. If a page was never fetched because a robots.txt excluded the crawler at the time, there is no stored copy to restore, and no archive holds what no crawler recorded. Other archives besides the Internet Archive may fill a gap, but a page that was never crawled anywhere is simply gone from the record.
Exclusion is back on the schedule
In 2025 and 2026 the issue returned in a new form. Nieman Lab found hundreds of news sites limiting Internet Archive crawlers amid concerns that archived copies could become a route for AI scraping; by May 2026 its follow-up counted more than 340 U.S. local-news sites in that category. Some restrictions were expressed in robots.txt and others through harder anti-scraping controls. The result is the same preservation problem: material a crawler cannot fetch today cannot become tomorrow’s snapshot.
When you meet a “blocked” notice or a bare gap, assume nothing. Ask whether the file blocks crawlers or only search engines, whether the block predates or postdates the material you care about, and whether another archive captured the page. A missing snapshot tells you how the web was crawled, which is not the same as knowing what the web contained.