A CAPTCHA or anti-bot challenge is built to distinguish ordinary human browsing from automation. Some preservation crawlers are simple HTTP clients; newer ones can drive real browsers and execute JavaScript. Either way, the archive still has to survive whatever scoring, challenge, or blocking system sits in front of the site. The same defenses that stop abusive automation can therefore stop preservation too.
Why a challenge stops a crawler cold
A site can refuse automation more forcefully than robots.txt ever could. Modern anti-bot systems score traffic using browser behavior, network reputation, request patterns, and scripted challenges. Archive-It’s own documentation notes that sites may block crawler IP ranges or otherwise prevent capture, even when the underlying content is public; its crawler overview explains why browser-driven capture helps but does not make an archive invisible to anti-bot defenses.
What the archive keeps instead
That enforcement leaves a visible shape. When a crawler hits the wall, the archive often stores the wall itself: a “Checking your browser” frame, a “Something went wrong” error, or a three-second animation frozen into a memento. A reader checking the archive later cannot always tell whether a page was removed, censored, or simply refused a handshake. The barrier gets archived whole, and the record is the wall’s side of a conversation.
This differs from the other gates in this series: robots.txt is a request an archive can weigh, a login wall sits behind an account, while a CAPTCHA is enforced mid-request on anonymous traffic, judging how a visitor looks rather than who they claim to be.
A clean example: Stack Exchange blocks the archive by accident
A useful case appeared on Stack Exchange in May 2025. Users noticed that Wayback Machine saves were failing because Cloudflare was intercepting the requests. A Stack Exchange staff member confirmed that an anti-scraping bot rule had caught the Internet Archive and changed the rule to exempt it. Archiving immediately began working again.
The example is valuable because nobody had deleted the questions and nobody had decided they should be private. The preservation failure was a side effect of a security rule. Had the block remained in place for years, the archive gap would have looked intentional from the future even though it began as collateral damage.
Preserving without dismantling the defenses
The fix is not to tear defenses down; sites have good reasons for them, from credential stuffing to abusive scraping. It is to make room for known preservation paths when the operator wants a public record. Allowlisting archival crawlers, offering exports, or arranging authenticated preservation can preserve material without weakening the public-facing barrier. If every automatic visitor is treated identically, the web can keep its defenses and quietly lose its memory.