Category: Digital Extinction (What Time Forgot)

The internet forgets.

Websites disappear. Forums go silent. Downloads vanish, domains expire, and millions of links now lead nowhere. Entire communities that once felt permanent survive only in fragments, screenshots, archives, and the memories of the people who were there.

**Digital Extinction — What Time Forgot** explores the lost, abandoned, and disappearing web: the places, files, communities, and strange little corners of the internet that time quietly erased.

Posted on by

Archive replay errors that make preserved content appear lost

Some archived pages open broken. Instead of the old home page you get a blank screen, empty image boxes, or an error wall, and the natural read is that the content is gone. Sometimes the archive does hold captures of the page or some of its resources; what failed is the replay step that tries to reconstruct them into a usable view.

How a saved page is rebuilt

Web archives do not store a rendered screen. A crawl keeps the HTML document and, separately, the images, stylesheets, and scripts it references, each stamped with its own capture time. To display a page, the Wayback Machine rewrites links inside the stored HTML so they point at stored copies of those assets. When an asset was never captured — it lived on another host, sat behind a robots.txt ban, or was too deep for the crawler — the rewritten link points at a hole. The words survive; the layout collapses around them.

Preserved but locked out

For years the Wayback Machine applied robots.txt at replay, not just at crawl time. The file, invented in the mid-1990s to steer search-engine crawlers, could be applied years later to erase a domain’s past: a site closes, the parked domain adds a disallow, and the archived history vanishes from view, though no capture was deleted. The Internet Archive acknowledged the complaints in an April 2017 announcement about such “disappeared” sites, then said it would rely less on robots.txt, extending a change it had already made for U.S. government and military sites.

Replay can fail for reasons that have nothing to do with deletion: missing embedded resources, JavaScript that no longer executes correctly, incompatible media players, or temporary access errors can all leave a stored capture looking empty or damaged. Archive-It’s replay troubleshooting guidance explicitly recommends checking crawl reports to determine whether media was collected even when Wayback cannot replay it.

Capture gap or replay failure?

Ask the index, not only the rendered page. The Wayback Machine’s CDX index API lists capture records for a URL independently of normal replay. If no capture record exists for the URL and time you need, that points toward a collection gap. If capture records do exist while replay is broken, inspect other timestamps and individual captured resources before concluding the material itself is absent.

What “preserved” really means

Stored, restorable, and reachable are three different states, and replay lives between the last two. A page that refuses to open is not proof of loss, any more than a clean opening proves completeness — the text can be a shell whose images were never captured. These errors are a request to look closer, not a verdict. The page you want may be one good replay away.

Posted on by

Orphaned files reachable by a known URL but absent from every index

Give a browser the exact address and a server answers, whether or not any page links to that file. Being reachable and being findable are different things, and the gap widens when nobody maintains the links. An orphaned file is a URL that still returns a normal response while every route to it has quietly disappeared.

Files outlive their links

Search engines do not roam the internet freely. Google’s own account of how Search works says there is no central registry of pages: a URL is usually discovered because a known page links to it or a site submits a sitemap, and there is no guarantee it will be crawled, indexed, or served. Redesign a site, drop an old directory from navigation, and the files in it stop being visited without being deleted.

The Old Dominion longitudinal URL study, which examined 27.3 million URLs spanning the early web through 2023, found that most URLs eventually disappeared from the live web even while some survived for decades. That churn helps create orphans: files can remain reachable after the navigation, directories, and search results that once exposed them have changed.

Absent from an index is not deleted

The Internet Archive’s 2026 re-analysis of a Pew dataset drew a related line from the archive side. Of 5.4 million sampled URLs, 18 percent were still alive on the web but absent from the Wayback Machine, a category the Archive called “endangered.” Importantly, the analysis did not check every smaller public archive, so absence from Wayback is not proof that no archive anywhere has a copy.

An index is a finding aid, not a census. A file loaded thousands of times can show up in no search results at all.

The known URL is the recovery key

Recovery runs on references. The address may survive in an old bookmark, printed citation, forum post, server log, directory listing, or archived links page. That is also the limit. An unindexed URL whose address was never preserved anywhere is effectively unrecoverable, even if the server would still answer it. Files outlive their links all the time. Proving that something is absent from every index is nearly impossible; the practical preservation question is whether any surviving reference still tells you the address. If none does, a live file can be as unreachable as a deleted one.

Posted on by

Geographic access restrictions and uneven archive coverage

The internet does not present the same face to everyone. A server decides what to send based on where the request appears to come from, and that treats human readers and crawlers alike: both arrive at a site from some country. What an archive holds is therefore shaped less by how hard it crawled and more by where its crawlers happened to sit.

Where you connect changes what you see

Geoblocking is server-side denial of access based on where a request appears to come from. A large 2018 measurement study tested major sites from vantage points in 177 countries and found CDN-based geoblocking in nearly all of them; in its broader sample, about 4.4 percent of domains used a CDN geoblocking feature in at least one country. The motive can be legal, commercial, sanctions-related, or operational rather than political.

That blur is worth remembering. The same study found that 9 percent of a widely used list of censored domains returned an ordinary block page in at least one country. Geographic restriction can look exactly like censorship, and vice versa, which is precisely why crawlers get confused.

A crawler’s vantage point becomes an archive’s bias

Any archive crawler has a geographic vantage point. If a site blocks that region, serves a regional substitute, or varies content by location, the archive may preserve the block page or the regional version rather than what visitors elsewhere saw.

Different archives also have different collection mandates and strengths. A 2013 study profiling twelve public web archives found that holdings varied enough by top-level domain and language that querying only one archive could miss captures held elsewhere. Earlier research likewise found substantial country-level imbalance in Internet Archive coverage. Geography is therefore one source of unevenness among several.

Regional captures add a second record

The practical fix is comparison, not faith in one universal archive. National archives, institutional collections, and other public web archives may hold different snapshots of the same site because they crawl different scopes at different times and from different infrastructure. The Memento protocol was designed to help systems discover versions across multiple archives, although its original public Time Travel aggregator was discontinued in 2025.

What no archive can rebuild

Recovery has hard limits. If every archive that might have encountered a page was served a block page, and no alternative collector preserved the real version, there is no hidden master copy to reconstruct later.

Treat coverage claims the way you treat the robots.txt logic: suspicious. Before concluding that old content is lost, check whether it looks different from inside the country where it lived. The record of the web is not one archive — it is the sum of every vantage point that happened to look, including yours.

Posted on by

Personalized pages without a single canonical version to archive

Two people can open the same address at the same minute and read different pages. One gets the national edition, the other local headlines; one the full site, the other a stripped-down page built for a phone. The Web Archive holds copies of that address, but no single copy stands for what either person saw, because each page was constructed separately for its visitor.

One address, many versions

The variation was documented precisely in a 2013 web-archiving paper. Researchers at Old Dominion University crawled the CNN front page twice with the same URL, changing only the user-agent string so the site saw a desktop browser in one crawl and an iPhone in the other. The site returned a different page each time, and both captures sat in the archive under the same address, distinguishable only by content. The difference was rarely subtle: one ESPN page held 201 links on desktop and 58 on mobile, and NBC News served different local stories and weather depending on where a request appeared to come from.

The same variation arrives through signals no user-agent can carry: geolocation, cookies, login state, A/B tests, and reading history. None of these make the URL lie; they mean a URL can name a family of representations rather than one fixed document. The New York Times experimented with personalized homepage modules based on signals including location and reading behavior, exactly the kind of publishing where one archived homepage cannot represent every visitor’s experience.

What one capture keeps

A crawler arrives with a particular browser profile, network location, session state, and crawl configuration. The Library of Congress notes in its web-archiving FAQ that archived sites may differ from the live web because interactive and dynamically generated material cannot always be captured completely. A personalized page adds another problem: even a technically perfect capture may represent only one visitor state. Everything the crawler did not look like goes unrecorded: the local section a New York IP would have seen, the recommendations built from a logged-in reading history, the variant served in an A/B test.

The archived copy is faithful to its own visit. Nothing preserves the relationship between that visit and the ones it missed, and nothing but the content distinguishes the two CNN pages under one URL. A history of the web that treats each address as a single evolving document quietly misreads a web where documents are a population.

Context needed to read a capture

Interpreting a preserved version later requires knowing what the page was built for. When was it captured, with which software, from which region, as which device, logged in or not, after which scripts ran? The Old Dominion study argues that those environmental dimensions belong beside the timestamp if future readers are going to understand why two captures of one URL differ.

Recovery has a hard limit. A version nobody captured is gone regardless of how much context survives, and the only full record of what a particular person saw was that person’s own browser. As pages are assembled per viewer, the archive captures one witness’s testimony to a page that existed in many forms. Preserving the personalized web means asking who saw it — because the thing nobody saw was never on the web at all.

Posted on by

CAPTCHAs and anti-bot barriers that also block preservation

A CAPTCHA or anti-bot challenge is built to distinguish ordinary human browsing from automation. Some preservation crawlers are simple HTTP clients; newer ones can drive real browsers and execute JavaScript. Either way, the archive still has to survive whatever scoring, challenge, or blocking system sits in front of the site. The same defenses that stop abusive automation can therefore stop preservation too.

Why a challenge stops a crawler cold

A site can refuse automation more forcefully than robots.txt ever could. Modern anti-bot systems score traffic using browser behavior, network reputation, request patterns, and scripted challenges. Archive-It’s own documentation notes that sites may block crawler IP ranges or otherwise prevent capture, even when the underlying content is public; its crawler overview explains why browser-driven capture helps but does not make an archive invisible to anti-bot defenses.

What the archive keeps instead

That enforcement leaves a visible shape. When a crawler hits the wall, the archive often stores the wall itself: a “Checking your browser” frame, a “Something went wrong” error, or a three-second animation frozen into a memento. A reader checking the archive later cannot always tell whether a page was removed, censored, or simply refused a handshake. The barrier gets archived whole, and the record is the wall’s side of a conversation.

This differs from the other gates in this series: robots.txt is a request an archive can weigh, a login wall sits behind an account, while a CAPTCHA is enforced mid-request on anonymous traffic, judging how a visitor looks rather than who they claim to be.

A clean example: Stack Exchange blocks the archive by accident

A useful case appeared on Stack Exchange in May 2025. Users noticed that Wayback Machine saves were failing because Cloudflare was intercepting the requests. A Stack Exchange staff member confirmed that an anti-scraping bot rule had caught the Internet Archive and changed the rule to exempt it. Archiving immediately began working again.

The example is valuable because nobody had deleted the questions and nobody had decided they should be private. The preservation failure was a side effect of a security rule. Had the block remained in place for years, the archive gap would have looked intentional from the future even though it began as collateral damage.

Preserving without dismantling the defenses

The fix is not to tear defenses down; sites have good reasons for them, from credential stuffing to abusive scraping. It is to make room for known preservation paths when the operator wants a public record. Allowlisting archival crawlers, offering exports, or arranging authenticated preservation can preserve material without weakening the public-facing barrier. If every automatic visitor is treated identically, the web can keep its defenses and quietly lose its memory.

Posted on by

Robots exclusions and gaps in historical archive coverage

A website’s robots.txt file is usually written for one audience: search-engine crawlers. It lists paths that Google and Bing should skip. For most of the web’s history, archives quietly consulted the same file too, and that file — not crawling effort — has shaped what survives.

How crawler rules gate access and capture

robots.txt is a convention from the mid-1990s, not a law. It announces which parts of a site a bot should not fetch, and indexing bots follow it because it is in their interest to be let back in. Web archives followed it too, and they treated it two ways at once. The file decided whether pages were captured when a crawler passed by, and it decided whether already-stored captures could be shown to the public.

That second role is odd, because robots.txt is written against current conditions. The Internet Archive’s own team documented the result: a site goes out of business, the domain is re-registered to a parking service that ships a deny-all robots.txt, and the whole history of the site stops being viewable in the Wayback Machine. Nobody deleted the pages. A new file simply hid them.

Separate the policy from the coverage

The Archive changed course in 2016–17, but not as absolutely as this history is sometimes retold. It first stopped consulting robots.txt for U.S. government and military sites. In April 2017, Wayback Machine director Mark Graham said the Archive wanted to rely less on robots.txt more broadly because current rules could hide older captures after a domain changed hands. The post described a direction, not a universal declaration that robots exclusions could never affect future collection.

The old replay behavior created a particularly confusing failure mode: a newly restrictive robots.txt could make historical captures disappear from public view even though the older pages had already been stored. That is why a past “blocked by robots.txt” notice could describe access policy rather than literal absence of captured bytes.

What a missing capture can and cannot prove

An empty Wayback entry proves little. It cannot establish that a page never existed, was removed deliberately, or was censored. Usually it means something mundane: nothing linked to the page, so no crawler found it; the crawler passed while the site excluded it; the page sat behind a login or JavaScript that left nothing to store; or stored copies exist but are being filtered from view.

Recovery has hard limits. If the underlying captures survived and the block was only a file, removing the rule has restored full histories — documented as far back as 2006. If a page was never fetched because a robots.txt excluded the crawler at the time, there is no stored copy to restore, and no archive holds what no crawler recorded. Other archives besides the Internet Archive may fill a gap, but a page that was never crawled anywhere is simply gone from the record.

Exclusion is back on the schedule

In 2025 and 2026 the issue returned in a new form. Nieman Lab found hundreds of news sites limiting Internet Archive crawlers amid concerns that archived copies could become a route for AI scraping; by May 2026 its follow-up counted more than 340 U.S. local-news sites in that category. Some restrictions were expressed in robots.txt and others through harder anti-scraping controls. The result is the same preservation problem: material a crawler cannot fetch today cannot become tomorrow’s snapshot.

When you meet a “blocked” notice or a bare gap, assume nothing. Ask whether the file blocks crawlers or only search engines, whether the block predates or postdates the material you care about, and whether another archive captured the page. A missing snapshot tells you how the web was crawled, which is not the same as knowing what the web contained.

Posted on by

Streaming media missed by page-level archiving

Save a web page and you can usually replay it: the HTML, the images, the styles come back as files. Save a page carrying a video and you may end up with a player and nothing to play. Audio and video on the modern web are rarely a single file parked at one URL. They are streams, a timed conversation between a player and a server, and a crawler that only fetches pages is not part of that conversation.

Why the stream slips past a page crawl

The difference is delivery. Progressive download puts a whole media file on a server and lets the browser pull it the way it pulls an image: one request, one file, happily replayed. Streaming is the opposite. In the Flash era, video travelled over RTMP, a protocol on its own port. Today’s adaptive formats, Apple’s HLS and MPEG-DASH, run over HTTP but still offer no single file: the player asks for a playlist or manifest, which lists dozens of short segments, which the player fetches in sequence from CDN addresses that can expire in minutes. No step ever asks for “the video.” A crawler that does catch the playlist catches a short text file; the segments it names are separate requests, often to addresses that have already rotated.

A page crawl never gets there. Heritrix, the crawler behind the British Library’s web archiving, could capture data over HTTP and FTP and nothing else, and the consequence surfaced in 2009 when the Library took on One & Other, Antony Gormley’s art project on the fourth plinth in Trafalgar Square. For a hundred days, 2,400 participants each took the plinth for an hour while cameras recorded them: 2,400 hours of .flv video, about a terabyte, streamed over RTMP. Test crawls of oneandother.co.uk returned only static HTML pages without the videos. The pages were archived. The art was not.

What a complete capture has to keep

The British Library’s remedy shows what media capture costs. Instead of crawling, staff ran a browser together with a streaming recorder, Jaksta, that caught the packets flowing to the embedded Flash player over RTMP. Four virtual machines scheduled at 90-minute intervals did the work. For replay they stood up Red5, a Java media server, and Flowplayer, then patched the Wayback banner to swap the original players for archived ones, necessary because the sponsor’s licence allowed streaming only, with copy protection equivalent to the live site.

The pattern has not aged away. On HTTP-based platforms, media capture still needs tooling beyond a page fetch. Archive-It recommends browser-driven Brozzler crawls and checks whether the media files themselves were collected; its current streaming-media guidance also notes that some services require yt-dlp support and that some captured media still will not replay correctly in Wayback. A playlist without its segments is only a list of missing pieces.

The limits bear repeating. The Library’s pipeline sat outside its normal workflow: the videos were served from a separate machine, not packaged with the rest of the archive. Some videos never replayed to full length, and re-capturing did not help; the sponsor later supplied the original masters on disk, which carried the same errors. The archive could not repair a source that was broken at birth. And a stream is only captured while it is running. Once it ends unrecorded, there is nothing to re-crawl.

A complete capture therefore needs more than the page: the playlist or manifest, every segment or file, the codecs, any keys, and a record of what was shown and when. Otherwise the archive keeps a play button that points nowhere.

Posted on by

Form-driven databases that cannot be preserved by following links

A web crawler starts at the entry page and follows every link it can reach. Content that exists only in answer to a search question falls outside that walk: the database has no link to offer, so it produces a page only when a human types a query.

Records that exist only when asked

In 2001 Michael Bergman used the term deep Web for material exposed through searchable databases and estimated at the time that it dwarfed the conventionally crawlable web. The exact scale estimate aged badly; the underlying mechanical point did not. A database may reveal records only after it receives a query.

Every search box is that kind of database. A crawler never sees what the box reveals, because the records have no addresses until someone asks for them.

The catalogue behind the form

The Haddon catalogue shows the cost. Built by Marcus Banks in Oxford with UK Economic and Social Research Council funding, it documented about 1,000 pre-war ethnographic films, searchable online from 1996.

The hardware aged. By 2005 the database engine no longer ran on current operating systems, funding had ended, and the catalogue went dark. The Oxford-led Gone Dark study documented the case: web archiving had not captured the searchable database, but the underlying data had been preserved offline, making later revival possible.

The same gap in the archives

The same study’s Kwetu case shows the pattern in web archives: the Wayback Machine holds the front pages and images, but “the search function does not work and no access to anything behind the search paywall is available.” The database behind Kwetu.net’s search portal, over one million manuscripts, stayed out of every crawl and lives on only in former owners’ hands.

Exports and documented queries

What preserves a form-driven database is not a crawl of its search page but an export of its contents, or ordinary links that give records stable addresses. The UK National Archives calls material reachable only through forms, pick lists, or search boxes not “machine reachable” and recommends static links or downloadable alternatives.

Documented queries capture a fraction. Each saved query is one row of a larger table; fifty useful questions preserve fifty answers, not the five thousand records behind the form. A database dump preserves everything, but it needs the operator’s cooperation, which a crawler never gets.

The form is a door, not the archive. What survives a database’s death is whatever was copied to disk while the door was still open.

Posted on by

Dynamic pages that web crawlers capture only as empty shells

Some pages reach the browser half-finished. The server sends scaffolding — a title, empty containers, a script tag — and the visible content arrives afterwards, built by JavaScript that fetches the rest from the site’s own service. Ask for the page without running the script, and you get the envelope with no letter inside.

The page that assembles after you load it

Open an account page or a live dashboard and the first HTML holds little more than a spinner. The real text and figures come in a second delivery: scripts call the site’s private API and pour results into the empty containers. The underlying data sits in a database the visitor never touches. To a crawler, the page is a promise that something will appear, provided a script runs.

Why a crawler saves the shell

Traditional link-following crawlers are strongest when useful content arrives in the initial HTML and its linked resources. Client-side rendering complicates that model because important URLs may not appear until JavaScript executes or a visitor scrolls, clicks, or hovers. Archive-It now documents this explicitly: its standard crawler combines Heritrix with Umbra so browser-like processing can expose resources that plain crawling would miss.

Reddit’s 2018 redesign turned threads into exactly that kind of page, and its archived copies began opening as blank shells — an empty container beneath the Wayback Machine banner — while the same discussions, captured on server-rendered old.reddit.com, kept their text. As one bug-report commenter noted, sites generated entirely in the browser are only going to get harder to archive.

Fuller capture means running a browser

Recovery often means giving the crawler a real browser environment. Archive-It’s newer Brozzler technology runs an actual browser and records the exchanges between that browser and the site. Even then, Archive-It warns that dynamic content can remain difficult to capture or replay completely, especially when interaction is required.

Even that captures one moment of a moving thing

A rendered snapshot is still a single view of a live system. Content behind a login becomes a login wall; scroll-loaded feeds show only the first screen. Prices, scores, recommendations, and weather vary by visitor and by instant, and the database behind the API is never captured. The archive keeps one assembly among many, made for one viewer, not the site’s true contents.

This is an archive gap with no deletion in it. The threads were not removed; live browsers still drew them. The content no longer travels with what a crawler records, so archives end up with storefronts whose stock sits in a warehouse they cannot enter. Dynamic pages are written into the record half-delivered.