Tag: Archive Gaps

Records that exist only when asked

In 2001 Michael Bergman used the term deep Web for material exposed through searchable databases and estimated at the time that it dwarfed the conventionally crawlable web. The exact scale estimate aged badly; the underlying mechanical point did not. A database may reveal records only after it receives a query.

Every search box is that kind of database. A crawler never sees what the box reveals, because the records have no addresses until someone asks for them.

The catalogue behind the form

The Haddon catalogue shows the cost. Built by Marcus Banks in Oxford with UK Economic and Social Research Council funding, it documented about 1,000 pre-war ethnographic films, searchable online from 1996.

The hardware aged. By 2005 the database engine no longer ran on current operating systems, funding had ended, and the catalogue went dark. The Oxford-led Gone Dark study documented the case: web archiving had not captured the searchable database, but the underlying data had been preserved offline, making later revival possible.

The same gap in the archives

The same study’s Kwetu case shows the pattern in web archives: the Wayback Machine holds the front pages and images, but “the search function does not work and no access to anything behind the search paywall is available.” The database behind Kwetu.net’s search portal, over one million manuscripts, stayed out of every crawl and lives on only in former owners’ hands.

Exports and documented queries

What preserves a form-driven database is not a crawl of its search page but an export of its contents, or ordinary links that give records stable addresses. The UK National Archives calls material reachable only through forms, pick lists, or search boxes not “machine reachable” and recommends static links or downloadable alternatives.

Documented queries capture a fraction. Each saved query is one row of a larger table; fifty useful questions preserve fifty answers, not the five thousand records behind the form. A database dump preserves everything, but it needs the operator’s cooperation, which a crawler never gets.

The form is a door, not the archive. What survives a database’s death is whatever was copied to disk while the door was still open.

Some pages reach the browser half-finished. The server sends scaffolding — a title, empty containers, a script tag — and the visible content arrives afterwards, built by JavaScript that fetches the rest from the site’s own service. Ask for the page without running the script, and you get the envelope with no letter inside.

The page that assembles after you load it

Open an account page or a live dashboard and the first HTML holds little more than a spinner. The real text and figures come in a second delivery: scripts call the site’s private API and pour results into the empty containers. The underlying data sits in a database the visitor never touches. To a crawler, the page is a promise that something will appear, provided a script runs.

Why a crawler saves the shell

Traditional link-following crawlers are strongest when useful content arrives in the initial HTML and its linked resources. Client-side rendering complicates that model because important URLs may not appear until JavaScript executes or a visitor scrolls, clicks, or hovers. Archive-It now documents this explicitly: its standard crawler combines Heritrix with Umbra so browser-like processing can expose resources that plain crawling would miss.

Reddit’s 2018 redesign turned threads into exactly that kind of page, and its archived copies began opening as blank shells — an empty container beneath the Wayback Machine banner — while the same discussions, captured on server-rendered old.reddit.com, kept their text. As one bug-report commenter noted, sites generated entirely in the browser are only going to get harder to archive.

Fuller capture means running a browser

Recovery often means giving the crawler a real browser environment. Archive-It’s newer Brozzler technology runs an actual browser and records the exchanges between that browser and the site. Even then, Archive-It warns that dynamic content can remain difficult to capture or replay completely, especially when interaction is required.

Even that captures one moment of a moving thing

A rendered snapshot is still a single view of a live system. Content behind a login becomes a login wall; scroll-loaded feeds show only the first screen. Prices, scores, recommendations, and weather vary by visitor and by instant, and the database behind the API is never captured. The archive keeps one assembly among many, made for one viewer, not the site’s true contents.

This is an archive gap with no deletion in it. The threads were not removed; live browsers still drew them. The content no longer travels with what a crawler records, so archives end up with storefronts whose stock sits in a warehouse they cannot enter. Dynamic pages are written into the record half-delivered.