Some pages reach the browser half-finished. The server sends scaffolding — a title, empty containers, a script tag — and the visible content arrives afterwards, built by JavaScript that fetches the rest from the site’s own service. Ask for the page without running the script, and you get the envelope with no letter inside.
The page that assembles after you load it
Open an account page or a live dashboard and the first HTML holds little more than a spinner. The real text and figures come in a second delivery: scripts call the site’s private API and pour results into the empty containers. The underlying data sits in a database the visitor never touches. To a crawler, the page is a promise that something will appear, provided a script runs.
Why a crawler saves the shell
Traditional link-following crawlers are strongest when useful content arrives in the initial HTML and its linked resources. Client-side rendering complicates that model because important URLs may not appear until JavaScript executes or a visitor scrolls, clicks, or hovers. Archive-It now documents this explicitly: its standard crawler combines Heritrix with Umbra so browser-like processing can expose resources that plain crawling would miss.
Reddit’s 2018 redesign turned threads into exactly that kind of page, and its archived copies began opening as blank shells — an empty container beneath the Wayback Machine banner — while the same discussions, captured on server-rendered old.reddit.com, kept their text. As one bug-report commenter noted, sites generated entirely in the browser are only going to get harder to archive.
Fuller capture means running a browser
Recovery often means giving the crawler a real browser environment. Archive-It’s newer Brozzler technology runs an actual browser and records the exchanges between that browser and the site. Even then, Archive-It warns that dynamic content can remain difficult to capture or replay completely, especially when interaction is required.
Even that captures one moment of a moving thing
A rendered snapshot is still a single view of a live system. Content behind a login becomes a login wall; scroll-loaded feeds show only the first screen. Prices, scores, recommendations, and weather vary by visitor and by instant, and the database behind the API is never captured. The archive keeps one assembly among many, made for one viewer, not the site’s true contents.
This is an archive gap with no deletion in it. The threads were not removed; live browsers still drew them. The content no longer travels with what a crawler records, so archives end up with storefronts whose stock sits in a warehouse they cannot enter. Dynamic pages are written into the record half-delivered.