Posted on

Measuring web loss without confusing absence from an archive with nonexistence

There is a tempting but dangerous equation in web archaeology:

No Wayback capture = the page did not exist.

That conclusion feels reasonable because web archives are enormous. It is also wrong.

An archive can fail to contain a page for many reasons that have nothing to do with whether the page once existed. A crawler may never have discovered it. The site may have blocked crawling. The page may have required a form submission or login. The resource may have been streaming media, a database result, or dynamically generated content that the crawler could not preserve correctly.

Archives are samples, not omniscient recordings

The Library of Congress describes web archiving as a process that begins with selected seed URLs and follows links according to collection scope. Its own technical guidance acknowledges that current tools cannot capture all web content, specifically naming difficult categories such as streaming media, deep-web content, databases, and multimedia-rich experiences.

That means archive coverage is shaped by selection and technology before researchers ever begin measuring loss.

A direct measurement of web decay shows why definitions matter. Pew Research Center’s 2024 study of disappearing online content sampled pages from Common Crawl and then tested whether those URLs were still accessible on the live web. It found 38 percent of sampled pages from 2013 inaccessible by 2023. That is a measurement of live accessibility, not a claim that 38 percent had vanished from every archive and mirror.

An absent or poorly timed archive capture is therefore evidence about the archive’s holdings. It is not automatically evidence about the historical Web.

Define what “lost” means before counting it

A study of web loss needs a starting population. One useful method is to begin with URLs known to have existed because they appear in an old directory, published bibliography, crawl dataset, sitemap, software package, or contemporaneous list.

Researchers can then ask separate questions:

Is the URL live today? Does it redirect? Does the same document survive at another URL? Is there a capture in one archive? In several archives? Does an independent copy survive in a PDF, mirror, repository, CD-ROM, or email attachment?

Those questions produce different kinds of loss.

A dead original URL is link loss. A missing page at its original domain may still have a complete archived copy. A page absent from the Wayback Machine may survive in another web archive. A document may have moved without a redirect. And a genuinely vanished work may leave only quotations or screenshots.

Collapsing all of those conditions into “gone” produces a dramatic number and a weak study. Pew’s 2024 study, for example, found that 38 percent of its sampled 2013 pages were no longer accessible a decade later. When the Internet Archive later rechecked Pew’s broader dataset against the Wayback Machine, it showed that many URLs dead on the live web still had archived copies, while also warning that it had not checked every smaller web archive. “Dead live URL,” “absent from Wayback,” and “vanished everywhere” are different measurements.

Careful claims are narrower and more useful

The strongest language matches the evidence: “No capture was found in the archives searched” rather than “the page was never archived anywhere.” “The original URL no longer resolves” rather than “the document was deleted.” “The page was observed live in 2012 and dead in 2017” rather than inventing an exact disappearance date between those observations.

This restraint matters to Dead Internet Theory because a real phenomenon—massive web decay—does not become stronger when archive gaps are counted as proof of disappearance.

The old Web has lost an extraordinary amount of material. Measuring how much requires resisting the urge to turn every blank spot in the archive into a gravestone.

Posted on

Preserving context when only screenshots remain

A screenshot is often what survives after everything interactive is gone.

It can preserve a headline, layout, username, photograph, comment, error message, or strange little piece of interface that no crawler captured. That makes screenshots valuable. It also makes them dangerously easy to overread.

A screenshot preserves pixels. A website is more than pixels.

What the image actually proves

With sound provenance, a screenshot can show that a particular screen displayed particular visible material at a recorded moment. It can preserve typography, relative placement, colors, advertisements, timestamps printed by the site, and content that was visible only after interaction or login. Without provenance, the pixels alone cannot prove when, where, or even whether the image came from the claimed live page.

What it usually does not preserve is the target of a hyperlink, the HTML source, hidden text, alt text, JavaScript state, network requests, content below the captured viewport, or the difference between something rendered by the server and something inserted by a browser extension.

That difference is why professional web archiving normally saves structured web material rather than only taking pictures of it. The Library of Congress describes the WARC web-archive format as a container for harvested resources plus record information such as target URI, date, type, and other metadata. Its preservation guidance also says archived content should identify the archiving institution, capture time, and limitations in functionality.

A PNG by itself carries far less context.

Provenance makes a screenshot stronger

Suppose the only surviving evidence of a deleted forum post is Screenshot_2024-03-11.png.

That is better than nothing, but a researcher immediately wants more. What was the full URL? Who took the screenshot? Was the user logged in? What time zone did the site’s displayed timestamp use? Was this the whole thread or a cropped section? Was the image edited after capture?

A useful preservation package records the original URL, capture date and time, browser or app, account state if relevant, and a short explanation of what was being documented. Keeping the uncropped original alongside any annotated derivative is better than saving only the version with arrows and circles added later. A checksum can help show that the preserved image itself has not changed since it entered the collection.

Nearby screenshots can also restore context. Capturing the page above and below the disputed item, the profile that posted it, or the navigation that led there can be more useful than one perfectly framed viral rectangle.

Screenshots are evidence, not a replacement website

Sometimes nothing better survives. In that case the screenshot deserves preservation precisely because it may be the last visible trace of the artifact.

But the limits should travel with it.

A screenshot cannot normally prove that a link worked. It cannot reconstruct a deleted video. It may not reveal whether text was selectable, dynamically loaded, or changed seconds later. And without provenance, future researchers may be unable to distinguish a genuine capture from a mockup.

The proper conclusion is not that screenshots are useless. It is that they need witnesses.

When the living webpage is gone, a screenshot may be the body. URLs, timestamps, capture notes, surrounding files, and independent corroboration are what give it a name.

Posted on

Community rescue downloads during a platform’s closing window

A platform shutdown announcement changes preservation from a long-term project into an evacuation.

Suddenly there is a deadline. Pages that have sat online for ten years may have ninety days left. Volunteers have to discover URLs, build crawlers, divide the workload, find storage, and decide which failures deserve another attempt before the servers go dark.

GeoCities is one of the clearest examples of what that looks like.

The closing notice creates a narrow window

Yahoo announced in 2009 that GeoCities would close, giving users and preservation groups a limited period to act. Archive Team’s account of the GeoCities rescue says its harvesting effort ran from April through October 2009 and involved several dozen people and hundreds of machine instances. In a contemporaneous update after only 48 hours, Jason Scott reported that more than 200,000 GeoCities sites had already been saved.

That scale sounds enormous until you remember what GeoCities was: years of personal homepages, fan sites, hobby pages, neighborhood directories, abandoned experiments, images, downloads, counters, frames, and broken links spread across a giant hosting platform.

No volunteer group could manually decide the historical importance of every page before shutdown day.

So rescue work becomes triage.

Volunteers save what they can discover

Large rescue projects usually begin by collecting URL lists and crawling broadly. Known account names, public indexes, search results, external links, sitemaps, and previously downloaded lists can all help identify material. Different volunteers may attack different address ranges or file types in parallel.

That strategy favors coverage over perfect interpretation. It is often better to save a million imperfectly cataloged files before the deadline than to beautifully describe ten thousand pages while the other 990,000 disappear.

The GeoCities rescue also demonstrates why multiple projects matter. The Internet Archive’s 2009 GeoCities special collection says its own deep crawls relied on public directories and links and explicitly warns that it did not have a comprehensive list of every GeoCities page. Archive Team, ReoCities, OoCities, and other projects captured overlapping but different portions of the service.

Independent rescue efforts accidentally create redundancy.

Emergency archives inherit emergency flaws

A rushed crawl can miss pages that were not linked publicly, content requiring login, scripts that generated pages dynamically, external images hosted elsewhere, robots-blocked resources, or files referenced through broken navigation. A crawler may save HTML while missing the JavaScript or media needed to reproduce how the page behaved.

It can also lose social context. A folder of GeoCities pages does not automatically preserve the neighborhood system, guestbook conversations, user identities, or the experience of navigating the service in 1999.

That does not make the rescue a failure. It changes what the collection can honestly claim to be.

A shutdown archive is evidence gathered under deadline. Its gaps should be documented rather than hidden.

The brutal advantage of a closing notice is that at least people know the clock is running. Many websites disappear without one. When a platform gives the internet six months to save itself, six months can feel generous right up until someone realizes how large the internet used to be.

Posted on

Digital estate planning for an independent website’s survival

An independent website can survive its creator. It can also disappear because nobody knows the registrar password.

That sounds trivial until you list what a small site actually depends on: a domain registration, DNS, hosting, billing, email, source files, databases, backups, TLS certificates, analytics, software updates, and often one person who understands how all of it fits together.

If that person dies, becomes incapacitated, or simply walks away, the content may still be perfectly intact while the machinery keeping it visible begins expiring one renewal at a time.

Survival requires authority, not just a backup

Sheldon Brown’s bicycle website is a useful counterexample to the usual disappearing-homepage story. Brown died in 2008, but the site did not freeze and vanish. Its own contact page says the site is actively maintained, updated, and expanded by his wife Harriet Fell and his friend John Allen. Even after Harris Cyclery closed in 2021, the site continued.

What survived was not merely a directory of HTML files. People survived who had the ability and willingness to operate the site.

That distinction matters legally as well as technically. The American Bar Association notes in its digital property guidance that an executor or personal representative does not automatically gain access to every online account. Access can depend on state law, platform terms, and whether the account owner gave specific consent.

A website can therefore be backed up beautifully and still be stranded behind accounts nobody is authorized to control.

A practical succession plan is boring on purpose

For an independent site, continuity planning should identify at least the domain registrar, DNS provider, hosting account, billing method, email accounts, code or content repositories, database backups, and any third-party services required for the site to function.

It should also answer operational questions. Who is allowed to renew the domain? Where are recovery codes stored? Who can move the site if the host closes? What content is meant to remain public? Which files are private? Are there licenses or copyrights that affect what a successor may publish?

Passwords themselves do not belong scattered through a will or printed beside the keyboard. A better arrangement uses a secure password manager, documented emergency access, and instructions telling the successor what systems exist and what authority they are supposed to exercise.

Preservation and continued operation are different goals

Not every site needs to remain live forever.

A successor may decide that the best outcome is a static read-only copy. Another site may need an active maintainer because links, software, or data continue changing. Some projects are better deposited with an archive or institution than operated indefinitely by a family member who never asked to become a system administrator.

The important part is making the choice before the infrastructure makes it for you.

Independent websites feel permanent because a domain name can remain familiar for decades. Underneath, many are surprisingly personal machines. One missed renewal, one inaccessible email account, or one dead credit card can remove years of work from the live Web.

Digital estate planning is what turns “somebody should keep this” into an arrangement that somebody actually can.

Posted on

Search snippets and cached quotations as traces of missing originals

A missing webpage rarely disappears without leaving fingerprints.

Its URL may survive in a bibliography. A sentence may have been quoted in a forum post. A search engine may still display words it indexed from an older version. A PDF may cite the title and author. Somebody may have copied one paragraph into an email before the original vanished.

None of those fragments is the lost page. Together, however, they can establish more than an empty 404 ever will.

Search engines can preserve a temporary textual trace

Search results are built from material a crawler previously encountered, so a snippet can sometimes reflect content that is no longer visible on the current page. Google has explicitly noted that pages may change between crawls and that quoted-search snippets are formed around matching text its systems found in the document. Its explanation of quoted searches and snippets is a useful reminder that the text shown in a result is evidence of an indexed state, not necessarily the page as it exists now.

That makes exact-phrase searching surprisingly useful when reconstructing lost material. A distinctive sentence copied elsewhere may reveal a title, author, old URL, or secondary source that points toward an archived version.

Scholarly publishing shows why these traces matter. A large study of web references found extensive reference rot: cited web resources disappear, move, or change after publication. The authors of “Scholarly Context Not Found” also found that archive holdings often lacked a capture close enough to publication time to guarantee it represented what the author originally cited.

In those cases, the quotation inside the paper may outlive the page it referenced.

Fragments can prove less than people think

A surviving quotation can establish that a particular author reproduced those words and attributed them to a source. It may help identify the missing document or recover a phrase that appeared there.

It does not automatically prove the quotation was accurate, complete, or taken from the version of the page you care about.

Search snippets are even more limited. They are generated for retrieval, not historical preservation. A snippet can linger as a clue after the underlying page changes or disappears, but it is not a stable archived copy. Snippets can omit surrounding paragraphs, combine nearby text, or reflect a crawl date nobody recorded.

Reconstruct outward from the fragment

The strongest recovery process keeps every clue and searches for agreement between them: exact quotations, page titles, old URLs, author names, publication dates, citations, screenshots, mirrors, and web-archive captures. If several independent traces point to the same wording, confidence rises.

But the result should still be described honestly. “A 2012 paper quotes the missing page as saying X” is stronger evidence than nothing. It is not the same claim as “we recovered the complete 2012 page.”

Digital archaeology often works this way. The original artifact is gone, but enough secondary debris survives to outline its shape.

A sentence in a search result can be a clue. It should never be mistaken for the whole crime scene.

Posted on

CD-ROM website snapshots as evidence outside web archives

Not every surviving old website survived on the Web.

During the 1990s and early 2000s, companies, magazines, conference organizers, software vendors, and computer manufacturers routinely shipped CD-ROMs containing promotional material, documentation, demos, and sometimes complete copies of websites. The original purpose was convenience: customers could browse material without downloading everything over a slow connection.

Decades later, those discs can become preservation evidence.

A disc freezes a directory tree

A conventional web crawler visits URLs over the network. It can preserve only what it reaches and what the server returns at that moment. A CD containing a site works differently. The publisher may have copied the HTML, images, downloads, and directory structure directly from the material used to build the site.

That can preserve files even when no public crawler ever visited them.

Silicon Graphics provides a clean supporting example. A December 1994 CD snapshot of its Silicon Surf website survives independently of the original server. The IRIX Network archive identifies it as the earliest known surviving SGI website snapshot; Archive95’s source record documents the CD image’s date, scope, and known defects, while the Dan Rich mirror exposes the files themselves.

The important point is the preservation path, not another history of SGI: a physical distribution disc became an independent witness to a website.

Offline copies can contain things crawlers miss

A disc may preserve images, PDFs, executables, audio, or nested pages that were difficult to discover from the live site’s navigation. It may also retain original filenames and relative links, making the relationship between files easier to reconstruct.

That does not mean the disc is a perfect copy of the public website.

Publishers often modified sites for offline use. Links to CGI scripts, databases, search engines, counters, forms, or external servers may not work from a disc. A CD may contain only a product section rather than the whole domain. Material could have been added to the live site after the disc was mastered. The date printed on the CD packaging may describe publication or manufacturing rather than the exact moment the files were copied.

Treat the disc as a dated witness, not the entire truth

A useful reconstruction records the disc label, volume name, filesystem timestamps, packaging, readme files, and any evidence identifying the edition or distribution date. It then compares the files with surviving live captures, mirrors, printed documentation, or later versions.

That comparison helps distinguish three different claims: this file existed on the disc, this file was probably part of the website, and this was exactly what a public visitor saw on a particular day. Those are not automatically the same statement.

Web history is often reconstructed from whatever escaped destruction. Sometimes that is a Wayback capture. Sometimes it is a forgotten mirror. And sometimes the best surviving evidence is a scratched corporate CD that somebody shoved into a drawer in 1995 and never threw away.

Posted on

Email attachments as accidental archives of vanished web documents

Some of the Web’s most useful backup systems were never intended to be backup systems at all.

A PDF disappears from a small organization’s website. The download page is gone. The domain later expires. Years afterward, somebody searching an old mailbox finds the same PDF attached to a message sent when the document was new.

The website failed. Email quietly kept a copy.

Attachments escape the fate of the original server

When a document is emailed as an attachment, another copy leaves the server that originally published it. That copy may end up in several inboxes, desktop mail archives, corporate retention systems, exported MBOX files, PST files, or personal backups.

That makes email an accidental distributed archive.

Professional archivists treat attachments as part of the record for exactly this reason. The Library of Congress recommends that email attachments remain in their original format. NARA’s transfer instructions for email with attachments likewise call for preserving the message structure and labels such as date, sender, recipients, subject, body, attachment filename, and related transmission information.

The attachment preserves the file. The message around it may preserve something just as important: why the file was sent.

Correspondence can restore provenance

Suppose an old attachment is named report.pdf. By itself, that filename says almost nothing.

The message may say, “Attached is the final version we posted this morning,” and carry a date, sender, organization, and thread discussing corrections. That surrounding correspondence can help establish when the document circulated and how people understood it at the time.

This is valuable when the original web page is gone because the email may provide provenance that a loose recovered file lacks.

But email evidence has limits.

An attachment may be a draft rather than the public version. The sender may have renamed it. Someone may have forwarded a modified copy. The email date proves when that message was sent, not necessarily when the underlying document was created. If several versions survive, checksums and internal document metadata can help determine whether they are actually identical.

Recovery does not automatically mean republication

There is also a privacy problem.

An attachment that once appeared publicly can often be republished without much mystery. An attachment found in private correspondence is different. The mailbox may contain personal information, unpublished drafts, addresses, internal comments, legal material, or information that was never intended for public distribution.

Preservation and publication are separate decisions.

Old email is therefore an unusually powerful archaeological layer. It can contain files that vanished from websites years ago and messages that explain what those files were. The Smithsonian’s email-preservation work treats messages and associated attachments as a collection-level preservation problem rather than disposable mail clutter. The best recovery work keeps the chain of evidence intact: preserve the original message, preserve the attachment separately, record hashes, document where it came from, and distinguish what is known from what is inferred.

Sometimes the last surviving copy of a public web document is sitting in a twenty-year-old inbox nobody thought to search.

Posted on

Bit rot and corrupted files inside personal digital collections

Digital loss does not always arrive with an empty folder.

Sometimes the filename is still there, the drive still mounts, and the backup directory still looks reassuringly full. The damage appears only when an old photograph opens with a stripe through it, a ZIP file refuses to extract, or a video stops halfway through because some of the stored data no longer matches what was originally written.

That family of problems is often described loosely as bit rot, data corruption, or loss of fixity. The cause may be media failure, transfer error, filesystem damage, software fault, or some other unintended change; not every corrupt file literally represents a storage bit spontaneously flipping. What matters to preservation is that the file still exists while its contents are no longer what they were supposed to be.

A copy is not proof that the copy is good

Modern storage devices contain error detection and correction, so random corruption is not something every file experiences every week. But silent corruption is a documented storage problem rather than a campfire story. A large USENIX study of production storage systems analyzed 1.53 million disk drives over 41 months and found more than 400,000 checksum-mismatch events. Personal collections are much smaller, but the underlying lesson is the same: stored data can change without an obvious deletion event.

The preservation world treats this as a normal engineering problem. The Library of Congress says fixity information should be established as early as possible and then monitored through routine integrity reviews. The National Digital Stewardship Alliance likewise makes file integrity one of the core areas in its Levels of Digital Preservation.

A checksum is the basic tool. Software runs the file through a hashing algorithm and records a value derived from its contents. Months or years later, the checksum can be calculated again. If the value changed unexpectedly, the archive knows the file changed too.

That is important because corruption is often silent. Without a previously recorded checksum, an old file may look suspicious but there is no reliable baseline proving whether it changed.

Checksums detect damage; copies repair it

A checksum cannot reconstruct a damaged photograph. It only tells you that the photograph is no longer identical to the copy you intended to preserve.

Recovery requires another known-good copy: perhaps on a second drive, an offline disk, another family member’s computer, or remote storage. This is why good preservation combines integrity checking with redundancy. Several identical copies stored in genuinely independent places give you something to restore when one copy fails.

There are limits. If three backups were all made after corruption occurred, they may faithfully preserve the same broken file. If no checksum was recorded before the damage, it may be impossible to determine which surviving version is original. And a perfectly intact file can still become unusable if the software or format needed to interpret it disappears.

Personal digital collections therefore need more than storage space. They need occasional verification.

The disturbing thing about bit rot is not that bits decay like old newspaper. It is that digital collections encourage us to believe existence and integrity are the same thing. They are not. A file can sit exactly where you left it for fifteen years and still fail the moment you finally need it.

Posted on

Backup copies that fail because nobody tested restoration

A backup is a promise about the future. Until somebody restores it, nobody has proved the promise is true.

That sounds like administrator folklore until a real failure turns it into accounting. Files may be copied to the wrong place. A scheduled job may have stopped running months ago. Encryption keys can be missing. Database dumps can be corrupt. The backup may contain data but not the configuration needed to make the application understand it. A green checkmark can simply mean that a script finished doing the wrong thing.

GitLab provided an unusually public example in 2017. During a database incident, an engineer accidentally removed production data while troubleshooting replication. The obvious next step was to restore from backup. That was when GitLab discovered that several recovery paths it expected to rely on were not usable — and its own postmortem later identified a blunt root cause: nobody owned the backup procedure, so nobody was responsible for testing it regularly.

Its detailed postmortem of the January 31 database outage is still one of the clearest explanations of why backup systems need to be tested as recovery systems.

Copies existed; recovery still failed

GitLab’s environment had multiple mechanisms: database dumps, disk snapshots, replication, and staging data. On paper, that sounds comfortably redundant.

In practice, the pg_dump backups that were supposed to be uploaded to Amazon S3 had been failing because the backup job used an incompatible PostgreSQL client version. The S3 bucket was empty. Error notifications were also being rejected because of an email configuration problem, so the team did not know the dumps were missing.

Other recovery methods had different limitations. Replication was already broken during the incident. Some snapshots were not suitable for the needed restore path. GitLab ultimately recovered from an older copy and estimated that several hours of database changes could not be restored, affecting projects, comments, user accounts, issues, snippets, and other records. Git repositories and wikis were not lost because they were stored separately.

The important lesson is not that GitLab lacked backups. It is that the organization discovered the real state of those backups during the emergency.

Restoration is the actual test

A useful backup process asks questions that copying alone cannot answer. Can a clean system read the backup? How long does restoration take? Are application versions compatible? Are encryption keys and credentials available? Are attachments stored somewhere different from the database? Does the restored application contain the records operators think it contains?

Checksums can detect corruption. Automated monitoring can detect missed jobs. Those are useful, but neither proves the complete service can be rebuilt.

Periodic restoration tests do. They can restore into an isolated environment, verify record counts or representative files, and document the exact steps required. GitLab’s current restore documentation includes post-restore checks, while NIST’s 2026 OT Backup Quick Start Guide explicitly treats backup testing and recovery exercises as part of effective backup management.

For personal archives, the principle is the same at smaller scale. Open a sample of the photos. Extract the ZIP. Mount the disk image. Restore the database somewhere harmless. Verify that the supposedly preserved material is actually readable.

The worst time to learn how your backup works is immediately after the original disappears.