Save a web page and you can usually replay it: the HTML, the images, the styles come back as files. Save a page carrying a video and you may end up with a player and nothing to play. Audio and video on the modern web are rarely a single file parked at one URL. They are streams, a timed conversation between a player and a server, and a crawler that only fetches pages is not part of that conversation.
Why the stream slips past a page crawl
The difference is delivery. Progressive download puts a whole media file on a server and lets the browser pull it the way it pulls an image: one request, one file, happily replayed. Streaming is the opposite. In the Flash era, video travelled over RTMP, a protocol on its own port. Today’s adaptive formats, Apple’s HLS and MPEG-DASH, run over HTTP but still offer no single file: the player asks for a playlist or manifest, which lists dozens of short segments, which the player fetches in sequence from CDN addresses that can expire in minutes. No step ever asks for “the video.” A crawler that does catch the playlist catches a short text file; the segments it names are separate requests, often to addresses that have already rotated.
A page crawl never gets there. Heritrix, the crawler behind the British Library’s web archiving, could capture data over HTTP and FTP and nothing else, and the consequence surfaced in 2009 when the Library took on One & Other, Antony Gormley’s art project on the fourth plinth in Trafalgar Square. For a hundred days, 2,400 participants each took the plinth for an hour while cameras recorded them: 2,400 hours of .flv video, about a terabyte, streamed over RTMP. Test crawls of oneandother.co.uk returned only static HTML pages without the videos. The pages were archived. The art was not.
What a complete capture has to keep
The British Library’s remedy shows what media capture costs. Instead of crawling, staff ran a browser together with a streaming recorder, Jaksta, that caught the packets flowing to the embedded Flash player over RTMP. Four virtual machines scheduled at 90-minute intervals did the work. For replay they stood up Red5, a Java media server, and Flowplayer, then patched the Wayback banner to swap the original players for archived ones, necessary because the sponsor’s licence allowed streaming only, with copy protection equivalent to the live site.
The pattern has not aged away. On HTTP-based platforms, media capture still needs tooling beyond a page fetch. Archive-It recommends browser-driven Brozzler crawls and checks whether the media files themselves were collected; its current streaming-media guidance also notes that some services require yt-dlp support and that some captured media still will not replay correctly in Wayback. A playlist without its segments is only a list of missing pieces.
The limits bear repeating. The Library’s pipeline sat outside its normal workflow: the videos were served from a separate machine, not packaged with the rest of the archive. Some videos never replayed to full length, and re-capturing did not help; the sponsor later supplied the original masters on disk, which carried the same errors. The archive could not repair a source that was broken at birth. And a stream is only captured while it is running. Once it ends unrecorded, there is nothing to re-crawl.
A complete capture therefore needs more than the page: the playlist or manifest, every segment or file, the codecs, any keys, and a record of what was shown and when. Otherwise the archive keeps a play button that points nowhere.