Category: Algorithmic Reality: The Internet You Are Allowed to See

A study of the systems that decide what becomes visible online and what quietly disappears from view.

This category covers search ranking, recommendation engines, personalized feeds, trending systems, content moderation, discovery algorithms, engagement optimization, and the growing dependence on a small number of platforms to decide what people encounter.

The focus is on visibility as power. The modern internet contains more information than any person could ever inspect directly, which means filters are unavoidable. The important question is who controls those filters, what they optimize for, and what gets buried as a result.

The internet may be enormous.

Your version of it is considerably smaller.

Posted on by

Distinguishing an undiscoverable web from a nonexistent web

A page that you cannot find is not necessarily a page that no longer exists.

That distinction is the difference between a dead web and an undiscoverable one.

Search engines expose only the portion of the web they have discovered, crawled, indexed, and decided to surface for a query. Independent systems can expose different material because they crawl and rank differently. Brave Search, for example, says it serves results from its own independent index, while Marginalia Search deliberately emphasizes non-commercial and independent sites with its own crawler and index software.

If the same query produces different obscure sites across different indexes, the missing material was not necessarily gone.

One map simply did not show it.

Test absence with more than one route

Suppose an old hobby topic appears to have vanished from the modern web.

A reasonable investigation might try:

  • ordinary search with several phrasings;
  • exact quoted phrases;
  • site-specific searches on likely hosts;
  • another search engine with a separate index;
  • curated directories or webrings;
  • old blogrolls and bookmark collections;
  • direct links from surviving related pages;
  • web archives if historical material is suspected.

These methods test different failure points.

An archive result may show that the page is genuinely dead today. A direct live link may show that the page exists but ranks poorly. A separate search index may reveal that one crawler found it while another did not.

Discovery failure can imitate disappearance

This matters because people experience the web primarily through interfaces.

If search stops surfacing independent forums, personal pages, niche blogs, and specialist archives, those sites can disappear from ordinary experience long before they disappear from servers.

From the user’s perspective, that feels like extinction.

Technically, it may be obscurity.

The distinction is not comforting in every case. A site nobody can discover may have almost the same practical reach as a deleted one.

But the remedies are different.

Deletion requires preservation or reconstruction.

Poor discovery requires better indexing, links, directories, feeds, cross-site recommendations, or deliberate exploration.

Dead Internet Theory needs this distinction

Claims that “there is nothing left out there” are difficult to evaluate if the only measuring instrument is the same ranked search interface being criticized.

A better test asks two separate questions:

Does the material still exist?

Can ordinary users still find it?

Those questions overlap, but they are not interchangeable.

The web can lose pages.

It can also lose roads.

Posted on by

Reproducing search experiments across accounts, locations, and time

A screenshot of one search result page proves exactly one thing:

that result page appeared under those conditions at that moment.

It does not automatically prove that everybody saw it, that the same account will see it tomorrow, or that the ranking represents a stable policy.

Google’s own current documentation explains why. Search results can differ because of time, location, language, device type, recent searches, and personalization. Even with personalized recommendations disabled, location and other query context can still affect results. See Why your Google Search results differ from others and Google’s personalization documentation.

That makes reproducibility part of the experiment.

Record the conditions before interpreting the result

At minimum, a search test should preserve:

  • the exact query;
  • date and local time;
  • search engine;
  • signed-in or signed-out state;
  • whether personalization is enabled;
  • approximate search location;
  • language and region settings;
  • device or interface type;
  • the result positions or a saved capture.

If several people are comparing results, they should agree on which of those conditions are supposed to stay constant.

Otherwise they may be measuring different systems without realizing it.

A user in Missouri and a user in London searching the same restaurant phrase should not be expected to get the same answer. That is useful localization, not evidence of manipulation.

Change one variable at a time

Suppose you want to know whether account history affects a query.

Run the search signed in. Then run it without personalization under otherwise similar conditions. If possible, repeat the test using another account while keeping location, language, time, and device stable.

If the results differ, you have evidence that some changed condition mattered.

You still may not know which internal ranking signal caused the difference.

The same method works for location: hold the account and query constant, deliberately change geographic context, then compare.

Time is harder because the web itself changes. A result captured Monday and another captured Friday may differ because ranking changed, or because new pages appeared, old pages disappeared, breaking news occurred, or a page was updated.

Repetition separates patterns from anecdotes

One strange result can be interesting.

A repeated result under documented conditions is evidence.

For Dead Internet Theory, that standard is important because algorithmic visibility is easy to demonstrate badly. A surprising screenshot invites a large story about what the platform “allows” people to see.

Sometimes the large story is real.

Sometimes the screenshot was local, personalized, temporary, or already gone.

The cure is not blind trust in search engines.

It is better experiments.

Posted on by

Measuring how many distinct domains a person encounters in a month

If the internet feels like the same fifteen websites over and over, that impression can be measured.

One simple experiment is to count how many distinct domains a person actually visits during a month.

Browsers already keep enough local history to make that possible. Firefox, for example, stores browsing history as part of a user’s profile data and can preserve it through backup and sync features. See Mozilla’s current documentation on Firefox browsing data and backups.

The interesting part is not extracting the data.

It is deciding what to count.

Define the unit before counting it

Does news.example.com count separately from example.com?

What about docs.example.com and shop.example.com?

A reasonable approach is to reduce URLs to their registrable domain, so multiple pages and subdomains belonging to the same site count once. The exact rule should be documented because different definitions can produce very different totals.

Then define a visit.

A page that opened automatically in the background should probably not count the same way as a page the user actively selected. Redirects, authentication domains, CDNs, payment processors, embedded trackers, and browser-internal pages can inflate the number without representing meaningful destinations.

A privacy-conscious analysis can stay entirely local. The browser history never needs to leave the machine. A script can extract only the hostname and date, discard full URLs and query strings, reduce hosts to domains, calculate totals, then delete the working copy.

One number is useful but incomplete

Suppose a person encountered 312 distinct domains in one month.

That sounds diverse until you learn that 80 percent of their page visits happened on five of them.

Or perhaps they visited only 90 domains but spent substantial time reading dozens of independent sites.

Useful companion measures include:

  • total distinct domains;
  • visits per domain;
  • percentage of activity concentrated in the top 5 or top 10;
  • number of domains visited only once;
  • how many were reached directly, from search, or through social links;
  • how many were independent sites versus major platforms.

Even then, the numbers do not measure quality.

Ten excellent specialist sites may be better than 1,000 random junk domains.

The value of the exercise is narrower.

Dead Internet Theory often describes a subjective sensation: the web seems smaller, more repetitive, more centralized.

Browsing history can turn part of that sensation into a testable question.

Maybe the web became smaller.

Maybe your route through it did.

Posted on by

Random-page tools and the recovery of accidental discovery

Search is powerful only after you know enough to ask a question.

Random discovery solves a different problem:

What might be interesting that I would never have searched for?

The web still contains tools built around that idea. The IndieWeb’s current discovery page lists projects that send visitors to randomly selected personal sites or blogs, including collections with a dedicated random button.

Wikipedia’s long-running Random article function uses the same basic idea inside one enormous corpus.

The mechanism is almost aggressively unsophisticated.

Pick a candidate.

Go there.

Randomness removes relevance on purpose

A recommendation engine tries to reduce surprise.

It uses history, similarity, popularity, or context to increase the odds that the next item matches an existing preference.

A random-page tool does the opposite. It gives up some relevance in exchange for a chance encounter.

That trade can look wasteful. Many random destinations will be boring, irrelevant, broken, or incomprehensible.

But the failures are part of the method.

If every result already fits the reader’s established interests, the system is not producing much serendipity. It is refining a profile.

Random selection can expose a tiny personal homepage, a niche hobby, a programming language you have never used, or somebody’s obsessive catalog of municipal drain covers without requiring those pages to win a popularity contest first.

The candidate pool determines the surprise

“Random” needs an asterisk.

A random Wikipedia article comes from Wikipedia, not the whole web.

A random IndieWeb blog comes from whatever list the tool knows about.

A random button on a curated directory samples the curator’s collection.

If the candidate pool contains 2,000 personal sites, the tool may be perfectly random while still excluding 99.999 percent of the web.

That means a serious evaluation has two parts: how the destination is selected, and how the candidate set was built.

The first can be mathematically random while the second is heavily curated.

That is not necessarily a flaw. A deliberately small corpus can produce better exploration than throwing users into truly arbitrary URLs filled with parking pages, tracking infrastructure, login screens, and junk.

For Dead Internet Theory, random-page tools provide a useful experiment.

If the ranked web feels repetitive, temporarily remove ranking from the equation.

You may discover that the internet is not empty.

Your usual route through it may simply be extremely narrow.

Posted on by

Independent search indexes and the tradeoffs of smaller coverage

Two search engines can look independent while one of them is quietly renting most of its map from somebody else.

That distinction matters because a genuine independent index has its own crawler, its own collection of pages, and its own ranking decisions.

Brave Search is a clear current example. Its documentation says results are served from Brave’s own independent index rather than simply repackaging Google or Bing. See Brave Search independence.

Marginalia Search is another, much smaller example. It describes itself as using custom crawler and index software while deliberately prioritizing non-commercial and independent parts of the web. See Marginalia Search.

These two systems are both independent.

They are not trying to be the same thing.

A smaller index can be worse and better at the same time

Coverage is an obvious tradeoff.

A giant index has more chances to contain the obscure PDF, local business, old forum post, breaking-news page, or specialist document you need. A smaller crawler may never have discovered it.

But more pages do not automatically mean a more useful result set.

A smaller search engine can intentionally crawl different neighborhoods, weight commercial material differently, expose personal websites more aggressively, or simply make different ranking mistakes.

Marginalia’s value, for example, is not that it has secretly indexed more of the web than the largest search engines. Its value is that it is tuned toward material those systems may not place prominently.

Compare with a query set, not one lucky search

The fair way to compare independent indexes is boring but useful.

Choose a defined set of queries: current news, obscure technical terms, local information, old-web material, product searches, academic topics, and known-answer tests. Record whether each engine finds the target material and what unrelated material it surfaces.

One spectacular result proves very little.

So does one failure.

Search engines optimize differently, crawl different pages, refresh at different speeds, and maintain different spam defenses.

The larger Algorithmic Reality lesson is that a search engine is not merely a window onto the web.

It is a window onto its index of the web.

Independent indexes are valuable because they create genuinely different windows.

They are also reminders that every window has edges.

Posted on by

Social bookmarking as an archive of human selection

A bookmark says something a search result does not:

somebody decided this was worth saving.

Social bookmarking turns that private act into a public discovery system. People save URLs, add tags, group links into collections, and sometimes expose those collections to anyone who wants to browse them.

Modern services still support this pattern. Raindrop.io, for example, allows collections to be made public so visitors can browse saved links without signing in. Its documentation describes public collections as shareable pages whose bookmarks can be organized with tags and nested collections. See Raindrop.io public collections.

That is not the same thing as a search engine ranking pages for a query.

It is an archive of human attention.

Tags preserve how people understood a link

Suppose one person saves an obscure 1998 electronics page under repair, crt, and television.

Another saves the same page under retrocomputing and high-voltage.

The page itself has not changed, but the surrounding tags reveal how different people placed it in their mental map of the web.

Search engines infer relationships from text, links, behavior, and many other signals. Social bookmarking can expose the classification directly because the user typed it.

That does not make the classification correct.

Tags can be sloppy. Collections can be idiosyncratic. A popular user can save mediocre material. Many excellent pages may never be bookmarked at all.

But the mechanism is visible.

A collection can outlive its original social moment

Social bookmarking becomes especially interesting when the social layer fades but the saved links remain.

A collection from years ago may still reveal which tutorials, tools, niche sites, papers, and odd personal pages people considered useful at the time. Even if comments, follower relationships, or popularity counts disappear, the surviving URL-and-tag structure can preserve a faint map of human selection.

The map also decays.

Links rot. Domains change hands. A bookmark can point to a page that now means something completely different. Public collections can be made private or deleted. Tags lose meaning outside the community that used them.

So a bookmark archive is not a perfect historical record.

It is evidence of attention.

For Dead Internet Theory, that distinction matters. Search rankings show what a machine currently decides to surface. Social bookmarks can show what actual people once chose to keep.

Those two maps overlap.

They are not the same map.

Posted on by

RSS subscriptions and discovery outside a ranked home feed

RSS does not decide what you should care about next.

It delivers updates from sources you already chose.

That sounds almost too simple now. A site publishes a feed. A reader subscribes. New entries appear in the reader. Depending on the software, they may be shown newest-first, grouped by source, filtered by rules, or otherwise organized by the user rather than by a platform’s central home-feed ranking.

The format itself remains actively maintained. The RSS Advisory Board continues to publish and support the RSS specifications and related guidance.

Subscription and discovery are different jobs

RSS is excellent at continuity.

Once you find a useful independent site, podcast, newsletter archive, or project blog with a feed, RSS can keep that source in your daily information environment without requiring the source to win a fresh ranking contest every time it publishes.

That changes the power relationship considerably.

A followed site cannot suddenly disappear because its latest post underperformed on engagement. The feed reader does not need to infer whether you are still interested. You subscribed. The update arrives.

But RSS does not solve the first half of the problem.

You still have to discover the source.

That discovery may come from search, a blogroll, a webring, a friend’s recommendation, a directory, a forum post, or pure accident. RSS becomes useful after that moment.

Reader control has costs too

A feed list can become overwhelming.

Subscribe to 300 prolific sites and a chronological reader can produce its own version of information overload. High-volume publishers dominate simply by publishing more. Quiet but valuable sources can still get buried under unread counts.

Different readers address this with folders, filters, starring, priority views, or manual cleanup. The important difference is that those controls are generally explicit and local to the reader.

The user can see the subscription list.

They can remove a source.

They can decide whether oldest or newest comes first.

That does not make RSS neutral. It makes the selection rule easier to understand.

A feed can preserve a relationship with a site

This matters for independent publishing because ranking systems repeatedly force small sites to re-earn visibility.

RSS allows a one-time discovery event to become an ongoing direct connection.

For Dead Internet Theory, that is useful because a site can appear to vanish from someone’s internet without disappearing from the web at all. It may simply stop surfacing in ranked feeds and ordinary search.

RSS offers a blunt alternative:

Find the site once.

Then stop asking an algorithm for permission to see it again.

Posted on by

Blogrolls as personal recommendations with visible authorship

A blogroll is an algorithm with a human name attached to it.

It is usually just a list of websites, writers, feeds, or projects that one publisher reads or recommends. The IndieWeb page on blogrolls describes them plainly as lists of sites a person reads, follows, or recommends, and documents modern examples that are still being maintained.

That simplicity gives blogrolls a property modern recommendation systems often hide: visible authorship of the recommendation itself.

If I click a link because it appears on someone’s blogroll, I know who made that choice.

The bias is obvious because the person is obvious

A blogroll is not neutral.

It reflects one person’s interests, friendships, habits, professional circle, blind spots, and changing attention. Someone who reads mostly programming blogs will produce a very different map of the web from someone who follows experimental music or antique radios.

But the source of that bias is inspectable.

A reader can ask: Do I trust this person’s taste? Why might they recommend these sites? Are all the links from the same social circle? Has the list been updated recently?

That is a different relationship from a recommendation labeled merely For You.

A platform can recommend an excellent site without exposing why it was selected. A blogroll exposes at least one important piece of context immediately: this particular person thought the site was worth linking.

Maintenance is the weak point

The cost is upkeep.

People stop publishing. Domains die. A writer’s interests change. Lists become fossils.

The IndieWeb documentation itself includes both active and dead blogroll examples, which is useful evidence of the mechanism’s limitation. Human curation ages unless humans keep curating.

There is also no guarantee of breadth. One person’s list can be wonderfully specific while missing entire communities.

Yet that narrowness can be useful.

A strong blogroll is not trying to summarize the web. It is a trail left by a person through the parts of the web they actually value.

Some current blogrolls even publish OPML files so a reader can import many recommended feeds directly into an RSS reader. That turns the list from a static sidebar into a portable discovery path.

Dead Internet Theory often asks why the web feels less personal.

Part of the answer may be that discovery itself became less personal in the old sense of the word.

A blogroll does not say people like you also liked this.

It says something much more concrete:

I liked this. Maybe you will too.

Posted on by

Webrings as reciprocal discovery between independent sites

A webring is a tiny discovery engine built out of websites agreeing to point at one another.

The classic structure is simple: each member site includes links such as previous, next, or random. Click one and you move to another independently operated site in the same ring. Keep going and eventually you circle through the group.

The IndieWeb description of webrings still documents active examples and defines the basic idea as a collection of sites linked in a circular structure. Modern rings may preserve strict next-and-previous order, send visitors to a random member, or publish a directory of members alongside the navigation controls.

That sounds almost comically primitive beside a modern recommendation system.

That is also the point.

The recommendation is built into membership

Joining a webring does more than create one inbound link.

A member is effectively saying: people who reached my site may also enjoy these other independent sites.

No central service needs to infer the reader’s interests from watch time, click history, location, or a profile assembled over several years. The shared subject or community of the ring is the recommendation signal.

This makes the selector unusually legible. If a ring is about retrocomputing, personal journals, tiny websites, or handmade art, the reader already knows why those sites were grouped.

The tradeoff is obvious too: somebody had to decide who belongs.

A badly curated ring can become irrelevant. A narrowly defined one can become insular. A ring with hundreds of inactive members can turn the next button into a tour of dead links.

Broken members break the route

Webrings depend on maintenance in a way giant search engines do not.

Sites vanish. Domains expire. Owners redesign pages and remove the ring links. A coordinator stops accepting new members. A once-lively chain can slowly become a collection of 404s.

That fragility is not a hidden failure mode. It is the direct consequence of using reciprocal human participation as infrastructure.

But the same structure creates a kind of discovery that ranked feeds often struggle to reproduce: sideways movement between peers.

A reader does not have to know the name of the next site. They do not have to formulate a search query. They do not need the next destination to be globally popular.

They only need to click next.

For Dead Internet Theory, that matters because an internet can feel empty when discovery collapses toward a handful of large surfaces.

Webrings offer a very small counterexample.

The independent sites may already be there.

Sometimes the missing technology is simply a road between them.

Posted on by

Human-edited directories as an alternative discovery system

Before search engines became the default map of the web, one common alternative was much simpler:

People made lists.

A human-edited directory organizes websites into categories chosen by editors rather than continuously ranking billions of pages for every query. That sounds primitive compared with modern search, but it changes the discovery problem in interesting ways.

Curlie is a surviving example. It describes itself as a human-edited directory run by volunteer editors. Its editorial guidelines are public, and its editors are instructed to select, evaluate, describe, and organize sites according to published criteria. Curlie’s editor information explains that editors apply to manage categories and review suggested sites.

The selection process is therefore opinionated, but not invisible.

Curation makes the selector legible

A search engine can rank thousands of candidates through signals most users never see.

A directory instead says, in effect: these sites were selected for this category.

The editor may still make mistakes. The category structure may be awkward. A useful site may be omitted. But the basic selection model is understandable, and Curlie’s public guidelines even describe what kinds of sites are generally included or excluded.

That transparency has value.

A reader exploring a directory can browse sideways through neighboring categories instead of only asking for a precise keyword. That makes directories useful for accidental discovery and for subjects where the user does not yet know the right search terms.

Humans have a crawl budget too

The weakness is scale.

Volunteer editors cannot inspect the whole web. Categories can become stale. Suggested sites can wait for review. Editors can become inactive. New subjects can grow faster than the directory structure adapts.

Curlie itself acknowledges backlogs and depends on volunteers to maintain categories. Its model trades automated breadth for human judgment.

That means a missing site proves very little.

It may have been rejected under the guidelines. Nobody may have suggested it. An editor may not have reached it yet. The relevant category may have little active maintenance.

Human curation therefore does not solve Algorithmic Reality by producing a perfectly neutral internet.

It produces a different kind of selection.

The important difference is that the selection rule is easier to inspect: named categories, public editorial policies, human review, and visible organizational choices.

Modern discovery is often framed as a choice between good algorithms and bad algorithms.

The older web reminds us that there is another option.

Sometimes the map can simply admit that somebody drew it.