Posted on

Reproducing search experiments across accounts, locations, and time

A screenshot of one search result page proves exactly one thing:

that result page appeared under those conditions at that moment.

It does not automatically prove that everybody saw it, that the same account will see it tomorrow, or that the ranking represents a stable policy.

Google’s own current documentation explains why. Search results can differ because of time, location, language, device type, recent searches, and personalization. Even with personalized recommendations disabled, location and other query context can still affect results. See Why your Google Search results differ from others and Google’s personalization documentation.

That makes reproducibility part of the experiment.

Record the conditions before interpreting the result

At minimum, a search test should preserve:

  • the exact query;
  • date and local time;
  • search engine;
  • signed-in or signed-out state;
  • whether personalization is enabled;
  • approximate search location;
  • language and region settings;
  • device or interface type;
  • the result positions or a saved capture.

If several people are comparing results, they should agree on which of those conditions are supposed to stay constant.

Otherwise they may be measuring different systems without realizing it.

A user in Missouri and a user in London searching the same restaurant phrase should not be expected to get the same answer. That is useful localization, not evidence of manipulation.

Change one variable at a time

Suppose you want to know whether account history affects a query.

Run the search signed in. Then run it without personalization under otherwise similar conditions. If possible, repeat the test using another account while keeping location, language, time, and device stable.

If the results differ, you have evidence that some changed condition mattered.

You still may not know which internal ranking signal caused the difference.

The same method works for location: hold the account and query constant, deliberately change geographic context, then compare.

Time is harder because the web itself changes. A result captured Monday and another captured Friday may differ because ranking changed, or because new pages appeared, old pages disappeared, breaking news occurred, or a page was updated.

Repetition separates patterns from anecdotes

One strange result can be interesting.

A repeated result under documented conditions is evidence.

For Dead Internet Theory, that standard is important because algorithmic visibility is easy to demonstrate badly. A surprising screenshot invites a large story about what the platform “allows” people to see.

Sometimes the large story is real.

Sometimes the screenshot was local, personalized, temporary, or already gone.

The cure is not blind trust in search engines.

It is better experiments.

Posted on

Package removal and the reproducibility of old software projects

An old software project can survive in GitHub, compile nowhere, and still look perfectly preserved to anyone browsing the source.

The missing piece is often a dependency.

Modern projects rarely contain every component they need. Instead, they name packages hosted elsewhere and ask a package manager to download the exact versions during installation or build time. If one of those versions disappears, the project may become unreproducible even though its own repository is untouched.

The most famous demonstration arrived in March 2016, when a developer unpublished a collection of packages from npm, including the tiny left-pad module. npm’s own postmortem, “kik, left-pad, and npm”, acknowledged that unrestricted unpublishing had allowed one removal to disrupt dependent software and promised policy changes.

A lockfile can describe a missing thing

Lockfiles help reproducibility by recording precise dependency versions instead of accepting whatever happens to be current. That is useful, but a lockfile is not an archive.

If package-lock.json, yarn.lock, Cargo.lock, or another lockfile says a project needs version 1.2.3 of a package, the build is only reproducible if version 1.2.3 can still be obtained. The lockfile can tell you exactly which brick is missing. It cannot manufacture the brick.

The left-pad event made that distinction painfully obvious. A vast dependency graph had been built on the assumption that registry entries would remain fetchable. When one small package vanished, the failure propagated far beyond the original author.

npm responded by tightening removal rules. Its follow-up on changes to the unpublish policy made dependency impact part of the decision. The current npm unpublish policy continues that approach: older packages can be removed only under limited conditions, including having no dependents in the public registry.

Reproducibility requires preserving dependencies too

There are several ways to reduce the risk. Organizations can maintain registry mirrors or caches. Projects can vendor critical dependencies into their own source tree when licensing permits. Build systems can use artifact repositories that retain the exact packages used for releases. Preservation projects can archive both source repositories and the package ecosystems around them.

Even that is not perfect. Old dependencies may require obsolete compilers, operating systems, certificate chains, or package-manager behavior. A package file may survive while the service needed to resolve it does not.

That is why preserving software is harder than saving source code. The program may be a small island connected by hundreds of bridges to other projects. If enough of those bridges disappear, the island is still visible. You just cannot get there anymore.