A backup is a promise about the future. Until somebody restores it, nobody has proved the promise is true.
That sounds like administrator folklore until a real failure turns it into accounting. Files may be copied to the wrong place. A scheduled job may have stopped running months ago. Encryption keys can be missing. Database dumps can be corrupt. The backup may contain data but not the configuration needed to make the application understand it. A green checkmark can simply mean that a script finished doing the wrong thing.
GitLab provided an unusually public example in 2017. During a database incident, an engineer accidentally removed production data while troubleshooting replication. The obvious next step was to restore from backup. That was when GitLab discovered that several recovery paths it expected to rely on were not usable — and its own postmortem later identified a blunt root cause: nobody owned the backup procedure, so nobody was responsible for testing it regularly.
Its detailed postmortem of the January 31 database outage is still one of the clearest explanations of why backup systems need to be tested as recovery systems.
Copies existed; recovery still failed
GitLab’s environment had multiple mechanisms: database dumps, disk snapshots, replication, and staging data. On paper, that sounds comfortably redundant.
In practice, the pg_dump backups that were supposed to be uploaded to Amazon S3 had been failing because the backup job used an incompatible PostgreSQL client version. The S3 bucket was empty. Error notifications were also being rejected because of an email configuration problem, so the team did not know the dumps were missing.
Other recovery methods had different limitations. Replication was already broken during the incident. Some snapshots were not suitable for the needed restore path. GitLab ultimately recovered from an older copy and estimated that several hours of database changes could not be restored, affecting projects, comments, user accounts, issues, snippets, and other records. Git repositories and wikis were not lost because they were stored separately.
The important lesson is not that GitLab lacked backups. It is that the organization discovered the real state of those backups during the emergency.
Restoration is the actual test
A useful backup process asks questions that copying alone cannot answer. Can a clean system read the backup? How long does restoration take? Are application versions compatible? Are encryption keys and credentials available? Are attachments stored somewhere different from the database? Does the restored application contain the records operators think it contains?
Checksums can detect corruption. Automated monitoring can detect missed jobs. Those are useful, but neither proves the complete service can be rebuilt.
Periodic restoration tests do. They can restore into an isolated environment, verify record counts or representative files, and document the exact steps required. GitLab’s current restore documentation includes post-restore checks, while NIST’s 2026 OT Backup Quick Start Guide explicitly treats backup testing and recovery exercises as part of effective backup management.
For personal archives, the principle is the same at smaller scale. Open a sample of the photos. Extract the ZIP. Mount the disk image. Restore the database somewhere harmless. Verify that the supposedly preserved material is actually readable.
The worst time to learn how your backup works is immediately after the original disappears.