50 Backup and Restore Jokes Every Sysadmin Has Lived

The backup job ran successfully every night for four years. The backup contained nothing. Nobody had ever opened it.

"We have backups." This is a sentence said with confidence by people who have never tried to restore one.

There are two kinds of admins: Those who test restores, and those who are about to learn what "corrupt archive" means.

The backup is fine. The restore is the problem.

"Did the backup complete?" "Yes." "What did it back up?" "…the empty directory."

I followed the 3-2-1 rule: 3 copies, 2 media, 1 offsite. All three were on the same disk.

The offsite backup was offsite. The offsite location was the same building. The building was on fire.

"The snapshot is from last night." The snapshot is from last August. The cron job stopped running quietly.

Schrödinger's backup: The state of the backup is unknown until you try to restore it.

The tape backup was perfect. We no longer have a tape drive.

"We're using cloud backup, so we're safe." The cloud backup was in the same region as the production database.

The restore took 14 hours. The restore was for a 3GB database. The restore was reading from a tier-three glacier.

"We don't need to test the backup." Famous last words. Followed shortly by "Can you call the CTO?"

Backup ran at 2 a.m. Database was being migrated at 2 a.m. Backup captured a corrupted mid-flight state. This went on for 90 days before anyone tried to restore.

"Why is the backup so small?" It's been writing to a full disk for six months. Nothing has actually been backed up.

I had a great backup. I also encrypted it. I lost the encryption key. I now own an expensive collection of random bytes.

The replication was real-time. The replication faithfully replicated the deletion.

"We have RAID, so we're covered." RAID is not a backup. RAID will protect your typo from itself with mathematical precision.

The backup script exited with code 0. The backup script also exited before any data was written. Code 0 means many things to many people.

User: "Can you restore the file I deleted this morning?" Me: "The backup runs nightly." User: "Why?"

Backups are about trust. Restores are about humility.

"Did anyone check the backup log?" The backup log was 14,000 lines of green, with one line of red on March 4th, repeated daily since.

The off-site backup was a USB drive in a drawer at the previous CIO's house.

"The backup is incremental." The full backup the incrementals are based on no longer exists.

I once restored a backup from 2016 because the more recent ones were all empty.

"We have point-in-time recovery." The point in time is 11 hours before the problem started. Good luck.

Restore plan: 1. Locate latest backup. 2. Pray. 3. Update resume during step 2.

The backup was checked monthly. The check was "did the file exist." It existed. It was 0 bytes.

"Why don't we restore from yesterday's backup?" Yesterday's backup is the one that overwrote Wednesday's good one.

I trust people. I just trust verified restore tests more.

"How long would a full restore take?" Nobody has ever attempted it. We estimate somewhere between four hours and four lifetimes.

The backup window is supposed to be at night. The backup window is now 26 hours long. It runs continuously.

"The vendor handles the backups." The vendor went out of business in 2022. Nobody told anyone.

I ran a test restore on a copy. It worked beautifully. The production restore did not. The test environment was missing the constraint that broke production.

Backups are like seat belts. Untested, theoretical, and you find out at the worst possible moment.

"I deleted it from my laptop. Can you restore it from the server backup?" The laptop wasn't backed up. We don't back up laptops. We have told you this. You signed a form.

The disaster recovery site was tested annually. The annual test had been skipped for three years. Nobody wrote it down.

"Why is the restore failing?" Because the backup format changed in the last upgrade. The old backups are unreadable by the new software.

The encrypted backup was decryptable by exactly one person. That person was on a hiking trip in the mountains. No signal.

I have one job at 2 a.m.: Figure out whether the backup includes the table that just got dropped.

"We have backups going back five years." The backups going back five years are on a tape format the current hardware cannot read.

Restore time on paper: 2 hours. Restore time in practice: 2 days. The difference is everyone in the company watching.

The backup script worked. The backup script wrote to /backup. /backup was a regular directory. /backup was on the same drive as the source. The drive failed.

"Did we test the failover?" We tested the failover. We did not test the failback. We are still in failover. It has been a year.

The cloud provider had backups of our backups. The cloud provider charged us $14,000 to give them back.

"The backup was successful." The backup was successful at being a backup of nothing.

Some teams do tabletop disaster recovery exercises. Our tabletop exercise is just the actual disaster, every 18 months.

I labeled the backup tapes by hand in 2014. I cannot read my own handwriting in 2024. The future is now.

"We're moving to a new backup system." The migration plan does not include verifying the old backups before retiring them.

Being responsible for backups is: Doing perfect work that nobody sees, until the day something fails and everybody learns your name at the same time.