45 Sysadmin Horror Jokes Every On-Call Engineer Knows

The intern ran rm -rf in the wrong terminal. The terminal was prod. It was 4:55 p.m. on a Friday.

"Don't worry, the change is reversible." Narrator: It was not reversible.

The cert expired during the live customer demo. The sales rep refreshed three times. The sales rep is now my problem.

"I'll just push this small fix before the weekend." That sentence has started more outages than any virus in history.

The DBA dropped a table. The table was named users. The environment was named prod-east.

The backup ran for two years. The backup was writing to /dev/null.

Someone typed sudo shutdown -h now into the wrong ssh window. The server was in another country. It would not come back without a hand at the rack.

"It's just a config reload." The config reload took the load balancer down with it.

I labeled the cable. I labeled it correctly. The label was on the wrong cable.

The deployment script worked perfectly. The deployment script targeted the wrong environment. The wrong environment was production.

"Did the change go through?" "Yes." "Did the rollback go through?" "…also yes."

The senior engineer left a note on the server: "Do not reboot under any circumstances." The new vendor rebooted it during planned maintenance.

The disk filled at 4:43 p.m. The cleanup script ran at 4:44 p.m. The cleanup script was the disaster.

We had two databases: prod and prod-backup. We restored from prod-backup. prod-backup had not been written to since 2019.

"Just one tiny tweak to the firewall rule." The tiny tweak locked us all out of the firewall.

The auto-scaler scaled. The auto-scaler scaled to 4,000 instances. The bill arrived on a Sunday.

I ran the migration script in a test environment. The test environment shared a database with prod. I did not know this until 4:57 p.m.

The DNS change propagated faster than the rollback could be approved.

"The migration finished early." No migration has ever finished early. The migration crashed.

We had a runbook. The runbook was on the wiki. The wiki was on the server that went down.

The cleaner unplugged the server to plug in the vacuum. No logs. No alerts. No witnesses. We rebuilt the truth from disk timestamps.

"Are you sure that's the right server?" "Yes." "…how sure?" "Less sure now."

Someone enabled debug logging on the payment service. The logs filled the disk in 11 minutes. The payment service stopped processing. The debug logs captured the moment beautifully.

"I'll just rotate the keys real quick." Thirty-eight services lost authentication at 4:51 p.m.

The chmod was -R. The directory was /.

The vendor pushed an update. The update was a critical security patch. The critical security patch deleted our license file.

"The cluster is healthy." The cluster had two nodes. One was unreachable. The other was unreachable from the first one.

I deleted the snapshot before confirming the restore worked. The restore did not work. It is 5:02 p.m.

The deploy hook ran. The deploy hook ran twice. The deploy hook ran three times before anyone noticed it was looping.

"I think the load balancer is fine." The load balancer was returning 200 OK to its own health checks while every backend was on fire.

We split the database. We migrated half the tables. We forgot about the foreign keys. The application discovered them for us.

"The change was approved by change management." Change management approved the title of the ticket. Not the contents.

The expired cert was on the internal CA. The internal CA also signed the cert that lets us into the internal CA.

I rebooted the server. The server came back up. A different server came back down. We still don't know how those two were connected.

"It's a known issue." It was known by exactly one person. That person left six months ago.

Someone wrote a cron job that ran every minute. The cron job ran a database migration. The database migration locked a table. The table was used by every page.

The monitoring system was monitoring itself. It told us everything was fine for six hours. It was not fine.

"Let's just restart the VPN." The restart required VPN access to authorize.

The on-call phone died. The backup on-call phone was on the dead phone's plan. The outage page ran for ninety minutes.

I sshed to host01. host01 was actually host10. I did the maintenance on host10 instead of host01. The customer demo ran on host10.

"It only takes effect on restart." The restart was at 3 a.m. The pager went off at 3:02.

The old engineer wrote a comment in the config: # do not change this line The new engineer changed it. It was the only line keeping the cluster honest.

The disaster recovery drill was perfect. The disaster recovery drill ran on test data. The real disaster used prod data. The runbook did not apply.

"The system is self-healing." The system was healing into a wall.

I closed the laptop to go home. The laptop had the ssh session for the rolling restart. The restart paused at host 3 of 24. Monday morning was something.