DevOps and Sysadmin Jokes: Deploys, On-Call, Docker, and Backups

"Is the deploy done?" "It's deploying." "How long?" "Yes."

DevOps is the culture of blaming YAML indentation in front of management.

The pipeline passed locally. The pipeline always passes locally.

I have eight monitoring tools and none of them tell me what is broken.

Rollback plan: Step 1: pray. Step 2: refresh the dashboard. Step 3: act surprised.

"Who has access to prod?" Silence. "Right, everyone, then."

The CI job failed because a dependency we did not pin updated itself overnight.

Infrastructure as code, except half of it is a Confluence page from 2019.

"The pipeline is flaky." Translation: I have given up understanding it.

Every DevOps engineer has the same recurring nightmare: Uncommitted changes on a server nobody can SSH into anymore.

I do not write code. I write YAML that writes code that writes more YAML.

The blast radius of a single Terraform apply is somewhere between a typo and a resume update.

"It builds on my machine." "Then we will ship your machine."

Senior engineers can sense a misconfigured load balancer the way dogs sense thunder.

DevOps interview question: "Walk me through what happens when you push a commit." Correct answer: "Honestly, I am still figuring that out."

The deploy completed successfully. The service is also down. These facts are unrelated, apparently.

We have a runbook. It is six months out of date and references a tool we no longer use.

"Just spin up a new environment." That sentence has cost me entire weekends.

The cheapest part of cloud infrastructure is the demo. Everything after that is a negotiation.

I have a folder called `scripts` and a folder called `scripts-old`. Neither is safe to delete.

"We follow GitOps." Meaning: we have a manual step nobody documented.

The on-call rotation includes one person who has been on vacation for two weeks and one who quit last month.

The Terraform state file is the single most important file in the company and lives in an S3 bucket nobody remembers creating.

"Why is the build slow?" Because we cache nothing and rebuild everything from scratch every time, like artisans.

The post-mortem identified six root causes. We addressed two.

Kubernetes solved one problem and created a job category.

"The metrics look fine." The customer says the app is down.

I do not have impostor syndrome. I have a dashboard that does not match what is happening in production.

The DevOps handover document is a Slack DM from the previous engineer that just says: "good luck, sorry."

There are two states for a CI runner: On fire, or about to be.

"We will fix it in the next sprint." It has been in the next sprint for two years.

The most expensive line in any cloud bill is the one nobody can identify.

I once tried to delete an unused IAM role. Three services went down. I added a comment saying "important" and walked away.

Real DevOps maturity is admitting you copy your last working Helm chart and edit the names.

"Self-healing infrastructure." It heals by restarting until the alert stops paging.

The deploy pipeline has 47 stages. Forty-six of them are linting.

"Did the migration finish?" "It's at 99%." "How long has it been at 99%?" "...three hours."

Disaster recovery plan: A Google Doc that has not been opened since the offsite.

The most reliable monitoring tool in production is a customer in a support ticket.

"We need observability." We have nine tools that each show one quarter of the picture.

The cost of switching cloud providers is two years and a vice president's reputation.

There is no such thing as a temporary feature flag. There are only flags older than the engineer who added them.

"Why is staging different from production?" One is loved. The other is real.

A junior engineer ran `terraform apply` on the wrong workspace. The whole team learned what idempotent does not actually mean.

DevOps culture: Everyone owns the pipeline. The pipeline is broken. Nobody owns the pipeline.

The shortest blameless post-mortem I ever read was four pages of names.

Secret management is just deciding which env file you trust the most.

"Zero downtime deploy." The deploy was zero downtime. The app was down for other reasons.

I added a single line of YAML. Three namespaces stopped resolving.

The DORA metrics say we are an elite performer. The customer says the login page is broken.

"How do we improve deploy frequency?" Stop deploying things that break.

Our SLO is 99.9%. Our actual uptime is whatever the public status page says it is.

There are two kinds of engineers in a major outage: The ones in the war room and the ones who turned their Slack notifications off.

The pipeline retries failed jobs three times. Flaky tests have learned to behave on the third attempt.

"We should automate this." It is now a manual step inside an automated process.

Every alerting rule in the system was added in response to a specific outage and nobody remembers which one.

I do not deploy on Fridays. I deploy on Thursdays at 4:45 and tell myself that is different.

The Helm chart works. The Helm chart values file works. The combination is undefined behavior.

"What does this Bash script do?" Nobody knows. It has been in the deploy pipeline since 2018.

The kubectl command I needed was in a Slack message from a former coworker. The workspace was archived last month.

"We are platform engineers now." Same on-call rotation, fancier title.

The fastest way to expose a security gap is to give a new hire admin access for their first week.

I trust my pipeline the way I trust a vending machine. Most of the time it works. The rest of the time I lose a dollar and rage-shake it.

The infrastructure team and the application team agree on one thing: The other team's monitoring is worse.

DevOps is mostly explaining to someone in a meeting that the thing they want already exists, but they cannot find it because nobody documented it.

"It works on my machine." "Then we will ship your machine." And that is how Docker was invented.

My Dockerfile is 14 lines. My image is 2.7 gigabytes. These numbers feel disproportionate.

"Just use Alpine." Three hours later: missing libc, missing curl, missing the will to live.

The first rule of Docker: It works locally. The second rule of Docker: It does not work in CI.

Every Dockerfile I have ever written starts with a comment that says "TODO: optimize layers" and stays that way for three years.

"Why is the build slow?" Because you invalidated the cache by editing the COPY line.

Docker Hub rate limits are how I learned that anonymous pulls are a luxury, not a right.

I asked the junior to dockerize the app. He added FROM ubuntu:latest at the top and called it a day.

A `docker ps` returns nothing. A `docker ps -a` returns 47 containers. None of them are mine, apparently.

"Why is my container exiting immediately?" Because your CMD ran and finished. That is the whole job of a CMD.

Building from a Dockerfile is fine. Understanding someone else's Dockerfile is archaeology.

There are two kinds of people: Those who pin image versions, and those who learned to the hard way.

"latest" is not a version. It is a promise from someone who has already moved on.

Multi-stage builds saved my image size. They also turned my Dockerfile into a screenplay.

I ran `docker system prune -a`. The build cache I had been nurturing for months is gone. The disk is happy. I am not.

Bind mounts on macOS are slow because the universe is fair.

"Just rebuild the image." Sure. I have forty minutes.

Every `docker run` command in production was assembled from three different blog posts and one Slack message.

Docker Compose is what we use until we say the word Kubernetes out loud at a meeting.

"The container is running." "The app inside the container is not." "Yes, but the container is running."

I added a HEALTHCHECK to the Dockerfile. Now my container restarts every 30 seconds and I have no idea why.

There is a special place in production for the engineer who runs containers as root.

Container networking is straightforward once you accept that nothing is straightforward about container networking.

"Why is the volume empty?" Because you mounted a host directory over the one that had data in it. That is just how mounts work.

I have a Dockerfile that has not been edited in four years. It is the only thing in our infrastructure that has never gone down.

Three engineers, three machines, three different results from the same `docker build`. This is what BuildKit was invented to fix and somehow has not.

Docker security scan results: 247 known vulnerabilities. Fix: pin to a different image with 248 known vulnerabilities.

The Dockerfile says `FROM node:18`. The lockfile expects 20. The build passes. The runtime does not.

"Just exec into the container and check." The container is `distroless` and does not have a shell. Good luck.

Every production incident eventually leads back to someone using `:latest` somewhere.

I tried to write a small image. It is 180 megabytes. I checked the size of a Hello World binary in Go. It is 6 megabytes. I hate everything.

"The container is on the host network." Then it is not really a container, friend.

Docker Desktop wants to update. Docker Desktop always wants to update. The update will break my volumes. I know this. I click later.

Layer caching is a beautiful idea. Layer caching that respects your COPY order is a unicorn.

"Use a .dockerignore file." The one nobody adds until their image hits 5 gigabytes.

I built a container that runs Postgres, Redis, the app, the worker, and the cron job. It also runs nothing for very long.

ENTRYPOINT versus CMD is a thirty-minute conversation that resolves to "use both and hope."

"Why is the timezone wrong inside the container?" Because containers have no concept of where you live until you mount /etc/localtime.

Docker on ARM. Docker on x86. Same image. Different behavior. Welcome to multi-arch.

I once spent four hours debugging a permissions issue that turned out to be a UID mismatch on a bind mount. That is an entire afternoon I will never get back.

"Run the container in detached mode." Great. Now I have no idea what it is doing and no logs.

`docker logs` is the only debugging tool I have. It is also the only one I trust.

The image is small. The app inside the image is broken. We call this engineering progress.

A `RUN apt-get update` without `&& apt-get install` is a layer of pure waste. Nobody told me that for two years.

Every Dockerfile has one line that nobody on the team can explain and nobody is brave enough to delete.

"Why is the image so big?" Because you copied node_modules in before running `npm ci`.

Docker Compose worked until it didn't, and now the YAML file is 400 lines and somehow still missing a service.

I read "containers are lightweight" once in 2014 and have been disappointed ever since.

"The container has its own filesystem." Yes, and you just wiped it because you restarted without a volume.

There are two ways to debug a Docker network issue: Give up, or read the source of libnetwork. Both are valid.

The container starts in 200 milliseconds. The app inside takes 90 seconds to be ready. Guess which one the orchestrator believes.

"Use a sidecar." Now I have two containers that do not work instead of one.

I keep a `Dockerfile.dev` and a `Dockerfile.prod`. They have drifted so far apart they no longer know each other.

"Mount the Docker socket into the container." That is also known as: "give the container root on the host." Just so we are clear.

Every team has a `base-image` repo that nobody owns and everyone depends on.

I tried `docker-compose up` on a Monday morning. It is now Wednesday and I am still rebuilding images.

"Why does the image work on staging but not prod?" Because prod uses a different registry and you forgot to push the new tag.

I labeled my image with a git SHA. I deployed an image with a different git SHA. Nobody can explain this.

The fastest way to find out what is inside a container is to wait for it to crash and read the logs.

"Stateless containers." My container has a file in /tmp that the whole business depends on.

Pulling an image over hotel Wi-Fi is a spiritual exercise.

Docker volumes outlive the company that created the container that wrote them.

"Use docker init." It generated a Dockerfile that does not work for my app. Useful.

The container restarts. The container restarts. The container restarts. Kubernetes calls this self-healing. I call it denial.

I run the same `docker build` command twice in a row. Different output. No code changed. This is where my faith in determinism ended.

A container is just a process with delusions of isolation.

"Use a non-root user inside the container." The app no longer has permission to write to the volume it needs. Wonderful.

Docker Hub deprecated the image I built my whole stack on. This is what they call a learning opportunity.

I have a folder full of half-finished Dockerfiles named `Dockerfile`, `Dockerfile.bak`, `Dockerfile.old`, and `Dockerfile.final.actually-final`.

The thing nobody tells you about Docker is that the hard part is not building the image. The hard part is convincing other people to use yours instead of writing their own.

"It's a small change." Four words, one outage.

Deploy at 4:55 p.m. on Friday. What could possibly happen between now and Monday.

The PR description said: "Minor refactor." It was 2,400 lines.

"I tested it locally." Production is a different country.

We have a deploy freeze. Everyone has at least one exception.

The pipeline went green. The service went red. These are unrelated.

"Did you rollback?" "To what?"

Hotfix: the deploy you ship to fix the deploy you shipped twenty minutes ago.

Feature flag default: off. Feature flag in production: on. Nobody remembers who flipped it.

The deploy lock was held by someone on vacation.

"Why did the build pass?" "Because the tests passed." "Why did the tests pass?" "Because they don't test that."

Blue-green deployment in theory: seamless cutover. Blue-green deployment in practice: green is on fire, blue is wedged, both are serving traffic.

I clicked deploy. The button said "deploying." It said that for 47 minutes. It is still saying it.

"Don't deploy on Fridays." The most ignored advice in software.

The release notes are one line: "Bug fixes and improvements." The diff is 3,000 lines.

The deploy succeeded. The service stopped. The monitors did not alert. The users alerted.

Canary deployment: 1% of users get the experience first. They are the 1% chosen to discover that the database migration was missing.

"It worked in staging." Staging is a lovely place. Nothing real lives there.

We have CI/CD. We also have a person who runs the deploy by hand on Wednesdays for reasons nobody remembers.

"Who approved this PR?" The approver had two hours of sleep and was attending a meeting on mute.

Rollback procedure: step one, panic. step two, find the previous tag. step three, discover the tag does not exist. step four, panic differently.

"We follow trunk-based development." The trunk is on fire and nobody is checking in.

The deploy was at 3:00 p.m. The incident started at 3:01 p.m. The denial started at 3:02 p.m.

The deploy pipeline has 14 stages. It failed at stage 13. The error message is: "unknown."

"Did you bump the version?" "Did I need to?" "Yes."

Production access controls: strict, audited, multi-factor. Production access in practice: everyone has root because of that one time.

The migration was online. The migration was reversible. The migration also was not.

We rolled back. The rollback also broke.

"This deploy doesn't need a postmortem." Five hours later: we're writing one.

Database migration ran in 12 seconds in staging. It has been running for 90 minutes in production. The table has 800 million rows. Nobody told staging.

The release manager said: "Quiet release week." The pager has heard them.

"Just merge it. We'll fix forward." Nobody fixes forward. They roll back at 11 p.m.

Feature flag with 14 conditions, four environments, three user segments, and a kill switch. It's a config file with abandonment issues.

The deploy required a runbook. The runbook required a runbook.

"Have we deployed this before?" "Yes." "Did it work?" "…define work."

Friday deploy. Weekend incident. Monday postmortem. Tuesday Friday deploy.

The DORA metrics looked great. The customers did not.

I added a feature flag. The flag is now production code. It will be there in 2031.

The build cache was stale. The build cache is always stale. The build cache has been stale since the dawn of build caches.

"We need to deploy now." "Why?" "The CEO is demoing in twenty minutes."

The deploy worked. The service degraded. The degradation was the previous deploy waking up late.

Container image size: 4.2 GB. Application size: 12 MB. Nobody knows what the other 4.188 GB does.

"The pipeline is flaky." The flakiness is the only consistent thing about the pipeline.

I deployed to the wrong cluster. I noticed by reading customer tweets.

The CI was green for the first time in three months. We deployed immediately, suspicious of our luck.

"Did anyone test the rollback?" The silence answered.

We have feature flags so we can deploy without users noticing. We also have outages so users notice anyway.

The deploy took eight seconds. The meeting about whether to deploy took ninety minutes.

"This change is backward compatible." With what, exactly.

I deployed an empty change to test the pipeline. The empty change broke production. We still do not know why.

Production parity: the staging environment matches production in exactly the ways that do not matter.

"We use GitOps now." The Git repo has 14 open PRs and three force-pushes from last week.

The cost of a Friday deploy: one weekend. The cost of avoiding a Friday deploy: features shipped a week earlier. Management does the math wrong every time.

The deploy script ended with: "# TODO: handle the rollback case"

Postmortem note: "The deploy was uneventful." Line two of the same postmortem: a forty-minute outage.

The phone rang at 3:07 a.m. The page resolved itself by 3:09. I lay awake until 6.

"How was your week on call?" I stared into a middle distance. They did not ask again.

I sleep with the laptop on the nightstand. This is normal now.

The on-call handoff: "Nothing's broken." Three minutes later: everything is broken.

I keep the volume at maximum. The cat has opinions about this.

"Are you on call this weekend?" Do not ask me that on a Thursday.

The page came in. I opened the laptop. The laptop needed updates. The updates needed a restart. The service self-healed before I logged in.

I declined the wedding. I was on call. They understood. They have not invited me to anything since.

"Why didn't you escalate?" I did. Nobody answered.

The runbook said: "Restart the service." It did not say which one.

On-call dinner plans: Whatever can be reheated.

The incident commander asked for an ETA. I said "soon." That was an hour ago.

"Did the page wake you?" No. The page kept me awake. The previous page woke me.

I packed for vacation. I packed the laptop too. My partner noticed. We had a discussion.

The rotation is one week. The nightmares are one quarter.

"It's not your fault." The sentence everyone says. Nobody believes.

I forgot to hand off. The pager forgot to remind me. I got paged for someone else's week.

"I'm at dinner." "Are you on call?" "…yes." "Then no, you are not at dinner."

The page came in during the school recital. I typed into a phone with one hand and clapped with the other.

Postmortem question one: "Was the on-call engineer awake?" Always the wrong opening question.

I have a Pavlovian flinch at the PagerDuty ringtone. I changed the ringtone. Now I flinch at the new one too.

The on-call week before vacation is a cursed object.

"Can you take a quick look?" It's never quick. It's never a look.

I learned more in my first on-call week than in six months of normal work. I do not recommend the curriculum.

The page resolved. The Slack thread continued for two hours. Nobody slept.

On-call rule: If you say "quiet week" out loud, you lose it.

I missed the page. The page was for a service I do not own. I got a stern note anyway.

"Did you read the runbook?" "The runbook is the one I am writing right now."

The escalation path was three people. Two were on PTO. One was me.

I learned to type on a phone keyboard while standing in a parking lot in the rain.

The dog has learned what the pager sound means. The dog leaves the room when it goes off.

"It's only a one-week rotation." It is one week. It feels like four.

On-call is a great way to discover all the documentation gaps you swore you'd fix last quarter.

The midnight page was for an alert someone added without testing. They added a comment: "Adjust threshold later." They added it last March.

"Can we move this incident to business hours?" We have asked the universe. The universe declined.

I tried to take a vacation day on my on-call week. The scheduler laughed in cron.

The bartender asked why I kept checking my phone. I told him. He poured me another for free.

My partner used to find the pager dramatic. Now they roll over and go back to sleep before I even answer.

"Was it a real incident?" "Define real."

The page fired. The metric had a typo. The service was fine. The sleep was not.

I closed my laptop at 4:42 a.m. The sun came up at 6:11 a.m. In between, I dreamt of YAML.

Every on-call engineer has the same three browser tabs open: the alert, the dashboard, and a doc that does not help.

"Are you on call?" "Always." "That's not how rotations work." "You have not seen my rotation."

Recovery time after a week on call: three weekends.

The incident channel had 47 people in it. Three were doing work.

I joined the bridge. Nobody was talking. Forty people were muted. This lasted nineteen minutes.

"You handled that well." I did not. I just stopped responding while it was still happening.

The on-call calendar is the only calendar I check more than once a day.

"It self-recovered." The phrase that should bring relief. It brings dread.

I keep a notebook on the nightstand. It is full of pages that start "3:14 a.m. — " and trail off.

Friday at 4:55 p.m. The page fires. The weekend begins as it means to continue.

The new hire said: "I look forward to being on call." We smiled. We said nothing.

Hour one of an incident: panic. Hour two: focus. Hour three: a kind of calm only sleep deprivation can provide.

"Why is the same person on call every week?" Because everyone else found a reason to skip the rotation.

The page came in on the way to the dentist. I sat in the chair, opened a terminal on my phone, and confirmed disaster between rinses.

Compensation for being on call: a small stipend. The stipend does not buy back the night.

The CTO asked how on-call was going. I said "fine." The CTO went back to whatever the CTO does at 11 a.m.

The runbook had 31 steps. The first step was "don't panic." I was already on step 14 of panic.

After a year on call, I can tell which alert is firing by the ringtone alone. This is a skill nobody should have.

"What's the worst on-call you've had?" Whichever one I'm on this week.

My phone buzzed. I flinched before I read it. It was a calendar reminder.

"Did you see the alert?" "Which one?"

The dashboard has 47 panels. I watch two.

Severity levels at my company: P1, P2, P3, and "this one is real."

The runbook says: "If this fires, see the owner." The owner left in 2021.

Alert: CPU at 92%. Me: that's a Tuesday.

I muted the channel. The channel was the on-call channel.

"Why didn't anyone respond?" Because the same alert has fired hourly since March.

The alert was silenced. It fired anyway. I respect its commitment.

Disk full at 3:14 a.m. Disk fine at 3:17 a.m. Nobody acknowledged. This happens every night. Nobody investigates.

"It's just a warning." The warning has been firing for six months.

I added a new monitor. I deleted three to stay sane.

PagerDuty's most useful feature is the snooze button.

The alert title: "Something is wrong." Great. Thanks.

We have alerts for the alerts.

"The monitor is broken." "How do you know?" "It hasn't paged in a week."

I was paged for a service I have never heard of.

The dashboard is green. The users are calling. One of these is lying.

On-call onboarding: "Here is the runbook." "It's empty." "Good luck."

"Did you check the graph?" "Which one?" "Any of them."

I have four monitoring tools. Each disagrees with the other three.

The alert fired because the metric reporter died. The service was fine.

"This is a noisy alert." The ticket has been open for two years.

I dreamed in PromQL again.

"It self-recovered." The most common resolution note in the company.

Every dashboard has a panel labeled "misc" that secretly runs the business.

Anomaly detection flagged Sunday as an anomaly. It was Sunday.

The on-call handoff is one sentence: "Good luck with the queue."

I built a status page. It has its own status page.

The vendor said: "You'll never miss an alert again." They were right. I miss none of them. I also sleep none.

"What does this alert mean?" "Nobody knows. It came with the platform."

The threshold was set in 2019. The service grew tenfold. The threshold did not.

I clicked acknowledge. It fired again before the modal closed.

The escalation policy ends at "call CEO." Nobody has ever reached step five. Nobody wants to find out what happens.

"Why is this alert P1?" "Someone got paged on a weekend once and made it P1."

Three monitors. One service. One page each. Four minutes of phone vibrating.

I trust the customer ticket more than the dashboard.

"The SLO is at 99.9%." The users are at 100% angry.

The healthiest service in the company is the one nobody monitors. For now.

Synthetic monitor fails. Users fine. It's the synthetic. It's always the synthetic.

The alert message is "see playbook." The playbook says "see alert."

"Can we make this less noisy?" Three months later: same noise, new dashboard.

I built alert fatigue dashboards to track our alert fatigue.

We measured time-to-acknowledge. It got worse the more we measured.

The on-call sleep schedule: Phone face up. Brightness max. Volume max. Dignity zero.

"Is this a real one?" The only question that matters at 3 a.m.

The alert title was in Latin. I Googled it. It was a Kubernetes default.

We have 14 different definitions of "healthy."

The dashboard was beautiful in the demo. It has not been opened since.

"Are we under SLO?" "Define SLO."

Postmortem action item: "Improve monitoring." Number of postmortems with that exact line: all of them.

I unsubscribed from one alert channel. My Slack got 3% quieter. I was paged for what I missed.

"It's flapping." The word that excuses six months of inaction.

An alert fired with the description: "This shouldn't happen." It happens daily.

My favorite alert resolution note is one word. "Yes."

"Did you reboot it?" "Yes." "Properly?" "…define properly."

A sysadmin's version of optimism: "At least the backup server is still pinging."

Users: "The internet is down." Reality: One tab froze in Chrome.

I trust people. I just trust logs more.

"Nothing changed." Famous last words before a 4-hour outage.

Sysadmin rule #1: If you touched it last, it's your problem now.

The printer stopped working because it sensed confidence.

Every office has that one server nobody understands. It has been running since 2011 and controls reality itself.

"Quick question." There are no quick questions in IT.

A sysadmin can hear a failing hard drive from three rooms away.

"Can you just…" No. Whatever follows "can you just" is never simple.

My favorite exercise is running `ping` and staring aggressively at packet loss.

The Wi-Fi password changes whenever someone says: "IT doesn't really do much around here."

"The server room is too cold." Good. Fear keeps the systems stable.

Nothing humbles a sysadmin faster than DNS.

"Why is the website down?" Because someone thought production was the perfect place to test something.

There are two kinds of sysadmins: Those who test backups, and those about to learn why they should.

The network was working perfectly until someone said: "Looks stable today."

Sysadmins don't panic. They open six terminals and become unusually quiet.

"I clicked the link in the email." And that's how my afternoon disappeared.

Every cable in the rack has a purpose. Except the one nobody dares unplug.

"Temporary fix." Last updated: 2017.

A sysadmin's natural predator is upper management asking: "Can we do this without downtime?"

"The cloud" just means someone else's server you'll still get blamed for.

If you listen carefully, you can hear a sysadmin whisper: "Who changed the firewall rules?"

The problem with automation is that it automates mistakes really efficiently.

"We don't need documentation." Spoken moments before disaster.

A good sysadmin remembers passwords. A great sysadmin remembers where the password spreadsheet is.

I don't always check logs. But when I do, I regret it immediately.

"It worked on my machine." The anthem of chaos.

Every sysadmin has a folder called: `old_final_v2_REAL_THIS_ONE`

The server uptime was so impressive nobody wanted to reboot it out of fear.

"Can you recover the deleted file?" "Can you recover my will to live?"

A sysadmin's idea of adventure is updating production on a Friday.

Never trust a cable labeled correctly. It's a trap.

"The application is slow." Translation: Nobody knows what's wrong yet.

The louder someone says "simple change," the worse the outage will be.

A sysadmin can survive entirely on caffeine, sarcasm, and SSH.

"Why do we need monitoring?" Because surprises are expensive.

Users think restarting fixes everything. Sysadmins know sometimes it really does.

"I accidentally deleted the database." "The whole thing?" "…define whole."

The most dangerous sentence in IT: "I was cleaning things up."

A sysadmin's trust issues begin with expired SSL certificates.

"Can you whitelist my IP?" "Which one?" "The internet one."

Some people hear birds chirping. Sysadmins hear UPS alarms in their nightmares.

"The server has plenty of space left." Disk usage: 99.8%

The office gets nervous when the sysadmin says: "Huh. That's weird."

Sysadmins don't fear horror movies. They fear undocumented cron jobs.

Nothing strengthens team bonding like debugging production at 4:15 a.m.

Being a sysadmin is basically: Preventing disasters nobody notices while fixing emergencies everyone notices.

The intern ran rm -rf in the wrong terminal. The terminal was prod. It was 4:55 p.m. on a Friday.

"Don't worry, the change is reversible." Narrator: It was not reversible.

The cert expired during the live customer demo. The sales rep refreshed three times. The sales rep is now my problem.

"I'll just push this small fix before the weekend." That sentence has started more outages than any virus in history.

The DBA dropped a table. The table was named users. The environment was named prod-east.

The backup ran for two years. The backup was writing to /dev/null.

Someone typed sudo shutdown -h now into the wrong ssh window. The server was in another country. It would not come back without a hand at the rack.

"It's just a config reload." The config reload took the load balancer down with it.

I labeled the cable. I labeled it correctly. The label was on the wrong cable.

The deployment script worked perfectly. The deployment script targeted the wrong environment. The wrong environment was production.

"Did the change go through?" "Yes." "Did the rollback go through?" "…also yes."

The senior engineer left a note on the server: "Do not reboot under any circumstances." The new vendor rebooted it during planned maintenance.

The disk filled at 4:43 p.m. The cleanup script ran at 4:44 p.m. The cleanup script was the disaster.

We had two databases: prod and prod-backup. We restored from prod-backup. prod-backup had not been written to since 2019.

"Just one tiny tweak to the firewall rule." The tiny tweak locked us all out of the firewall.

The auto-scaler scaled. The auto-scaler scaled to 4,000 instances. The bill arrived on a Sunday.

I ran the migration script in a test environment. The test environment shared a database with prod. I did not know this until 4:57 p.m.

The DNS change propagated faster than the rollback could be approved.

"The migration finished early." No migration has ever finished early. The migration crashed.

We had a runbook. The runbook was on the wiki. The wiki was on the server that went down.

The cleaner unplugged the server to plug in the vacuum. No logs. No alerts. No witnesses. We rebuilt the truth from disk timestamps.

"Are you sure that's the right server?" "Yes." "…how sure?" "Less sure now."

Someone enabled debug logging on the payment service. The logs filled the disk in 11 minutes. The payment service stopped processing. The debug logs captured the moment beautifully.

"I'll just rotate the keys real quick." Thirty-eight services lost authentication at 4:51 p.m.

The chmod was -R. The directory was /.

The vendor pushed an update. The update was a critical security patch. The critical security patch deleted our license file.

"The cluster is healthy." The cluster had two nodes. One was unreachable. The other was unreachable from the first one.

I deleted the snapshot before confirming the restore worked. The restore did not work. It is 5:02 p.m.

The deploy hook ran. The deploy hook ran twice. The deploy hook ran three times before anyone noticed it was looping.

"I think the load balancer is fine." The load balancer was returning 200 OK to its own health checks while every backend was on fire.

We split the database. We migrated half the tables. We forgot about the foreign keys. The application discovered them for us.

"The change was approved by change management." Change management approved the title of the ticket. Not the contents.

The expired cert was on the internal CA. The internal CA also signed the cert that lets us into the internal CA.

I rebooted the server. The server came back up. A different server came back down. We still don't know how those two were connected.

"It's a known issue." It was known by exactly one person. That person left six months ago.

Someone wrote a cron job that ran every minute. The cron job ran a database migration. The database migration locked a table. The table was used by every page.

The monitoring system was monitoring itself. It told us everything was fine for six hours. It was not fine.

"Let's just restart the VPN." The restart required VPN access to authorize.

The on-call phone died. The backup on-call phone was on the dead phone's plan. The outage page ran for ninety minutes.

I sshed to host01. host01 was actually host10. I did the maintenance on host10 instead of host01. The customer demo ran on host10.

"It only takes effect on restart." The restart was at 3 a.m. The pager went off at 3:02.

The old engineer wrote a comment in the config: # do not change this line The new engineer changed it. It was the only line keeping the cluster honest.

The disaster recovery drill was perfect. The disaster recovery drill ran on test data. The real disaster used prod data. The runbook did not apply.

"The system is self-healing." The system was healing into a wall.

I closed the laptop to go home. The laptop had the ssh session for the rolling restart. The restart paused at host 3 of 24. Monday morning was something.

"Have you tried sudo?" "Yes." "Have you tried sudo !!?"

I don't have a problem. I have a uname -a and 12 open tabs.

The fastest way to get help on Linux is to post a wrong answer in a forum.

Vim users don't quit. They reboot.

"How do I exit vim?" Nobody knows. We all just live here now.

There are two kinds of Linux admins: Those who use tabs, and those who are wrong.

rm -rf is not a command. It is a personality test.

The man page for tar is longer than some novels and resolves about as cleanly.

"It works on my Arch."

A Linux admin's favorite mystery novel: /var/log/syslog from last Thursday.

I told my wife I needed a quiet weekend to recompile the kernel. She filed for divorce on Saturday.

"The server is slow." top says load average: 47.3

Junior: "What does this script do?" Me: "It has been running in production for six years. Do not touch it."

Linux is free if your time is worthless.

Every distro is the best distro until you have to install a printer.

The cron job nobody wrote, on the server nobody owns, is the one keeping the company alive.

I tried to explain inodes to a recruiter. The call ended early.

df -h says 100%. du -sh says 60%. Welcome to Linux.

"Just chmod 777 it." Famous last words of an entire security incident.

A Linux admin walks into a bar. The bartender says "permission denied."

SELinux is set to permissive in production. It has been since 2014. Nobody knows who did it.

"Did you read the changelog?" The changelog is 12,000 lines and starts with "misc fixes."

My pets are named less, grep, and awk. They come when called. Mostly.

Real admins don't use GUIs. Real admins ssh into the GUI server and edit the config file.

"I rebooted into single-user mode." "Why?" "To feel something."

Every Linux admin has a tmux session older than their relationship.

"I'll just edit /etc/fstab quickly." The last words of the server, captured forever in dmesg.

The most honest Linux distro would just be called yes-i-broke-it.

"You don't need swap on modern systems." OOM killer: "Hold my beer."

I once fixed a server by staring at it. I cannot reproduce this in QA.

iptables, nftables, ufw, firewalld. Four walls. None of them load on boot.

"Why is the disk full?" /var/log/journal: 38GB

A senior admin is just a junior admin who has been bitten by every default.

The kernel panic at 4:45 a.m. is the universe's way of asking if you really love your job.

"I use Gentoo, by the way." Nobody asked. Nobody ever asks.

My favorite Linux command is history | grep "what did past me do."

"What's your favorite editor?" This is how knife fights start at Linux meetups.

The server still running CentOS 6 has been promoted to deity status. Nobody touches it. Nobody upgrades it. Offerings are left in the rack.

"Just run the install script." curl | sudo bash is faith expressed in shell syntax.

I keep a notes file called fixed.txt. It is 4,000 lines. None of them are searchable.

"Is the service running?" systemctl: active (running) The service: deeply asleep

A Linux admin's vacation: still checking htop, just from a beach.

"Why is my shell prompt different?" Because your .bashrc has been quietly haunted since 2017.

The reason I distro-hop isn't because nothing satisfies me. It's because I haven't found a wallpaper I like yet.

"Use a configuration management tool." My configuration management tool is a Word doc titled server_passwords_FINAL.docx.

"Did you check the logs?" The logs are 14GB and the relevant line is on page 9,402.

Friday at 4:45 p.m. is when apt suddenly wants to remove 137 essential packages.

"The build is failing." "On Linux?" "No, on the CI runner that runs Linux but isn't really Linux."

I trust people. I don't trust their bash aliases.

"Just use Docker." The Docker container also runs Linux. The problem is still there. It is just wearing a costume.

My therapist asked what I do to relax. I said I read kernel commit messages.

Linux admins age in dog years. Every package upgrade adds six months.

The server is named after a Greek god. It has been down since Tuesday. The pantheon is silent.

"It's not Linux. It's GNU/Linux." There is always one. He is in every Slack channel. He never sleeps.

Being a Linux admin is: Fixing things nobody knew were broken using tools nobody documented for users nobody warned you about.

"Should we use Nginx or Apache?" Ask the room and watch two engineers stop being friends.

Apache has been running on a server in the basement since 2008. Nobody knows the root password. It has 100% uptime.

"Nginx is faster." "Apache is more flexible." "Both of you are wrong, it's the database."

An .htaccess file is a small piece of paper that says: "please redirect this, please." Apache reads it and obliges. Nginx sets it on fire.

The first sign someone is migrating from Apache is the sentence: "Where is mod_rewrite in Nginx?"

The Apache config file has 47 includes. The Nginx config file has zero, but somehow loads the same amount of config.

"Apache is dead." Apache, still serving 30% of the public internet, says nothing.

Nginx errors: Cryptic, terse, line numbers wrong by two. Apache errors: A novel.

"Why is the site slow?" Because you have prefork MPM enabled and 400 simultaneous connections. Pick a lane.

I asked an Nginx user how Apache handles requests. They described prefork the way you describe a haunted house.

"Just use Caddy." This is what people say when they want to leave the conversation.

Apache modules are loaded by default. All of them. Forever. This is fine.

An Nginx config reload can save your evening. An Apache config reload can also save your evening, in a different and slower way.

"Why does my rewrite rule not work?" Because this is Nginx and you wrote it in Apache syntax. We have all done it.

The number of people who fully understand Nginx's location block precedence rules is fewer than the number who claim to.

Apache's `.htaccess` is the convenience of editing config without restarting and the curse of editing config without thinking.

"Nginx config is declarative." It is also case sensitive, whitespace sensitive, and order sensitive. So declarative in name only.

Every old project has an `httpd.conf` and a `nginx.conf` and uses neither.

"We are putting Nginx in front of Apache." That sentence is the entire 2014–2019 backend industry in nine words.

Apache's documentation is comprehensive. Nginx's documentation is correct. These are different things.

The day I learned `try_files` was the day I stopped fighting Nginx and started worshipping it. I still got 404s.

"It's just a config file." Yes, and War and Peace is just a book.

I have written `worker_connections 1024` in every Nginx config I have ever shipped and I have never once known why.

"Apache is bloated." The entire team's Slack client is currently using more memory than Apache.

Nginx fans say config-driven. Apache fans say battle-tested. Both mean: "the one I learned first."

An Apache admin retiring is a kind of geological event. The knowledge they leave behind is half oral tradition, half post-it note.

"Just put a reverse proxy in front of it." The answer to every web architecture question since 2012.

Apache: include the conf.d directory. Nginx: include the conf.d directory. Both of us, mostly: copy the example file from the docs and edit one line.

The first time I tried to deny by IP in Nginx I learned that order matters. The second time I learned it again.

"Use Lua in Nginx." Now my web server is also a programming language. Cool. Cool cool cool.

Apache 1.3 ran for 19 years on a server in a closet at a university nobody remembers. It was finally turned off in 2024. A librarian cried.

"Why is mod_php still a thing?" Because shared hosting exists. Stop asking.

Nginx's `if` directive is documented as evil. Literally. The docs use the word evil. I still use it weekly.

Apache served the first website I ever broke. Nginx serves the last website I will break.

"What does this Apache config block do?" It was added in 2011 to fix something that has not been a problem since 2013. We are afraid to remove it.

Nginx config syntax does not have inline comments inside a directive value. I learned this by losing two hours.

"Apache is too heavy for static files." The site serves 300 requests a day. It is fine.

The Nginx upgrade went smoothly. The new binary loaded. The old workers drained. Nothing exploded. This is suspicious.

"Why doesn't gzip work?" Because you set `gzip on;` but did not set `gzip_types` and the default is `text/html` only. Welcome to Nginx.

Apache's `RewriteCond` and `RewriteRule` together are a small esoteric programming language. I have written four-line rewrites and called in sick the next day.

"Just use Apache for development and Nginx for production." The team is currently doing exactly that and the bugs are exquisite.

Nginx Plus is a paid product. Most people learn this by emailing support and getting a quote.

Apache's MPM choice is a decision you make once and regret on a different Thursday.

"Use HAProxy instead." That is a different argument, friend. Sit down.

Nginx is fine until you need WebSockets. Then Nginx is also fine, but only after you read the proxy_http_version note twice.

Apache is fine until you need WebSockets. Then Apache is also fine, but only after you enable two modules you have never heard of.

The Nginx config has one server block per virtual host. The Apache config has one VirtualHost block per virtual host. The production server has 47 of either kind and runs three sites.

"Why is the wrong site loading?" Because your default_server matches before your server_name does. Read the docs once more, slowly.

I once swore off Apache. I came back six months later because cPanel exists. There is no exit.

The holy war ended quietly. Nginx fronts everything. Apache sits behind it, doing the actual work, talking to nobody.

The backup job ran successfully every night for four years. The backup contained nothing. Nobody had ever opened it.

"We have backups." This is a sentence said with confidence by people who have never tried to restore one.

There are two kinds of admins: Those who test restores, and those who are about to learn what "corrupt archive" means.

The backup is fine. The restore is the problem.

"Did the backup complete?" "Yes." "What did it back up?" "…the empty directory."

I followed the 3-2-1 rule: 3 copies, 2 media, 1 offsite. All three were on the same disk.

The offsite backup was offsite. The offsite location was the same building. The building was on fire.

"The snapshot is from last night." The snapshot is from last August. The cron job stopped running quietly.

Schrödinger's backup: The state of the backup is unknown until you try to restore it.

The tape backup was perfect. We no longer have a tape drive.

"We're using cloud backup, so we're safe." The cloud backup was in the same region as the production database.

The restore took 14 hours. The restore was for a 3GB database. The restore was reading from a tier-three glacier.

"We don't need to test the backup." Famous last words. Followed shortly by "Can you call the CTO?"

Backup ran at 3 a.m. Database was being migrated at 3 a.m. Backup captured a corrupted mid-flight state. This went on for 90 days before anyone tried to restore.

"Why is the backup so small?" It's been writing to a full disk for six months. Nothing has actually been backed up.

I had a great backup. I also encrypted it. I lost the encryption key. I now own an expensive collection of random bytes.

The replication was real-time. The replication faithfully replicated the deletion.

"We have RAID, so we're covered." RAID is not a backup. RAID will protect your typo from itself with mathematical precision.

The backup script exited with code 0. The backup script also exited before any data was written. Code 0 means many things to many people.

User: "Can you restore the file I deleted this morning?" Me: "The backup runs nightly." User: "Why?"

Backups are about trust. Restores are about humility.

"Did anyone check the backup log?" The backup log was 14,000 lines of green, with one line of red on March 4th, repeated daily since.

The off-site backup was a USB drive in a drawer at the previous CIO's house.

"The backup is incremental." The full backup the incrementals are based on no longer exists.

I once restored a backup from 2016 because the more recent ones were all empty.

"We have point-in-time recovery." The point in time is 11 hours before the problem started. Good luck.

Restore plan: 1. Locate latest backup. 2. Pray. 3. Update resume during step 2.

The backup was checked monthly. The check was "did the file exist." It existed. It was 0 bytes.

"Why don't we restore from yesterday's backup?" Yesterday's backup is the one that overwrote Wednesday's good one.

I trust people. I just trust verified restore tests more.

"How long would a full restore take?" Nobody has ever attempted it. We estimate somewhere between four hours and four lifetimes.

The backup window is supposed to be at night. The backup window is now 26 hours long. It runs continuously.

"The vendor handles the backups." The vendor went out of business in 2022. Nobody told anyone.

I ran a test restore on a copy. It worked beautifully. The production restore did not. The test environment was missing the constraint that broke production.

Backups are like seat belts. Untested, theoretical, and you find out at the worst possible moment.

"I deleted it from my laptop. Can you restore it from the server backup?" The laptop wasn't backed up. We don't back up laptops. We have told you this. You signed a form.

The disaster recovery site was tested annually. The annual test had been skipped for three years. Nobody wrote it down.

"Why is the restore failing?" Because the backup format changed in the last upgrade. The old backups are unreadable by the new software.

The encrypted backup was decryptable by exactly one person. That person was on a hiking trip in the mountains. No signal.

I have one job at 5 a.m.: Figure out whether the backup includes the table that just got dropped.

"We have backups going back five years." The backups going back five years are on a tape format the current hardware cannot read.

Restore time on paper: 2 hours. Restore time in practice: 2 days. The difference is everyone in the company watching.

The backup script worked. The backup script wrote to /backup. /backup was a regular directory. /backup was on the same drive as the source. The drive failed.

"Did we test the failover?" We tested the failover. We did not test the failback. We are still in failover. It has been a year.

The cloud provider had backups of our backups. The cloud provider charged us $14,000 to give them back.

"The backup was successful." The backup was successful at being a backup of nothing.

Some teams do tabletop disaster recovery exercises. Our tabletop exercise is just the actual disaster, every 18 months.

I labeled the backup tapes by hand in 2014. I cannot read my own handwriting in 2024. The future is now.

"We're moving to a new backup system." The migration plan does not include verifying the old backups before retiring them.

Being responsible for backups is: Doing perfect work that nobody sees, until the day something fails and everybody learns your name at the same time.

"What does this service do?" It has been doing it since 2011. That is all anybody knows.

The deployment instructions are a Word document. The Word document references a server that was decommissioned in 2016.

"Can we rewrite it?" We tried. Twice. Both rewrites are also legacy now.

The build script is a 400-line batch file. It works. Nobody touches it.

There is a server under a desk in the finance office. It runs payroll. It has 1,400 days of uptime.

"Who wrote this code?" The blame says Steve. Steve left in 2014. The function still has Steve's TODO comment on top.

Two kinds of engineers: Those who have run a binary they cannot rebuild from source, and those about to.

"The documentation is on the wiki." The wiki was migrated. The page is gone.

I asked about the database trigger. Three people looked at the floor. One person said: do not.

"Just add it as a feature flag." The feature flag system has 600 flags. Half of them are unreferenced. The other half are load-bearing.

The application was written for Internet Explorer. The application is still in production. The customers are still happy.

"Who has access to deploy?" Steve. Steve left in 2014.

I trust new code. I just trust code that has survived ten years more.

The function is 2,400 lines long. It has one comment. The comment says: do not refactor.

"It works." The four most dangerous words to say about a legacy system right before you have to touch it.

The shell script calls a Perl script which calls a Python script which calls a stored procedure. This is the export pipeline. It runs nightly. It has never failed.

"We have unit tests." The test file has been excluded from CI since 2018.

The login page has not changed visually since 2009. The login page handles 80 percent of revenue.

"What's the architecture diagram?" It is a whiteboard photo taken in 2015. The whiteboard photo has resolution issues.

The original developer is now a VP. The original developer does not respond to Slack messages.

"How does the cron job authenticate?" A service account whose password is the founder's middle name. From 2010.

A senior engineer is somebody who has politely declined to be the owner of three different legacy systems.

"Can you migrate this off the mainframe?" "What's the budget?" "Two weeks."

The database has a column called `temp`. The column has been there for nine years. It is required.

"The system is well-tested." The test is a customer calling support when something breaks.

I read the source code. The source code references a JIRA ticket from 2013. The JIRA instance was sunset in 2019.

"Why does the report take eight hours to run?" Because in 2008 the dataset was small enough that nobody worried about it.

The application server is named `apollo`. There is no apollo2. There was supposed to be.

"What's the upgrade path?" "There isn't one."

Every team has one binary they cannot rebuild from source. It runs in production. It has been working since the previous office.

"Just add a new endpoint." The routing file is 4,000 lines and ordered by date added.

The system was written in a framework that is no longer maintained. The framework's GitHub repo has been archived. The README links to a personal blog that is now a parked domain.

I once tried to add a unit test to a legacy module. Three weeks later I had a working mock for a database that the function did not actually use.

"Just put it in a Docker container." The binary requires a specific kernel version. From 2012.

The customer-facing app is modern. The back-office tool is from a different decade. The back-office tool is where the money is.

"Can we deprecate this endpoint?" One customer integrated with it in 2014. They are the largest customer.

A new hire asked about the design choices. The design choices were not chosen. They accumulated.

"The code is self-documenting." The code documents a problem nobody can describe anymore.

The CI pipeline has a step called `unknown_step` that has been failing silently for two years. Everybody has memorized to ignore the red dot.

I asked which version of the dependency we use. The answer was: whichever version the build server happened to have cached in 2017.

"What language is this written in?" "Three languages. And a stored procedure."

The data migration script was written for the original launch. The data migration script is still in the repo. It has been run, by accident, twice.

"Why is there a hardcoded IP address?" Because DNS was unreliable in 2011 and nobody got around to changing it.

The legacy system has a single point of failure. The single point of failure is also the person who knows how to fix it.

"We don't need to refactor." The codebase is one file. It has 18,000 lines. It is named `helpers.php`.

I do not always touch legacy code. But when I do, I bill for it.

The system was supposed to be replaced in 2018. The RFP for the replacement is still in draft.

"Can you just move it to the cloud?" It depends on a hardware dongle that is plugged into a desktop tower in the supply closet.

Some people fear horror movies. I fear inheriting a Perl codebase with no tests.

Working on a legacy system is basically: Getting paid to learn the decisions of people who do not work here anymore and explain those decisions to people who will not work here next year.

"We'll clean it up later." Later was four years ago.

The TODO comment is older than the engineer reading it.

"It's a temporary fix." The temporary fix is now load-bearing.

I checked the git blame. The line was written by me. In 2019. I do not remember writing it.

"Technical debt is fine if we pay it down." We have not paid down technical debt since the company was founded.

Two kinds of teams: Those who track tech debt in a backlog, and those whose tech debt has its own backlog now.

"It's just a small workaround." The workaround has its own workaround.

The rewrite is scheduled for next quarter. It has been scheduled for next quarter since 2020.

"We'll add tests later." Later is now. There are no tests.

Every codebase has a folder called `helpers`. That folder is where dignity goes to die.

I trust new features. I just trust the bugs in old features more. At least I know their names.

"Move fast and break things." We moved fast. The things stayed broken.

The comment says: `HACK: fix this before launch`. Launch was eight years ago.

"Let's prototype it first." The prototype shipped on Tuesday.

A senior engineer is somebody who has watched a prototype become production three different ways and made peace with it.

"We can refactor in the next sprint." The next sprint has six feature tickets and zero refactor tickets, like always.

The function is doing four things. The function is named `process`. It has been like this for years.

"It's not technical debt if it works." It is. You are paying interest. You just don't see the statement.

The codebase has six different ways to handle errors. They were all introduced one sprint at a time, by different people, for different reasons.

"Just add a new column." The table has 84 columns. Forty-one of them are nullable. Twelve are nullable but populated. Three are populated only on Tuesdays.

I once wrote a one-line hotfix. The one-line hotfix is now a class.

"Why is the build slow?" Because every shortcut we took in 2018 is now a dependency.

The team agreed to allocate 20 percent of every sprint to tech debt. The team has allocated 0 percent in 18 months.

"We don't have time for tech debt." We also do not have time for the outages that the tech debt is going to cause. We just haven't scheduled those yet.

The codebase has a function called `handleSpecialCase`. It handles 14 special cases.

"What does this flag do?" It was added to fix a bug for one customer in 2017. The customer is no longer a customer. The flag is on by default.

The integration test suite passes locally. The integration test suite passes in CI. The feature does not work in production. The tests are mocking the wrong thing.

"It's a small change." The diff is 18 lines. The regression list is on its third page.

Every config file has a key called `legacy_mode`. It is set to true. It has been set to true since the option was introduced.

"Why did the deploy fail?" Because the deploy script silently assumed something that stopped being true two years ago.

I do not always document my workarounds. But when I do, the documentation is misleading.

"The previous engineer left good notes." The notes say: `see README`. The README is empty.

The codebase has three logging libraries. Nobody knows when the second and third were added. All three are required.

"We should standardize." We have tried to standardize three times. Each attempt added a new standard.

The architecture document was correct at the time of writing. The time of writing was a different product.

"How much tech debt do we have?" "How much do you have time to hear about?"

The team decided to bankrupt the legacy module and rewrite it. The rewrite is now four years old and considered legacy.

"Just delete the unused code." The unused code is referenced by a config in production. Somewhere.

There is a `deprecated_v2` folder. The code in it is still imported by three services.

"We chose this stack for velocity." We have not had velocity in three years.

I asked why the test runner has a 90-second startup time. The answer involved a fixture file from 2016 that nobody is willing to delete.

"It's idiomatic." Idiomatic in what year?

The on-call runbook has a section called `if X happens, do Y`. Nobody can remember why Y works. Y still works.

"Did anyone document the migration?" There were three migrations. The first two were undone. The third was never finished.

I read the design doc. The design doc describes a system that does not exist. The design doc is two years old.

"This was the right choice at the time." It was. It still cost us six months later.

Every team has a Slack channel called `tech-debt-discussion`. It has been quiet since Q2.

"We're going to do a tech debt sprint." The tech debt sprint got pulled in for a customer feature on day three.

The codebase has an abstraction. The abstraction has one consumer. The abstraction was written for a planned second consumer that never arrived.

"What's the latency budget?" It was 200 milliseconds. We are at 1,400. We renamed the dashboard.

The interest payment on tech debt does not show up in the budget. It shows up as new hires that are slower than the previous ones.

"We should rewrite this in Rust." The service handles 60 requests per minute. Calm down.

The CI pipeline has 14 steps. Number 9 has been failing for a year. We set it to non-blocking.

"We don't need a feature flag for this." We needed a feature flag for that.

The code review comment says: `we should refactor this`. The review is from 2019. The code is unchanged.

A new hire asked why we use the pattern. Nobody could explain it. We still use the pattern.

"We can address that in the next major version." There has never been a major version.

I do not fear horror movies. I fear opening a file named `utils.js` that is older than my career.

Some people read books. I read git blame and try to forgive my younger self.

Paying down tech debt is basically: Doing the work the previous version of the team should have done, with less context than they had, on a deadline they did not have.