❯ The Maintenance Routine Every Self-Hosted Setup Needs

The first time a production box bit me, it wasn't a dramatic crash. It was a Tuesday. Disk usage was at 94%, the Postgres autovacuum had been silently failing for eleven days, and a cron job I'd written eight months earlier was quietly spawning zombie processes every four hours. Nothing was on fire. Everything was just slightly wrong, in a way that would have become a fire in another week.

That's the thing about self-hosted infrastructure. It doesn't usually die from a single dramatic event. It dies from the slow accumulation of small neglects that nobody was watching because everyone assumed someone else was watching. If you're running anything yourself — a Postgres cluster, a couple of containers, a GPU box for inference, whatever — you need a maintenance routine. Not a monitoring dashboard you check when something feels off. A routine. Scheduled, boring, written down, and executed whether or not you feel like it.

Here's the one I actually run. It's built from scars, not from a blog post about "observability best practices."

The Sunday audit: thirty minutes, no exceptions

Every Sunday morning, before I've had enough coffee to be optimistic about anything, I spend thirty minutes walking a fixed list. The point isn't to fix things. The point is to notice them. Fixing happens later, during the week, prioritized against everything else.

The list is short and it's the same every week:

• Disk usage on every volume. Not "is it full" — I want to see the trend. If a volume grew 8% this week without a corresponding traffic increase, that's a leak. Log rotation failing, a temp directory filling up, a database table that's been growing since someone forgot to add a retention policy.
• Failed systemd units. systemctl --failed on every box. I've found more problems this way than any other single command. A backup service that's been "active (failed)" for three weeks is a backup service that isn't backing anything up.
• Cron and timer execution logs. Did every scheduled job actually run? Did it exit zero? Did it take the usual amount of time, or did it suddenly take four times as long, which usually means it's about to stop working entirely?
• Certificate expiry. Anything under thirty days gets flagged. Anything under fourteen gets fixed that day. Let's Encrypt renewals fail silently more often than people admit, usually because of a DNS change or a rate limit nobody noticed.
• Backup verification. Not "did the backup job run" — did the backup actually restore? I keep one small canary database that gets restored from backup every Sunday into a throwaway container, and I query it. If that fails, nothing else on the list matters.
• Error rate on the previous week's logs. Just a grep for the top ten error strings. New error strings are interesting. Old error strings that suddenly got louder are more interesting.

Thirty minutes is enough if you know what you're looking at. It's not enough if you're exploring. That's why the list is fixed. Exploration is for when something's wrong.

The monthly deep pass

Once a month, I do the things that are too slow or too disruptive for a weekly cadence.

Patch and reboot. Not "patch eventually." Patch this week, on a schedule, and reboot. The number of self-hosted setups running a kernel from fourteen months ago because "rebooting is scary" is staggering. If rebooting is scary, that's the actual problem — you don't have a tested recovery path. Fix that first, then reboot.

Rotate credentials. Database passwords, API keys, anything with a long expiry. Yes, this is annoying. Yes, it will break something the first few times because some service has the old password hardcoded in a config file you forgot about. That breakage is information. It tells you where your secrets management is fictional.

Review what's actually running. docker ps, systemctl list-units, whatever your stack uses. I guarantee you'll find at least one service you forgot was still running. Last month I found a Redis instance that had been up for 400 days serving absolutely nothing, just sitting there consuming 2GB of RAM because I'd moved a workload to a different machine and never cleaned up.

Look at the bills. Cloud egress, storage, GPU hours. The monthly bill is a maintenance signal. A 20% jump without a 20% jump in usage means something is leaking — usually a log shipping job gone rogue or a backup that's now storing ten times the data it used to.

Read one thing about your stack. Not a tutorial. Something about a known issue, a CVE in a dependency, a changelog for a major version you're three releases behind on. This is how you find out that the thing you're running has a bug that will bite you in six months.

The quarterly "what if" test

This is the part almost nobody does, and it's the part that saves you.

Once a quarter, I pick one failure mode and simulate it. Not in a lab — in the actual staging environment, or on a canary, or in a throwaway copy of production. The question is always the same: if this broke right now, what would I actually do?

• The primary database is gone. Not corrupted — gone. Do I have a restore that works, and how long does it take?
• The GPU box dies. Can I fail over to a rented instance, and how long until inference is back?
• A dependency gets compromised in a supply chain attack. Do I know what version I'm running, and can I pin to a known-good one in under an hour?
• The hosting provider has a multi-region outage. What still works, and what's the customer-visible impact?

You don't need to fix everything you find. You need to know. The gap between "I think we'd be fine" and "I've actually tested it" is where companies die.

Why this matters more now, not less

There's a version of the AI-hype story where self-hosting becomes irrelevant — everything runs on someone else's infrastructure, and you just wire up APIs. That's not what I see. The operators I talk to are running more of their own stack, not less, because inference costs at scale are brutal, because data residency matters, and because depending on a single vendor's uptime for your core product is a bet most founders eventually regret.

Which is exactly why the maintenance routine matters more. When you self-host, you own the failure modes. All of them. The AI researcher who resigned this week over existential risk got the headlines, but the risk most operators actually face is much more mundane: a disk filling up at 3am because nobody was watching the trend Fast Company. The boring risk. The one a Sunday audit catches.

The same pattern shows up everywhere. The DOE is pushing trustworthy AI and best practices for scaling it — and a big part of trustworthiness is just operational discipline AI Trends. The hiring-AI bias conversation is the same shape: the failures aren't exotic, they're the predictable result of nobody checking the data on a schedule AI Trends. Maintenance is unglamorous, and it's where the actual outcomes get decided.

Write it down or it doesn't exist

The single most useful artifact I have is a plain text file called runbook.md. It contains, in order: what to check weekly, what to check monthly, what to test quarterly, and what to do when each of the top ten failure modes actually happens. It's not clever. It's not automated. It's just written down.

The reason this matters: at 3am, when something is on fire, you will not remember the exact command to restore the Postgres cluster from a WAL archive. You will not remember which env var controls the failover. You will not remember that the backup bucket is in a different region than the primary. The runbook remembers for you. It also means that when you're on vacation, someone else can follow it — which is the only real test of whether your maintenance routine is a routine or a personal habit.

If you're running self-hosted infrastructure and you don't have a weekly audit, a monthly deep pass, a quarterly failure test, and a written runbook, you don't have a system. You have a collection of services that happen to be running right now. The difference between those two things is measured in weeks, and the bill comes due whether you're ready or not.

We build and run these systems for a living at Claw Way — if you want the runbook without the scars, that's what we do.

References

• rsync documentation - https://rsync.samba.org/documentation.html
• systemd documentation - https://man7.org/linux/man-pages/man1/systemctl.1.html
• cron(8) manual - https://man7.org/linux/man-pages/man8/cron.8.html

by Jonas Reyes — the builder's desk, Side Quest Studios
AI-assisted, curated for Side Quest Studios.