❯ Disk full panic: the habit that prevents it
by Jonas Reyes — the builder's desk, Side Quest Studios September 25, 2026
The disk filled at 2:47am. Not gradually — the way it always happens, which is to say it was fine when you went to bed and a smoking crater when you woke up. Postgres stopped accepting writes. The log shipper filled its buffer and started dropping lines. Then the whole VM went read-only and every service on it died in a cascade that took forty minutes to untangle and cost you a morning you'll never get back.
I've watched this movie maybe thirty times. The disk was never the problem. The problem was that nobody looked at it until the alarm went off, and by then you're not debugging, you're firefighting. There's a habit that prevents the whole thing, it costs about twenty minutes to set up, and almost nobody does it. Let's fix that.
The panic isn't the disk, it's the surprise
A disk that fills predictably is an inconvenience. A disk that fills at 3am on a Saturday is an outage. The difference between those two worlds is not monitoring — it's forecasting. Most people monitor disk usage as a percentage and alert at 90%. That's fine until a log file decides to write 40GB in an hour and blows straight past your threshold between two scrape intervals. You go from 38% to dead with no warning, and your alert fires at the same moment your database does.
The habit is this: you don't alert on usage. You alert on time-to-full. Usage tells you where you are. Time-to-full tells you where you're going, and that's the only number that matters at 3am.
The one-liner that changes your week
Before the alerting, before the dashboards, just look at your own machine. Run this:
That sorts every mounted filesystem by percentage used, largest first. On a healthy box the top line is your root partition sitting at 60-something percent and you move on. On a box that's about to hurt you, something in /var or /var/log is at 94% and has been quietly climbing for a week while you weren't looking.
Now the useful version. If you want to know what is eating the space, du is your friend but it's slow on big trees. Use ncdu instead — it's the same idea, interactive, and it'll walk a 2TB volume in under a minute:
The -x keeps it on one filesystem, which is what you actually want. You will find, roughly always, one of four things: a log file nobody rotated, a container image layer cache, a /tmp directory full of something that should have been cleaned up in 2023, or a database WAL directory that's grown because replication broke three weeks ago and nobody noticed.
That last one has bitten me personally. Postgres holds WAL segments until the replica confirms it's received them. Replica dies, replication slot stays open, WAL accumulates, disk fills. The fix on the replica side is fixing replication. The fix on the primary side, right now, is SELECT pg_drop_replication_slot('slot_name'); — but only after you understand why it broke, or you'll be back here next week.
Forecast it, don't just watch it
Here's where most monitoring setups fall down. Prometheus has predict_linear, and it is criminally underused. The idea: take the current disk usage trend over the last six hours, extrapolate it forward, and alert if that line hits 100% within the next twenty-four hours.
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24 * 3600) < 0
Read that as: "based on the last six hours of growth, will this filesystem run out of space in the next day?" If yes, page someone. That's it. That's the whole trick.
What this buys you is lead time. A percentage threshold fires when you're already in trouble. A forecast fires when you have hours to act — when you can drop a rotated log, extend a volume, or clear a cache during business hours instead of at 3am. You can even set a softer version at 48 hours and send it to Slack instead of PagerDuty, so the on-call engineer wakes up to a solved problem instead of an outage.
The one caveat: forecast alerts are noisy on filesystems with spiky, bursty growth. A build server that fills 80% during a CI run and drops back down afterward will trip this constantly. So either exclude those mounts or use a longer window. Six hours is a reasonable default; tune from there.
The boring infrastructure that makes it not matter
Alerts are a safety net, not a strategy. The strategy is that no single disk should be able to take you down. A few things I do on every box, in order of how much sleep they buy me:
Log rotation, actually configured. /etc/logrotate.d/ is where the default configs live and where you should be adding your own. The distro default rotates weekly and keeps four. That's fine for syslog and useless for an application that writes 5GB a day. Set daily, set rotate 14, set compress, set maxsize 500M. The maxsize directive is the one people forget — it forces a rotation even if the time window hasn't elapsed, which is exactly what you want when a service decides to have a bad day.
Separate volumes for the things that grow. Root filesystem should be boring and stable. Logs, database data, and container storage belong on their own volumes. When /var/log fills, you want the only thing that breaks to be logging, not your entire box going read-only. This is a two-minute change in your provisioning templates and it has saved me more outages than anything else on this list.
A cron job that cleans up after itself. Not a fancy garbage collector — a find command with a -mtime flag:
Run it nightly. It costs nothing and it catches the slow leaks that no alert will ever see coming because the growth is too gradual to trigger a threshold.
Why this matters more now, not less
Here's the thing that's been rattling around my head. Everyone's building elaborate AI systems — world-model companies sitting on piles of cash and refusing to say what they're actually doing, pipelines that consume enormous amounts of storage for training data and checkpoints and logs nobody reads. People building AI workflows to save time and somehow working more, which means more services, more containers, more stuff writing to disk at 3am.
Meanwhile the fundamentals haven't changed. A disk fills when something writes more than you planned for, and the only defense is knowing before it happens. The hype cycle doesn't care about your filesystem. The filesystem doesn't care about your hype cycle. But when the disk fills, everything stops, and it stops for everyone.
Do this today
Twenty minutes. That's the whole ask.
Run df -h on every server you own and look at the output with actual attention. Install ncdu and walk the worst offender. Find the one log file that isn't rotating and fix it. Add a predict_linear alert to whatever monitoring you're running — Prometheus, Grafana, whatever, they all support it. Set up the nightly cleanup cron.
You will not think about this again for six months. And then one day, at 2:47am, nothing will happen. No page. No outage. Because you saw it coming three days earlier, cleared the log, and went back to bed.
That's the win. Not the clever fix. The boring habit that means you never needed one.
References
- "Query functions: predict_linear()", Prometheus Documentation (2026)
- "pg_replication_slots", PostgreSQL 18 Documentation (2026)
- "logrotate(8) — maxsize directive", Linux man-pages, man7.org (2026)
AI-assisted, curated for Side Quest Studios.