❯ The 2 A.M. Habit That Stops Your Server From Filling Up

2026-09-09 · Ops, Founder ·3 min read

by Jonas Reyes — the builder's desk, Side Quest Studios September 10, 2026

The disk fills at 2 a.m., always. Here's the habit that stops it.

It is never a mystery: something has been writing to your server without a limit, and nobody was watching the number until the database refused to start. The 2 a.m. version of you deserves better than that.

What makes disk-full a special kind of outage is that it is self-inflicted and slow. Nobody ships you a "your logs are eating the disk" bug report. The number just climbs, quietly, for weeks — and then everything that writes to disk fails at once. Postgres won't write WAL. Docker won't start a container. Your app returns 500s on every request that touches a session.

Why it is almost never "the database"

The usual culprits are boring: container logs with no rotation, a build cache that grows forever, an analytics file that appends and never prunes. Container logs are the classic — Docker's default logging driver does no rotation at all, so a chatty service can quietly eat the whole disk.

In order of how often I have actually seen them fill a host:

Container logs. A service in a crash loop writes its stack trace a few thousand times an hour. This is the single most common cause, and it accelerates exactly when something is already wrong.
Docker build cache and dangling images. Every docker build leaves layers behind. Nobody deletes them, because nothing prompts you to.
An append-only file. An access log, an event stream, a SQLite database used as a queue. It only ever grows.
Deleted-but-open files. The file is gone from ls, but a process still holds the descriptor, so the space is not released. This one is genuinely confusing — df says full while du insists there is room.
Backups written locally. A nightly dump that lands on the same disk it is protecting is not a backup, it is a second copy of the problem.

The habit: cap first, then prune

Order matters. Pruning a disk you have not capped is mopping a floor with the tap still running. Two settings and one schedule cover most of it.

Cap container logs. Tell Docker to rotate at 10 MB with three files kept — see the logging driver docs. Set this in daemon.json so it is the default for everything, not something you remember per service.
Cap the system journal. Set a maximum size and 30-day retention in journald.conf (journald.conf(5)). Without a cap, the journal is the second log that eats the disk.
Cap your own application logs. If your app writes its own file, it needs its own rotation. logrotate(8) does this with a config that reads like a sentence: rotate weekly, keep four, compress. Do not skip the copytruncate decision — programs that hold the file open need it.
Prune on a schedule. Unused build cache and dangling images are the biggest single consumer on most hosts, and they are safe to remove. Put it on a timer instead of relying on memory.

How to find the culprit before you prune anything

Never start deleting on a full disk. Find the thing that is actually growing first, or you will free 20 GB and watch it refill in a week.

du -x -h --max-depth=1 / | sort -h — the top-level map. -x keeps it on one filesystem so mounted volumes do not confuse the answer.
docker system df — splits Docker's usage into images, containers, volumes and build cache. The build-cache line is usually the surprise.
lsof +L1 — deleted-but-open files. If this returns a multi-gigabyte row, restart that process and the space comes back instantly.

Try it once

bash bash (80x24)

df -h /\ndocker system df

Two numbers. If docker system df shows a build cache larger than a few gigabytes, you have found something to reclaim today — and the caps above stop it coming back.

The real habit is the alert, not the cleanup

Everything above buys you months. What buys you the 2 a.m. is a threshold that fires before the disk is full. Watch free space as a percentage and alert at 80%, not at 95% — by 95% the write failures have already started, and you are now debugging under a clock.

Better still, alert on the trend rather than the level. A disk at 60% and climbing 2% a day is a problem you have a month to fix; a disk at 60% that is flat is not a problem at all. Level tells you where you are. Slope tells you when you arrive.

What not to prune

Two things look like waste and are not. Named volumes can hold a database, and docker system prune with the wrong flags will happily remove something you needed. And any file a process currently holds open should be rotated, not deleted — deleting it frees nothing until the process lets go.

None of this is glamorous. It is the difference between waking up to a working server and waking up to a disk at 100%.

References

AI-assisted, curated for Side Quest Studios.