❯ Why Monitoring Matters More Than Fixing: Catch Drift Early

2026-09-09 · Ops, Founder ·4 min read

by Jonas Reyes — the builder's desk, Side Quest Studios September 25, 2026

Nobody calls you at 3am because a disk filled up. They call you because a disk filled up six weeks ago, quietly, and now the database won't accept writes and the whole order pipeline is down and somebody's asking whether we can "just restart it." We can. It'll be back in ninety seconds. And it'll be down again next Tuesday, at 3am, for the same reason nobody noticed the first time.

That's the whole game. Fixing is the easy part. Fixing is systemctl restart, it's a config rollback, it's a git revert and a redeploy. Anybody with root and a heartbeat can fix a thing that's already broken. What separates operators who sleep from operators who don't is that one group catches the drift before it becomes an outage, and the other group finds out from a support ticket.

Drift is the real enemy, and it doesn't announce itself

An outage is loud. Drift is silent. Drift is your disk going from 40% to 61% over three weeks because a log rotation rule stopped matching after somebody renamed a service. Drift is p99 latency creeping from 90ms to 240ms because a connection pool grew a slow leak. Drift is a certificate that expires in eleven days and a cron job that's been failing silently since the last base image update.

None of these are emergencies. All of them become emergencies if you wait.

The reason most teams wait is that they've been sold monitoring as alerting. Alerting is the fire alarm. Monitoring is the smoke detector in the hallway and the one in the attic and the one you forgot to test. If your entire observability strategy is "page me when the site is down," you don't have a monitoring strategy. You have a very expensive way to learn things after your customers already know.

What I actually run, and why it's boring on purpose

I run Prometheus and node_exporter. That's it for the base layer. Not because it's fashionable — it's a decade old at this point — but because I can read the config, grep the metrics, and it doesn't phone home. Every box gets node_exporter on :9100, every app gets a /metrics endpoint, and Prometheus scrapes both every fifteen seconds.

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['web1:9100', 'web2:9100', 'db1:9100']
  - job_name: app
    metrics_path: /metrics
    static_configs:
      - targets: ['web1:8000', 'web2:8000']

That's the whole file. You can hand it to an intern.

Then Alertmanager for routing, and — this is the part people skip — rules that fire on trends, not thresholds. A disk at 90% is already too late. I want to know when a disk grows more than 15% in 24 hours, because that's drift, and drift is a decision I get to make on a Tuesday afternoon instead of a Tuesday at 3am.

- alert: DiskGrowthAnomaly
  expr: |
    (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
     / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}) < 0.25
    and
    deriv(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h]) < -10485760
  for: 30m
  annotations:
    summary: "{{ $labels.instance }} {{ $labels.mountpoint }} shrinking fast"

Notice what that does. It fires when you're at 25% free and losing more than 10MB/s worth of ground per six-hour window. That's not "disk is full." That's "disk will be full in a known number of hours, here's your runway." Runway you can act on. A threshold you can only react to.

The thing that actually surprised me

I spent years building dashboards nobody looked at. Grafana, beautiful, twelve panels, dark theme, gorgeous. Nobody opened it. Not once, not during an incident.

What changed everything was a cron job. One Python script, twenty lines, runs every morning at 07:00, curls a handful of Prometheus queries, and emails me a plain-text summary. Not a graph. Text.

web1  disk /        71% (+2.1%/24h)   load 0.4   mem 61%
web1  disk /var     88% (+4.7%/24h)   <-- watch this
db1   p99 latency   240ms (was 112ms last week)  <-- why?
db1   replication   0.8s behind
cert  *.example.com expires in 22 days

That's it. That's the whole monitoring system that's caught more real problems than every dashboard I've ever built. Because I read it. Every morning. With coffee. And when I see a number with an arrow next to it, I go look.

You don't have to use my script. But write something that puts the trend in front of a human on a schedule, in text, before it becomes a graph somebody has to remember to open. The best observability tool is the one a tired person actually reads.

Monitoring is cheaper than fixing, and it's not close

Let me do the math I've done on real incidents. An outage at a mid-size SaaS company — order processing down, say — runs you somewhere between four and forty thousand dollars an hour depending on the vertical, plus the customer trust tax that never shows up on the invoice. The fix, once you're in it, is usually trivial. The cost is entirely in the detection lag and the scramble.

Now price out the alternative. Prometheus, node_exporter, Alertmanager, one Python script, one person who actually reads the email. That's maybe six hours of setup, then ten minutes a day. I've run this stack on fleets of forty boxes and fleets of four thousand. The math doesn't change. Catching drift early is the single highest-leverage thing you can do with a systems budget, and it's almost always cheaper than the alternative — because the alternative is paying an engineer to do archaeology at 3am.

Look at what's happening in the wider industry right now and the same logic scales up. The FAA is dropping $875 million on AI to help air traffic controllers do their jobs (TechCrunch) — that's a monitoring and decision-support investment, not a "fix the plane" investment. Meanwhile, the oversight problem with autonomous AI agents is the same problem in a new costume: agents act faster than humans can review (TechCrunch), which means the only way to keep them honest is to instrument them. Not to fix them after they do something dumb. To watch them go off the rails in slow motion, on a chart, before the invoice lands.

Same problem. Same fix. Same order of operations.

The rule I use

If a failure mode can be detected more than an hour before it becomes a customer-visible problem, it should be. That's the whole philosophy. Anything faster than that is incident response and you're already behind. Anything slower than that is negligence with extra steps.

So: instrument first. Alert on trends. Read the digest. Fix on your schedule, not the pager's.

The people who get this right aren't smarter than you. They just decided that monitoring was worth more than fixing, and they stopped treating it as the boring part. It's not the boring part. It's the whole job.

References

AI-assisted, curated for Side Quest Studios.