❯ 🔍 Watch the Drift, Not the Crash
by Jonas Reyes — the builder's desk, Side Quest Studios September 25, 2026
The invoice draft that used to be crisp now has a comma splice. The support bot that was polite last month now sounds like a used-car salesman on a bad day. Nothing crashed. Nothing errored. It just drifted — and you won't get paged when it happens.
The one sentence: your AI agent won't break loudly; it will slide quietly. Catch the drift on a weekly output review, not after a client complains.
Why drift beats crashes
Fixing a broken pipeline is easy. Catching a slow slide is not. A crash hands you a stack trace, a 500, a red panel. Drift hands you nothing — just output that is ten percent worse each week while every health check stays green.
The mechanism is boring, and that is exactly why it slips through. An agent's output is a function of four moving parts: the model revision behind the endpoint, your system prompt, the context you feed it, and the sampling settings. Any one of them can change without a deploy on your side. The provider ships a new revision under the same model name. A prompt you edited three weeks ago quietly grew four more rules and now argues with itself. A retrieval step starts pulling slightly older documents. None of these throw an exception. They only tilt the output.
Why the weekly habit beats a monitor: drift is a relative change, so the sensitive comparison is this week against last week, not this week against an abstract spec. You are looking for a delta, and a delta needs two data points close together.
What drift actually looks like
Same facts, different posture. Last month a summary returned:
Refund issued for order #4471. Funds post in 3–5 business days.
This month the same input returns:
Absolutely — we've taken care of your refund for order #4471, you're all set!
Nothing is factually wrong. The number, the timing, the intent all match. But the register moved from registrar to cheerleader, and register is what clients notice. Other common tells: the hedge count creeps up ("it's important to note that…"), citation links start 404ing, and a defined schema grows a stray sentence before the JSON. Each is a one-line observation; together they are a trend.
The counter-case worth naming: not every delta is decay. Sometimes a model revision genuinely improves a task, and your old baseline is the stale thing. That is why you write down what "normal" looks like — without a recorded baseline you cannot tell improvement from rot, and you will "fix" a prompt back to worse.
The pattern: a 10-minute Friday audit
- Schedule it. Same day every week, 15 minutes max, as a calendar block. A recurring block is the whole mechanism; motivation will not survive a busy Friday.
- Pull five real outputs from the week — not the demo prompt, the real traffic. Read them the way a client would, not the way the dashboard does.
- Score one thing you can count. "Percent of responses that needed an edit," or a 1–5 tone rating. One number, written down, is enough to see a slope.
- Adjust the system prompt when the trend bends, not when it breaks. Small prompt edits are cheap; rebuilding from scratch after a client thread goes sideways is not.
What this does not do
A weekly read won't catch a confident factual error that sounds fluent, and it won't replace a real evaluation harness for high-stakes output. It is the cheap, habit-level net — the thing that flags "getting worse" weeks before it becomes "a client's weekend." Pair it with structured evals when the stakes justify the cost; use the Friday read everywhere else.
Try it once
Pick one agent you run, set a Friday reminder, and read five of its real outputs this week with a skeptic's eye. Write down one number. Ten minutes. If you don't find drift yet, you've just calibrated what normal looks like.
References
OpenAI — Evals guide
Confident AI — quickstart (LLM evaluation)
AI-assisted, curated for Side Quest Studios.