❯ We Audited Our Custom LLM Layer - And Found We'd Already Adopted the Standard One

2026-10-13 ·4 min read

by the desk — Terminal King, Side Quest Studios October 13, 2026

We run a platform where each tenant has an AI agent that reads mail, drafts replies, writes articles and queries a knowledge base. The LLM plumbing for all of that lives in one module. The question that started this audit was uncomfortable: are we reinventing the wheel?

It is a fair question. The module had grown a model cascade, a provider fallback chain, a per-surface key scheme and a spend ledger. Every one of those is something an LLM gateway is supposed to do for you.

What we actually found

We expected to find a hand-rolled gateway that should be deleted. We found something more interesting: the module already routes through a gateway as its primary path.

if LITELLM_BASE_URL configured:
    route through the proxy, using the per-surface virtual key
on ANY proxy failure:
    fall back to the hand-rolled cascade

So the architecture was already correct — the standard tool primary, the custom code a fallback for when it is unavailable. That is not reinvention. That is a deliberate degradation path.

The lesson is that “are we reinventing the wheel?” is the wrong first question. The right one is: which layer is primary, and which is fallback? We could not answer that from the code comments. We answered it by reading the function body, and by measuring which callers set which environment variables.

The part that genuinely was missing: caching

Our audit found zero references to caching anywhere in the module.

This matters more than any routing code. Two effects dominate LLM cost, and both are caching effects:

  • Provider-side prompt caching. Published rates: a cache read costs 0.1x the base input rate; a cache write costs 1.25x, with a 5-minute TTL. A reused prefix is therefore roughly a 90% discount on that portion of the input.
  • Gateway routing and caching. A live production deployment reports 51% cost savings with an auto-router, with roughly 40% from day one as the stated expectation.

Both of those are larger than anything a hand-written model cascade can deliver. We had built the cascade and skipped the cache.

The second gap: coverage

We counted the callers: 16 files import the LLM call helper. Four set a budget surface.

That means twelve callers were making LLM calls with no per-surface budget attribution. They would still work — but the spend would land in the wrong bucket, or no bucket. A budget system that only 25% of your callers feed is not a budget system. It is a partially-instrumented one that produces confident, wrong numbers.

This is the failure mode to watch for: the mechanism exists, so it looks done. The question is always coverage, never existence.

What we are keeping

Not everything custom is waste. The same module also holds:

  • a delivery-time model — when to send so mail looks human, not machine-scheduled
  • a sensitive-content scrubber, applied before anything reaches a model
  • an HTML prettifier, a spam scorer, a knowledge-base distiller

None of that belongs in a gateway. A gateway routes and accounts for model calls. It has no opinion about what time a newsletter should go out. These were the correct things to build, and they are the reason the module has sixteen callers.

The rule we drew: a gateway owns transport, budgets, caching and fallback. Application code owns what the call means. Anything true of all LLM traffic goes in the gateway. Anything true of your traffic stays yours.

The honest counter-argument

A gateway is not automatically the answer, and it would be dishonest to present it as one. There is a well-argued 2026 write-up from an engineer who kept their own gateway after evaluating two alternatives and filing issues against them.

Two points from that side that we take seriously:

  1. Prefix-aware routing is not in the stock gateway, open or paid. If your workload depends on routing to whichever replica already holds a warm prefix, a stock gateway will not do it.
  2. The migration is not free. Pointing sixteen callers at a new transport, proving budgets still attribute correctly, and verifying the fallback path still fires is real work with real regression risk.

Our position: adopt the gateway for the parts it does well — transport, key management, budgets, caching — and keep your own code for the parts that are yours. Replacing everything with a gateway is as much a mistake as replacing nothing.

What we are changing

  1. Turn on gateway caching. The measured lever is 40–51% from routing, and about 90% on cached prefixes. We built the cascade and skipped this.
  2. Set the budget surface in all sixteen callers, not four. Until then the spend numbers are confidently wrong.
  3. Repoint the fallback at our primary provider rather than the one we are retiring, so the degradation path does not itself depend on a deprecated dependency.

The transferable lesson

If you are building LLM plumbing, ask three questions in this order:

  1. Which layer is primary? Read the function body, not the comment. The comment describes the intended design; the body describes the one that runs.
  2. Where is the cache? If the answer is “nowhere”, stop optimising anything else. Caching is the largest and cheapest lever available, and it is a configuration change.
  3. What is the coverage? A mechanism 25% of callers use is not a mechanism. It is a false signal that survives every audit checking only for its existence.

Build the parts that are specific to your problem. Rent the parts that are true of everyone’s.

References

  1. LiteLLM — Caching. docs.litellm.ai/docs/proxy/caching
  2. LiteLLM — 51% Cost Savings Reported From a Live Production Deployment. docs.litellm.ai/blog/auto-router-production-savings
  3. LiteLLM — Prompt Caching (published cache read/write rates). docs.litellm.ai/docs/completion/prompt_caching
  4. Don’t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks. arXiv:2601.06007. arxiv.org/abs/2601.06007
  5. vCache: Verified Semantic Prompt Caching. arXiv:2502.03771. arxiv.org/abs/2502.03771
  6. ndthuan — Why I Kept My Own LLM Gateway over LiteLLM and 9Router. ndthuan.com/own-llm-gateway-over-litellm-and-9router

AI-assisted, curated for Side Quest Studios.

© 2026 Terminal King