If you run coding agents day in and day out, you eventually notice something odd. Some weeks the bill looks almost boring. Then a few sharp spikes appear — usually on days when someone reopened an old chat on another machine “just for one question.” That pattern is not mysticism. It is prompt cache hit-rate.
This article is a practical field note from running Grok-style coding agents on real infrastructure work: WordPress multisite, APIs, ops. The same economics apply to any assistant that bills input tokens and offers a cached-prefix path.
What you are actually paying for
Every model turn is roughly two piles of tokens:
- Prefix — system instructions, tool definitions, skills, project rules, and earlier turns the provider can treat as a stable block.
- Suffix — your latest message, fresh tool results, new file dumps.
When the provider has recently seen an identical prefix, most of that block can be billed as cache read instead of full-price input. When the prefix changes — or the cache has gone cold — you pay for the whole prompt again.
Low cost is therefore not “the model is free.” It is cache_read ≫ uncached input on turn two and beyond. Hit-rate is the ratio that makes long agent work affordable.
The unit of caching is the session, not the topic label
People sometimes hear “same discipline” and imagine a magic bucket called APIs or WordPress. Caching does not work on labels. It works on byte-stable continuity:
- same conversation lineage (resume the same session),
- same model,
- same working directory and project rules,
- same tool and skill surface for that run.
Stay inside one long-lived session while you implement DutchBud API changes and then PolitiCap API changes, and you still benefit. The system prompt, tools, and earlier turns remain a warm prefix; only the new work sits in the suffix. Start a brand-new session for each microservice because “both are APIs,” and you throw away that warmth every time.
Why old sessions on another workstation hurt
Resuming a cold, multi-day chat from a laptop often coincides with hit-rate cliffs. Typical reasons:
- Different machine and directory — another checkout, another rules tree, another session store. Different prefix.
- Cold provider cache — even the “same” history may no longer be hot after idle time; you re-pay to warm it.
- Huge fossil history — a week-old mega-thread is expensive to rehydrate if cache misses, and noisy for the agent even if it hits.
The fix is boring: name a few durable sessions per workstream, resume those on the host where that work actually lives, and use a fresh session when the old one is no longer the job.
What keeps the prefix stable
| Do | Why it helps |
|---|---|
| One long session per workstream | Prefix grows; later turns are mostly cache read. |
| Resume by session id or clear title | Avoids accidental cold zombies. |
| Same host and project directory | Rules and session grouping stay aligned. |
| Same model for the life of the thread | Model switches reset the cache namespace. |
| Stable skills / MCP / project rules | Churn rewrites the prefix. |
| Short follow-ups in-thread | Tiny suffix, fat cached prefix. |
What busts the cache (and the budget)
- New session for every small ask (especially headless defaults without resume).
- Model hopping mid-thread.
- Enabling or disabling MCP servers and skills every hour.
- Pasting entire files every turn when a path would do — the novelty lands in the expensive suffix.
- Spawning subagents for work that fits in one grep — each child is its own context and often a cold prefix.
- Forking sessions casually when you only needed a side question.
Compaction is not the enemy — session hopping is
Long threads eventually hit the context window. Compaction shrinks history so the agent can keep working. That does change the prefix, so the cache has to warm again. That is still usually cheaper than abandoning the job for a random old session on another laptop, or starting over without the decisions you already paid to establish.
Use compaction when the thread is long but still the same job. Use a new session when the topic is truly new. Do not compact every turn “to save money”; you only force repeated re-warms.
A simple playbook for teams
- Name sessions — e.g. fleet-wordpress, bank-api, content — so people resume the hot line of work.
- Pin the workplace — do infrastructure coding where the tree of record lives; keep the laptop for light or personal threads.
- Automations must resume — headless jobs should pass a stable session id, not open a cold session on every cron tick.
- Measure — watch uncached input versus cache read. After a few turns in a healthy thread, cache read should dominate.
- Brief the agent with continuity — “continue,” “same PR,” “next endpoint” beats pasting the whole design doc again.
How this shows up in real work
On a family of products — compute, wallets, marketplaces, civic tools — a single afternoon often crosses repositories that still share one operational story. Implementing an API change on one service and then a related change on another is exactly the pattern continuous sessions reward: shared tooling and prior context stay cached; only the new diffs and tool traces are new.
Conversely, “I opened last month’s chat about ads because I was already in the TUI” is how you buy a full-price prefix for a two-line question.
Pricing reality check: Kimi K3 vs Grok 4.5 at high cache hit-rate
List prices only tell half the story. For agentic coding, most of a long session is repeated prefix — system prompt, tools, project rules, and prior turns. Both Moonshot’s Kimi K3 host and xAI’s Grok 4.5 sell that reuse cheaply as cache-read tokens. The interesting number is the effective input price once your hit-rate is high.
Figures below are public list rates as of mid-2026 (USD per 1M tokens). Always confirm live console docs; promotions and long-context tiers move. Short-context Grok 4.5 rates are used here (<200k prompt).
| Model (hosted API) | Input miss | Cache hit / cached input | Output | Context |
|---|---|---|---|---|
| Kimi K3 (Moonshot host) | $3.00 | $0.30 | $15.00 | ~1M |
| Grok 4.5 (xAI, short context) | $2.00 | $0.30 | $6.00 | 500k (long-context tier is higher) |
Moonshot has publicly talked about >90% cache hit-rate on coding/programming workloads with their serving stack when prefixes stay stable. That is exactly the long-session discipline described above. At a 90% hit-rate, blended input cost per million tokens is:
- Kimi K3: 0.9 × $0.30 + 0.1 × $3.00 = $0.57 / 1M input tokens
- Grok 4.5: 0.9 × $0.30 + 0.1 × $2.00 = $0.48 / 1M input tokens
So at the same high hit-rate, Grok 4.5’s effective input is slightly cheaper on this list sheet, and its output is far cheaper ($6 vs $15 per 1M). Agents that stream a lot of tool calls and prose pay output hard — that gap often dominates the bill more than a few cents on blended input.
If hit-rate collapses (new session every prompt, cold zombie chat, prefix thrash), both models snap back toward full miss prices: K3 at $3 and Grok at $2 on input, plus full output. The operational lesson is the same on either host: protect the prefix, measure cache_read, and treat 90% hit-rate as a target you engineer for — not a free gift from the price table alone.
K3’s larger context window can still win specific jobs (huge single-shot repos). Grok 4.5’s combination of lower miss input, matched cache-hit rate, and much lower output is why a well-run long Grok coding session feels “relatively very cheap” even when the work is heavy — provided you stay in one warm session.
Bottom line
Same subject, longer chunks, same living session is how you keep AI coding costs flat. The discipline is not mystical thrift. It is protecting the cache key: session, model, directory, and rules. Put novelty in the latest message. Leave the prefix alone.
Do that consistently and the expensive spikes become rare — usually a cold session, a wrong host, or a one-off that should have been a new short thread instead of a fossil.
3DN runs coding agents as part of how we build and operate software. The mechanisms above are general; product names and UIs differ by vendor, but prefix caching economics do not.
Leave a Reply