LLM Inference Cost and Latency Optimization
On this page
Learning Objective
Build a cost and latency model for an LLM feature before it ships, explain why the prefill and decode phases have fundamentally different performance characteristics, and apply model routing, output bounding, and asynchronous execution to bring an unaffordable design into budget.
Concept Explanation
The single most useful mental model for serving performance is that generation has two phases with opposite bottlenecks. Prefill processes the entire input prompt in one pass, and because every token can be computed in parallel, it saturates the GPU's arithmetic units and is compute-bound. Decode generates output tokens one at a time, each depending on the last, and each step requires streaming the model's weights and the KV cache through memory to produce a single token, which makes it memory-bandwidth-bound and badly underutilizes the arithmetic units. This asymmetry explains nearly everything about LLM serving economics: why output tokens are priced several times higher than input tokens, why time to first token and total generation time behave so differently, why batching helps enormously during decode and much less during prefill, and why a hundred-token answer takes far longer than a thousand-token prompt.
That asymmetry should drive the cost model. Input tokens are cheap and parallel; output tokens are expensive and serial. A design that asks the model to restate the input, reformat a large document, or emit verbose reasoning is paying the expensive currency to produce content that could often have been produced by code. Conversely, supplying more input context is relatively cheap in both latency and money, which is part of why retrieval works economically at all. The practical heuristics follow directly: bound output aggressively, ask for identifiers and structured fields rather than restated prose, and never ask the model to echo back content you already have.
Model routing is the largest available cost lever, and it is under-used because it requires admitting that most requests are easy. Model families span an order of magnitude or more in price between their smallest and largest members, and a typical production traffic mix contains a large majority of requests - classification, extraction, routing, short factual lookups grounded in retrieved text, formatting - that a small model handles at parity. Sending everything to the largest model is the equivalent of running every query on your largest database instance because some queries are heavy. A router that classifies difficulty cheaply and dispatches accordingly, with an escalation path when the small model expresses low confidence or fails validation, commonly cuts cost by well over half at indistinguishable end-to-end quality, and the honest way to confirm that is to measure both routes against the same eval set.
Latency deserves the same structural treatment. The first question about any LLM feature is whether the user is waiting, because that single answer changes the entire design. Interactive paths need streaming, a small fast model or a router, tight retrieval budgets, and a hard timeout with a graceful fallback. Paths where the user is not waiting - summarizing a document after upload, enriching records overnight, generating a weekly digest - should be queued, batched, and run against a batch API where one is available, since providers commonly offer a substantial discount for asynchronous work with a relaxed completion window. Teams routinely put non-interactive work on the interactive path simply because that was the code path that already existed, and pay both the premium price and the operational fragility for no user-visible benefit.
Finally, cost must be observable at the granularity where decisions are made. A single monthly invoice tells you nothing actionable. Emitting input and output token counts per request, tagged with tenant, feature, model, and prompt version, lets you answer the questions that actually change behavior: which feature is expensive, which tenant is anomalous, did the last prompt release increase the average prompt size, is the router sending more traffic to the large model than it used to. Combined with a spend circuit breaker on an hourly window, this turns cost from something discovered in arrears into something managed like any other resource.
| Phase | Bottleneck | Parallelism | Effect of Batching | What It Determines |
|---|---|---|---|---|
| Prefill (process the prompt) | Compute | All input tokens at once | Modest; the GPU is already saturated | Time to first token |
| Decode (generate the answer) | Memory bandwidth | One token at a time, strictly serial | Large; weights are reused across the batch | Tokens per second and total duration |
Worked Example: Bringing an Unaffordable Feature Into Budget
The arithmetic below is the conversation that should happen at design review rather than after the first invoice. Each step is a specific architectural change, and together they move the same feature by more than an order of magnitude.
FEATURE: summarize every inbound support ticket, 50,000 tickets/day
V1 - the obvious implementation
model: frontier, interactive endpoint
input: 3,400 tok (full thread + a 1,800-tok system prompt)
output: 600 tok (prose summary restating the thread)
per request: ~$0.019 daily: ~$950 annual: ~$347,000
STEP 1 - bound the output. It is the expensive currency.
Ask for structured fields, not prose: {issue, product, sentiment,
next_action, priority}. 600 -> 120 output tokens.
daily: ~$620 (-35%)
STEP 2 - stop paying for the system prompt 50,000 times.
The 1,800-token instruction block is byte-identical on every call.
Prompt-cache the prefix (Lesson 18): cached input is billed at a
fraction of the normal rate.
daily: ~$430 (-55% from V1)
STEP 3 - route by difficulty. Most tickets are not hard.
A cheap classifier sends ~85% to a small model, escalating on low
confidence or schema-validation failure.
daily: ~$140 (-85% from V1)
STEP 4 - nobody is waiting. Move it off the interactive path.
Queue it and submit through the provider batch API, which commonly
offers a large discount for a relaxed completion window.
daily: ~$75 (-92% from V1) annual: ~$27,000
Same feature. Same measured quality on the eval set. The four changes are
architectural, not clever prompting, and every one of them was available
before a line of code was written.Real-World Examples
- Character.AI published an engineering account in 2024 describing serving on the order of twenty thousand inference queries per second and stating that a combination of multi-query attention, hybrid attention horizons, cross-layer KV cache sharing, and int8 quantization had reduced its serving cost by roughly a factor of thirty since 2022. It is a concrete demonstration that inference cost at scale is an engineering variable rather than a fixed price of doing business.
- OpenAI introduced a Batch API in 2024 offering a substantial discount relative to synchronous calls in exchange for a completion window measured in hours, and other providers have shipped comparable asynchronous tiers. The existence of that pricing tier is a direct statement that interactive latency is a premium being paid, and it is wasted on any workload where no user is waiting for the result.
Review Questions & Answers
Prefill processes all input tokens in a single parallel pass that saturates the GPU's arithmetic units, so it is compute-bound and extracts a lot of useful work per unit of hardware time. Decode generates each output token serially, with every step requiring the model weights and the KV cache to be streamed through memory to produce one token, which makes it memory-bandwidth-bound and leaves most of the arithmetic capacity idle. The hardware is therefore far less efficiently used per output token than per input token, and the pricing and latency both reflect that. The design consequence is to bound output aggressively while treating additional input context as comparatively cheap.
Because the price gap between the smallest and largest members of a model family is often an order of magnitude, while prompt optimization moves token counts by a factor of two or three at best. If most of your traffic is classification, extraction, short grounded answers, and formatting, routing that majority to a small model captures most of the available savings in one change. It is risky when the routing decision is not validated, because a request the small model handles badly produces a silently worse answer rather than an error. The mitigations are measuring both routes against the same eval set, validating the small model's structured output, and escalating automatically on low confidence or validation failure.
The only question that matters is whether a user is actively waiting for the result. Enrichment, overnight summarization, backfills, digest generation, and offline classification all complete without anyone watching, so they can be queued and submitted through a discounted asynchronous tier with a relaxed completion window. Teams get it wrong because the synchronous request path already exists and reusing it is the shortest route to shipping, so non-interactive work inherits the interactive path by default and quietly pays the premium price while also being exposed to per-request timeout and rate limit failures it never needed to face.
Because the cost is driven by variables that change continuously and independently - prompt length, retrieved context volume, output length, model selection, retry behavior, and per-tenant usage patterns - and an aggregate figure arriving weeks later cannot be attributed to any of them. By the time the number is visible, several prompt versions and traffic mixes have passed. Per-request token metrics tagged with feature, model, tenant, and prompt version let a cost change be traced to the release that caused it within hours, and an hourly spend breaker converts a runaway loop from a budget event into an automatically contained incident.