Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 1 · Lesson 1 · 8 min read

LLM System Design: Why LLM Apps Are Different

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Budgeting a Single Request Before You Build It
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Explain the four properties that make a system built around a large language model behave unlike conventional software - non-determinism, latency measured in seconds rather than milliseconds, a per-request marginal cost that is nowhere near zero, and correctness that is probabilistic rather than binary - and describe what each one changes about how the system is designed, tested, and operated.

Concept Explanation

A conventional backend service is, in the useful approximation, a deterministic function: the same request produces the same response, and when it does not, that is a bug you can chase down to a specific line. Almost every habit of mainstream software engineering rests on that assumption. Unit tests assert exact equality. Caching is trivially safe because a repeated input has a single correct answer. Debugging works because a failure is reproducible. An LLM call breaks that assumption at the root: the model samples from a probability distribution over tokens, and even with temperature set to zero you can observe different outputs across calls, because GPU floating point reductions are not associative, batch composition changes how those reductions are grouped, and the provider may silently move you between hardware or model point-releases. Designing around a model means giving up exact-output assertions as your primary correctness tool and replacing them with distributional ones, which is the subject of the evals lesson later in this course.

Latency is the second break. A well-behaved internal HTTP call is a few milliseconds and its duration is roughly independent of the response body size. An LLM call is typically seconds, and critically its duration is close to linear in the number of output tokens the model decides to generate, which you cannot know before the call returns. That means your p99 is not governed by your infrastructure but by how verbose the model felt like being. Two metrics replace the single latency number engineers are used to: time to first token, which governs how responsive the product feels, and inter-token latency or tokens per second, which governs how long the full answer takes. A synchronous request-per-thread architecture that was perfectly fine at 20 milliseconds per call becomes a connection-pool exhaustion incident at 8 seconds per call, so LLM-facing paths need explicit queueing, aggressive timeouts, and usually streaming.

Third, marginal cost per request is a real number. In conventional web services the compute cost of serving one more request rounds to nothing, which is why almost nobody designs an endpoint around its per-call cost. Token-priced inference reverses that: cost scales roughly linearly with input tokens plus output tokens, so a system prompt that quietly grew from 300 to 3,000 tokens because six people each added a paragraph of instructions has multiplied the fixed cost of every single request tenfold, forever. Cost belongs in the architecture review the way storage capacity belonged in a 1990s design review, and it interacts with every other decision in this course: how much retrieved context you stuff into a prompt, how many re-ranking passes you run, how many times an agent loops.

Fourth, and most consequential for product design, correctness is probabilistic and failure is silent. A traditional service that cannot answer raises an exception, returns a 500, and trips your alerting. A language model that does not know the answer produces a fluent, confident, well-formatted, completely wrong one, and returns HTTP 200 with normal latency and normal token counts. There is no signal in your infrastructure telling you anything went wrong. Every observability instinct you have - error rates, status codes, exception traces - is blind to the dominant failure mode of the system you just built. This single property is why grounding, citations, evals, guardrails, and human review appear as their own lessons later: they exist to manufacture a failure signal that the runtime does not give you for free.

Put together, the practical stance is to treat the model as an unreliable, slow, expensive, non-deterministic third-party dependency, and to design the system as the machinery that makes that dependency safe to depend on. Keep deterministic logic outside the model wherever a deterministic implementation exists, because a regular expression or a SQL query will always be cheaper, faster, and more testable than asking a model to do the same thing. Put a budget, a timeout, and a fallback around every call. Cache aggressively where semantics allow. And accept that a meaningful part of the engineering work in an LLM system is not the model call at all, but the retrieval, validation, and evaluation scaffolding around it.

PropertyConventional ServiceLLM-Backed ServiceWhat It Changes
DeterminismSame input, same outputOutput varies run to run, even at temperature 0Exact-match assertions stop working; you need eval sets and scored metrics
LatencyMilliseconds, independent of response sizeSeconds, roughly linear in output tokens generatedStreaming, explicit queueing, and per-call timeouts become mandatory
Marginal costEffectively zero per requestPriced per input and output token, linear in prompt sizePrompt length and retry policy become architecture-level decisions
Failure modeException, 5xx, visible in metricsFluent wrong answer, HTTP 200, normal metricsCorrectness signal must be built: grounding, evals, guardrails, review

Worked Example: Budgeting a Single Request Before You Build It

Before writing any code, it is worth doing the arithmetic on a single request, because the numbers usually change the design. The following sketch prices one request of a document question-answering feature and immediately exposes that the system prompt, not the user question, dominates the bill.

Code
# Per-request budget sketch for a document Q&A endpoint

INPUT TOKENS
  system prompt / instructions      1,800   <- fixed cost on every single call
  retrieved chunks (8 x ~450 tok)   3,600
  conversation history                900
  user question                        40
  ----------------------------------------
  total input                       6,340

OUTPUT TOKENS
  typical answer                      350   <- but unbounded without max_tokens

LATENCY BUDGET
  retrieval (vector search + rerank)  120 ms
  time to first token                 600 ms   <- what the user actually feels
  generation (350 tok @ 60 tok/s)   5,800 ms
  ----------------------------------------
  wall clock                        ~6.5 s     <- streaming is not optional here

OBSERVATIONS THAT CHANGE THE DESIGN
  1. 28% of every bill is the static system prompt -> prompt-cache it.
  2. Retrieved context is 57% of input -> chunk count is a cost dial, not a quality dial only.
  3. Wall clock is 6.5 s -> a non-streaming synchronous endpoint will feel broken.
  4. max_tokens is unset -> a runaway generation can 10x both cost and latency.
The four properties that separate an LLM-backed request path from a conventional one: non-deterministic output, latency proportional to generated tokens, per-token marginal cost, and a silent wrong-answer failure mode that returns HTTP 200 and never touches your error metrics.
The four properties that separate an LLM-backed request path from a conventional one: non-deterministic output, latency proportional to generated tokens, per-token marginal cost, and a silent wrong-answer failure mode that returns HTTP 200 and never touches your error metrics.

Real-World Examples

  • Moffatt v. Air Canada (British Columbia Civil Resolution Tribunal, February 2024): Air Canada's website chatbot told a passenger he could apply for a bereavement fare retroactively, which contradicted the airline's actual published policy, and the tribunal ordered Air Canada to pay damages, rejecting the argument that the chatbot was a separate entity responsible for its own statements. It is the clearest demonstration that the fluent-but-wrong failure mode is a legal and product liability, not a quality metric, and that no HTTP error code ever fired.
  • Stack Overflow temporarily banned ChatGPT-generated answers in December 2022, with moderators explaining that the answers had a high rate of being incorrect while looking plausible and well-written enough that reviewing them took longer than the volume allowed. That is exactly the operational shape of probabilistic correctness at scale: the cost of verification, not the cost of generation, became the binding constraint.
Key TakeawaysAn LLM is an unreliable, slow, expensive, non-deterministic dependency that returns HTTP 200 when it is wrong. Design the system as the machinery that makes that dependency safe to depend on: keep deterministic work out of the model, budget tokens and latency per request explicitly, and build an independent correctness signal, because your existing error metrics cannot see the dominant failure mode.

Review Questions & Answers

Temperature zero removes sampling randomness from token selection, but it does not make the underlying computation bit-identical across calls. GPU floating point reductions are not associative, so the grouping of those reductions changes with batch composition, and batch composition depends on what other requests happen to be in flight. Providers also move traffic between hardware generations and ship model point-releases without changing the model name. The practical consequence is that exact-output assertions are not a usable correctness tool, so tests have to score outputs against criteria rather than compare them to a fixed string.

Concurrency capacity in such an architecture is roughly the thread or connection pool size divided by the per-request duration. Going from 20 milliseconds to 8 seconds per call is a 400x increase in how long each unit of concurrency is held, so the same traffic that used a few percent of the pool now exhausts it, and requests queue or fail in components that never previously showed backpressure. The fixes are architectural rather than incremental: asynchronous handling, explicit bounded queues with a rejection policy, hard per-call timeouts, and streaming so the connection produces output while it waits.

The user question is paid for once per actual user intent, and it is usually short. The system prompt is paid for on every single call regardless of what the user asked, so its length multiplies straight through your entire request volume and never amortizes. A system prompt that grows from 300 to 3,000 tokens adds 2,700 tokens to every request forever, which at scale dwarfs the variable portion. This is also why prompt caching, covered later in the course, targets exactly the static prefix rather than the variable tail.

Conventional observability is built around infrastructure-visible failure signals: exceptions, non-2xx status codes, timeouts, and saturation. An LLM producing a confidently wrong answer emits none of those. The request completes normally, the latency is typical, the token counts are typical, and the response is well-formed. The failure exists only at the semantic level, which means the correctness signal has to be manufactured deliberately through grounding checks, output validation, offline eval suites, and sampled human review, rather than inherited from the runtime.