Designing a Production LLM System End to End
On this page
Learning Objective
Combine every decision this course covered - problem fit, chunking, retrieval, index and filtering, model routing, caching, streaming, evals, guardrails, injection containment, and observability - into one coherent design for a real system, using a repeatable decision checklist rather than a remembered technique.
Concept Explanation
This is a synthesis lesson rather than new material. The transferable skill this course has been building toward is not any individual technique but the habit of walking a new LLM feature through the same ordered set of questions, because the expensive mistakes in this field are almost always made by skipping a question rather than by answering one badly. The order matters as much as the content: several of these decisions constrain the ones after them, and getting them out of sequence is how teams end up with a retrieval pipeline serving a problem that needed a SQL query.
The checklist opens with problem fit, from the sixth lesson: what kind of gap is this? If the answer lives in a structured system, it is a tool call. If it requires complete aggregation, it is an analytics query. If the gap is behavior, format, or tone, it is fine-tuning. If the corpus is small and stable, it is a cached long-context prompt. Only if the gap is private, changing, or post-cutoff knowledge requiring attribution does a retrieval pipeline earn its cost. Next comes the request budget from the first two lessons: how many input and output tokens, how much wall clock, how much money per request, and is a user waiting - since that last answer alone decides between an interactive streaming path and a discounted asynchronous batch path.
If retrieval is warranted, the next four questions come from the RAG and vector phases. How is the source split, respecting document structure, with contextual prefixes added before embedding and retrieval units decoupled from generation units. How are candidates found, which for a corpus containing both prose and identifiers means hybrid retrieval fused on rank, followed by a re-ranking stage and a relevance floor with a defined empty-result path. Where do the vectors live, decided by the memory arithmetic and by whether transactional consistency with your operational data is worth more than a specialized index. And how are filters applied, which is a security question whenever tenancy or permissions are involved, answered by pushing the predicate inside the search and taking the scope from the authenticated session.
The serving questions follow from phase four. Which model handles which requests, since routing the easy majority to a small model is the largest available cost lever. What is cached, starting with the provider prefix cache because it is free of correctness risk and requires only that the prompt be ordered with stable content first. Is the response streamed, which is mandatory for any interactive path and which constrains where guardrails can run. Then the production questions from phase five: how will you know if a change helped, which requires a stratified eval set built from real traffic and gating on per-stratum regression; what validation sits between the model and any action it triggers, with every failure path failing closed; what happens when untrusted content carries instructions, answered structurally through user-scoped tool permissions, outbound allow-lists, and human confirmation rather than through filtering; and what does a trace record, which determines whether a complaint takes minutes or days.
Two closing observations are worth carrying out of this course. The first is that most of the engineering in a good LLM system is not the model call. It is retrieval quality, budget enforcement, validation, evaluation, and observability, which is to say it is ordinary systems engineering applied to an unusual dependency, and teams with strong systems instincts have more transferable advantage here than they usually expect. The second is that every decision on this checklist should be revisited as the system meets real traffic, because almost none of them can be made correctly on the first pass: corpus size, query distribution, difficulty mix, and cost profile are all unknowable in advance, which is exactly why the eval suite, the traces, and the abstraction boundaries are what make the system improvable rather than merely shippable.
| Decision | Question to Ask | This Course's Lesson |
|---|---|---|
| Problem fit | Is this knowledge, behavior, a structured query, or an aggregation? | Module 6 |
| Request budget | How many tokens, how many seconds, how many cents, and is a user waiting? | Modules 1-2, 16 |
| Chunking | Does the split respect structure, carry context, and decouple retrieval from generation? | Module 7 |
| Retrieval | Hybrid or dense only, what candidate size, and what happens when nothing is relevant? | Modules 8-10 |
| Index and storage | Does the memory arithmetic fit, and is transactional consistency worth more than a specialized index? | Modules 11-14 |
| Filtering | Is the tenant predicate inside the search and taken from the session? | Module 15 |
| Model routing | Which requests genuinely need the expensive model? | Modules 16, 20 |
| Caching | Is the prompt ordered for prefix caching, and is every key tenant-scoped? | Module 18 |
| Streaming | Is a user waiting, and where can guardrails still run? | Modules 19, 22 |
| Evals | What stratified dataset gates a release, and what is the per-stratum baseline? | Module 21 |
| Guardrails and injection | What sits between the output and any action, and does everything fail closed? | Modules 22-23 |
| Observability | Does a trace make a specific complaint resolvable in minutes? | Module 24 |
Worked Example: An Internal Engineering Support Assistant, Designed End to End
The system below answers engineers' questions over internal runbooks, architecture documents, and incident postmortems, and can look up live service status. Running the checklist in order produces the design, and several of the answers reduce scope rather than add machinery.
SYSTEM: internal engineering assistant. 40k documents, ~900 daily questions.
1. PROBLEM FIT (M6)
"What is the on-call rotation for payments?" -> TOOL CALL (live system)
"How many incidents did we have in Q3?" -> ANALYTICS QUERY
"Why do we shard the ledger by account id?" -> RAG (private, changing,
needs attribution)
-> A router sends ~35% of traffic away from retrieval entirely.
2. BUDGET (M1, M2, M16)
Interactive, engineer is waiting -> streaming, TTFT target < 1.2 s.
Budget: 6k input / 700 output tokens, p95 wall clock 8 s.
3. CHUNKING (M7)
Markdown headings as boundaries; ~300-token children with 15% overlap;
contextual prefix "<doc title> > <section>"; children resolve to parent
sections for generation.
4. RETRIEVAL (M8-M10)
Hybrid BM25 + dense, fused with RRF -> 50 candidates. Service names,
error codes and ticket ids make lexical scoring non-optional.
Cross-encoder rerank -> top 5, relevance floor 0.45.
Empty case returns an honest refusal AND increments rag.no_context,
whose rising rate is the signal that documentation is missing.
5. INDEX (M11-M14)
40k docs x ~14 chunks = 560k vectors x 768 dims x 4B = 1.7 GB.
Fits comfortably beside the existing Postgres -> pgvector with HNSW.
No second system, no outbox, no reconciliation job. Behind a Retriever
interface so the decision is reversible if the corpus grows 50x.
6. FILTERING (M15)
Some postmortems are restricted. Predicate from the authenticated
session, applied INSIDE the search. CI test asserts a restricted doc
never appears for an unauthorized principal.
7. ROUTING + CACHING (M16, M18, M20)
Small model for routing/classification; mid-size for grounded answers;
escalate on low confidence. Prompt ordered stable-first so the 2.1k-token
instruction block hits the provider prefix cache. Embedding cache makes
nightly re-indexing mostly a no-op. NO semantic cache: engineering
questions differ in exactly the qualifying clauses that matter.
8. EVALS (M21)
260 cases from real questions, stratified: grounded_qa, tool_routing,
refusal_expected, restricted_access, multi_turn. CI gates on per-stratum
regression. Every reported bad answer is appended permanently.
9. GUARDRAILS + INJECTION (M22, M23)
Deterministic first: schema, citation ids must exist in the retrieved set,
outbound URLs allow-listed. The corpus is internal but NOT trusted -
postmortems quote customer emails and pasted logs, so indirect injection
is live. Status tools are read-only and scoped to the asking engineer.
Everything fails closed.
10. OBSERVABILITY (M24)
Full trace per request; 2% sampled for groundedness grading; thumbs-down
and regeneration tracked per prompt version; tokens tagged by team.
WHAT THE CHECKLIST REMOVED: an agent loop (steps are enumerable, so a
router plus a fixed pipeline is correct), a dedicated vector database
(1.7 GB), and a semantic cache (unsafe for this question distribution).Real-World Examples
- Klarna reported in February 2024 that an AI assistant built with OpenAI was handling roughly two-thirds of its customer service chats within its first month, equivalent by the company's estimate to the work of around 700 agents, and later publicly adjusted course, saying quality had suffered in places and that it was bringing human agents back for parts of the workload. Both halves of that story belong in a systems design course: the deployment demonstrated the scale such a system can reach, and the correction demonstrated why evals, quality metrics, and a human escalation path are load-bearing architecture rather than optional polish.
- GitHub published a controlled experiment in 2022 in which developers using GitHub Copilot completed a specified coding task substantially faster than a control group, reporting a difference of around fifty-five percent. Whatever the precise figure, the significant thing methodologically is that the value was established with a controlled comparison on a defined task rather than from model benchmark scores, which is the same argument this course makes for gating LLM system changes on outcome measurements rather than on offline proxies alone.
Review Questions & Answers
This is a self-directed synthesis exercise, so there is no single correct answer, and the value lies in the discipline rather than the conclusion. The questions that most often change a design are the first and the last: asking whether the gap is really knowledge rather than a structured query frequently removes an entire retrieval pipeline, and asking what a trace would need to record to resolve a specific complaint frequently reveals that the planned design is undebuggable. If the checklist produced no changes at all, that is usually a sign the questions were answered from the design already in mind rather than genuinely asked.
Because three of the questions are elimination questions. Problem fit removed about a third of traffic from retrieval entirely by routing it to tool calls and analytics queries, where the answers are exact and live rather than approximate. The index memory arithmetic came to 1.7 gigabytes, which fits comfortably beside the existing database, so a dedicated vector store would have added a second system, a dual-write consistency problem, and a reconciliation job for no benefit. And the question distribution made a semantic cache unsafe, because engineering questions differ precisely in the qualifying clauses that embeddings represent worst. Each of those is a component not built, which is a better outcome than a component built well.
Because the trust boundary is about where the text originated, not about where it is stored. Incident postmortems quote customer emails, paste log output, and embed third-party content, and support tickets and uploaded documents carry text written by people outside the organization. Once any of that reaches the context window it has the same status as every other token in the prompt, so an instruction planted in a quoted customer message is an indirect injection regardless of the fact that it sits in an internal wiki. The controls that follow are structural: tools are read-only and scoped to the asking engineer, outbound URLs are allow-listed, and validation runs before any action.
Chunk size, retrieval candidate counts, the relevance floor, embedding model choice, routing thresholds, cache thresholds, and index parameters all depend on the actual corpus, the actual query distribution, and the actual difficulty mix, none of which are knowable in advance. The implication is that the first things to build are the ones that make those decisions measurable later: a labelled eval set from real traffic, per-stage tracing, token and cost metrics tagged by feature and prompt version, and narrow interfaces around the retriever and the model provider. With those in place every subsequent parameter is a measured adjustment, and without them each one is a guess that cannot be evaluated.
Calling the API is executing one decision after it has already been made. Designing the system means recognizing that the model is an unreliable, slow, expensive, non-deterministic dependency that returns HTTP 200 when it is wrong, and that most of the engineering therefore lies in the machinery around it: deciding whether a model belongs in the path at all, bounding what each request may consume, retrieving the right context and knowing when nothing is relevant, validating before acting, containing what happens when untrusted content carries instructions, and building the evaluation and tracing that make the system improvable. That machinery is ordinary systems engineering applied to an unusual dependency, which is why strong systems instincts transfer here more directly than most people expect.
You have completed Building Production LLM Systems: RAG, Vectors & Cost. Take the twelve-point checklist from this final lesson and apply it, in order, to the next LLM feature you are asked to build.