Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 1 · Lesson 5 · 8 min read

Why Calling the LLM API Directly Fails in Production

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: The Same Endpoint, Naive and Hardened
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Enumerate the specific ways a direct, unguarded call from a request handler to a model provider fails in production - unbounded cost, cascading latency, retry storms, provider outages with no fallback, silent quality regressions, and data leaking into a third party - and describe the standard structural mitigations for each.

Concept Explanation

Nearly every LLM system starts the same way, and reasonably so: a request handler builds a string, calls the provider SDK, and returns the text. This is the right prototype. It becomes a production incident because each property from the first lesson of this course has a specific structural consequence that the prototype leaves unaddressed, and those consequences do not appear gradually. They appear the first time real traffic, a hostile user, or a provider incident arrives. Understanding this list is worth more than any individual technique in this course, because the rest of the course is largely the catalog of answers to it.

Cost failures come first because they are the most common and the most embarrassing. Without a max_tokens ceiling a single generation can run to the model's limit, and a prompt that accidentally includes an entire uploaded file multiplies input cost by a factor nobody reviewed. Without per-user and per-tenant rate limits, one automated client or one badly written loop can consume a month of budget in an afternoon. Without a spend alert tied to token metrics rather than to the monthly invoice, you discover it when finance does. The mitigations are unglamorous and non-negotiable: always set max_tokens, bound input size before assembly, rate limit per authenticated principal rather than per IP, emit token counts as a first-class metric, and put a hard circuit breaker on aggregate spend per hour.

Latency and concurrency failures come next. A multi-second call held synchronously inside a request handler consumes a connection, a thread, and often a database transaction for its entire duration, so a modest traffic increase saturates pools that had enormous headroom under conventional load. Worse, the naive retry loop turns a slow provider into an outage: when the provider degrades, every client retries, tripling load on an already struggling dependency at exactly the wrong moment. The fixes are the standard distributed systems toolkit applied here: hard timeouts on every call, retries with exponential backoff and jitter and a strict attempt cap, a circuit breaker that stops calling a failing provider entirely, bounded queues with an explicit rejection policy rather than unbounded buffering, and streaming so the client sees progress instead of waiting on a silent socket.

Dependency failures are the category teams underestimate most. A single hardcoded provider and model means a provider outage is a total product outage, and there is no code path to switch. Model deprecation is the slower version of the same problem: providers retire model versions on published schedules, and a system with the model name scattered across forty call sites has no way to test a replacement. The structural answer is an internal abstraction layer between your application and any provider, thin enough not to become its own maintenance burden, that centralizes the provider client, the model selection, the timeout and retry policy, the token accounting, and the logging. With that layer, falling back to a secondary provider or a smaller model under load is a configuration change; without it, it is a refactor conducted during an incident.

Finally, two failures that produce no error at all. The first is silent quality regression: with no eval suite, a prompt tweak, a provider-side model update, or a change in retrieved content degrades output quality and nothing detects it, because there is no test that can fail. The second is data governance. Sending user content to a third-party API without a data handling review, a redaction step for personal and regulated data, and a documented retention position is a compliance problem that predates any model-specific concern, and it has already produced real corporate bans. The prototype-to-production transition is precisely the addition of these missing layers: budget enforcement, resilience, an abstraction boundary, an eval gate, and a data policy.

Failure ModeHow It Shows UpStructural Mitigation
Unbounded token spendA runaway generation or an oversized prompt multiplies the billAlways set max_tokens, bound input size, per-tenant rate limits, hourly spend circuit breaker
Pool exhaustionMulti-second calls held synchronously saturate threads and connectionsAsync handling, hard timeouts, bounded queues with an explicit rejection policy
Retry stormProvider degrades, all clients retry, load triples on a failing dependencyExponential backoff with jitter, strict attempt cap, circuit breaker
Single provider dependencyA provider incident is a full product outage with no code path to switchInternal abstraction layer so provider and model are configuration, not code
Silent quality regressionOutput quality drops and no test fails, because no test existsVersioned prompts plus an offline eval suite gating every change
Data leakageUser or regulated data leaves the boundary without review or redactionRedaction before the call, documented retention terms, per-field data classification

Worked Example: The Same Endpoint, Naive and Hardened

The two versions below implement the same feature. The differences are entirely structural, and every one of them maps to a row in the table above.

Python
# NAIVE: works perfectly in the demo, fails in all six ways above
@app.post('/ask')
def ask(req):
    prompt = f'Answer this: {req.question}\n\nDocument:\n{req.document}'
    return openai.chat.completions.create(
        model='gpt-4o', messages=[{'role': 'user', 'content': prompt}]
    ).choices[0].message.content
# no max_tokens, no timeout, no rate limit, no fallback, no redaction,
# no token metrics, no prompt version, unbounded req.document

# HARDENED: same feature, with the missing layers made explicit
@app.post('/ask')
async def ask(req, principal):
    await rate_limiter.acquire(principal.tenant_id, cost=1)       # per-tenant, not per-IP
    if spend_breaker.is_open():                                    # hourly spend ceiling
        raise ServiceUnavailable('AI features temporarily paused')

    document = redact_pii(truncate_tokens(req.document, 8_000))    # bound + classify
    messages = prompts.render('ask', version=7, question=req.question, document=document)

    result = await llm.complete(                                   # internal abstraction layer
        messages=messages,
        max_tokens=800,                                            # hard output ceiling
        timeout=20.0,                                              # hard wall clock ceiling
        retries=Retry(max_attempts=3, backoff='exponential', jitter=True),
        fallback=['claude-sonnet-4', 'gpt-4o-mini'],               # provider + size fallback
        stream=True,
    )
    metrics.observe('llm.input_tokens', result.usage.input_tokens, tenant=principal.tenant_id)
    metrics.observe('llm.output_tokens', result.usage.output_tokens, tenant=principal.tenant_id)
    return stream_response(result, trace_id=principal.trace_id)
A naive request handler calling a provider directly, annotated with the six places it breaks, set against a hardened path where rate limiting, budget enforcement, redaction, an internal provider abstraction, retries with a circuit breaker, and token metrics each close one of those gaps.
A naive request handler calling a provider directly, annotated with the six places it breaks, set against a hardened path where rate limiting, budget enforcement, redaction, an internal provider abstraction, retries with a circuit breaker, and token metrics each close one of those gaps.

Real-World Examples

  • Samsung restricted employee use of external generative AI tools in 2023 after internal reports that engineers had pasted proprietary source code and internal meeting content into a public chatbot, and several banks including JPMorgan Chase imposed similar restrictions around the same period. The common thread is that the data governance question was never about the model's capability, it was about an unreviewed egress path from an internal system to a third party.
  • In March 2023 OpenAI took ChatGPT offline after a bug in an open-source Redis client library caused some users to see other users' conversation titles and, for a small number of subscribers, partial payment-related information. It is a useful reminder that the caching and session layers wrapped around a model are ordinary distributed systems components subject to ordinary distributed systems bugs, and that they need the same review as any other multi-tenant cache.
WarningNever call a model provider directly from a request handler in production. An unguarded call has no output ceiling, no timeout, no rate limit, no fallback, no token accounting, and no redaction, and each of those gaps has a well-documented incident behind it. Route every call through a thin internal abstraction that owns budget, resilience, provider selection, and observability.

Review Questions & Answers

Degradation is usually a capacity problem, and a naive retry loop responds to it by immediately multiplying the load offered to the struggling dependency. If every client retries three times without backoff, the provider sees roughly three times the request rate exactly when it has least capacity, which converts partial degradation into a full outage and lengthens recovery. The correct pattern is exponential backoff with jitter so retries spread out rather than synchronizing, a strict cap on attempts, and a circuit breaker that stops sending traffic entirely once a failure threshold is crossed, giving the dependency room to recover.

It creates one place that owns provider client construction, model selection, timeouts, retry and circuit breaker policy, token accounting, prompt version logging, and trace propagation. With it, switching providers during an outage, routing cheap requests to a smaller model, adding a spend breaker, or migrating off a deprecated model version are all configuration changes applied in one location. Without it, each of those is a search-and-replace across every call site, typically attempted under time pressure during an incident, which is exactly when a large mechanical refactor is most likely to introduce new problems.

Input size is under your control at assembly time, so truncation bounds it deterministically. Output length is chosen by the model during generation, and with no ceiling it can run to the model limit, which on a large-context model can be many thousands of tokens, at output token prices that are typically several times input prices. A prompt that unexpectedly triggers a long enumeration, a repetition loop, or a verbose format can therefore produce a bill and a latency far beyond anything the input size predicted, and the only hard control is an explicit max_tokens ceiling on every call.

Because the naive architecture has no mechanism that can ever detect it. Prompt changes, provider-side model updates, and changes in retrieved content can all degrade output quality while every infrastructure signal stays green: the requests succeed, latency is normal, token counts are normal. In conventional software a behavioral regression eventually breaks a test or raises an exception, but here nothing fails, so the regression persists until a customer complains, at which point there is no version history to bisect and no baseline to compare against. The eval suite is the only component that turns that invisible regression into a detectable one.