LLM Streaming Responses and Perceived Latency
On this page
Learning Objective
Explain why time to first token dominates perceived responsiveness, describe the transport and architectural consequences of streaming through a real request path, and identify what streaming makes harder - output validation, error handling after a committed status code, and cancellation.
Concept Explanation
Streaming does not make generation faster. A response that takes six seconds to complete still takes six seconds. What changes is when the user first sees evidence that something is happening, and decades of human-computer interaction research say that is what governs the perception of responsiveness. Jakob Nielsen's widely cited response-time limits put roughly one second as the threshold for a user's flow of thought to remain uninterrupted and roughly ten seconds as the limit of attention without a progress indicator. A six-second silent wait sits squarely in the range where users conclude something is broken; the same six seconds with text appearing after 500 milliseconds reads as a system thinking out loud. That is why essentially every consumer LLM product streams, and why a non-streaming interactive endpoint feels defective regardless of its actual throughput.
The metric that matters for this is therefore time to first token, not total duration, and it is worth tracking as a separate service level objective. Time to first token is governed by everything that happens before generation begins: the network hop, any retrieval and re-ranking, prompt assembly, and the prefill pass over the input. Each of those is a place to optimize independently, and a prompt cache hit is the single largest improvement available because it removes most of the prefill work. The second metric is inter-token latency, which determines whether the text appears at a comfortable reading pace, and it is bounded by the serving stack rather than by your application. Reporting a single average latency number for an LLM endpoint hides both, and a system that improved its total duration while regressing time to first token will feel worse to every user while looking better on the dashboard.
Streaming changes the request path structurally. The transport is usually Server-Sent Events, which is a simple unidirectional stream over ordinary HTTP and works through most proxies, or a WebSocket when bidirectional communication is genuinely needed. The important consequence is that every intermediate component in the path must forward chunks as they arrive rather than buffering the response: a load balancer, an API gateway, a reverse proxy, or a compression middleware configured to buffer will silently convert your streaming endpoint into a non-streaming one with extra complexity, and this is one of the most common causes of "streaming works locally but not in production". Serverless platforms with response buffering or short execution limits are a related trap. Long-lived connections also change capacity planning, since concurrency is now bounded by simultaneous open connections rather than by request rate.
The hardest consequence is that streaming forecloses several things you would normally do after generating. You cannot validate a complete response against a schema before sending it, because you have already sent the beginning. You cannot run a moderation or guardrail check on the full output before the user sees the first half. And once the response has begun, the HTTP status is already committed, so a mid-stream failure cannot be reported as a 500 and must be signalled in-band through an error event the client understands. Each has a standard mitigation. Incremental validation can check structural well-formedness as tokens arrive and terminate the stream on a violation. Guardrails can operate on a sliding buffer, holding a small window of tokens so a violation can be caught before it is emitted, at the cost of a little added latency. Content that must be fully validated before a user sees any of it - a financial figure, a legal statement, a generated tool call - should simply not be streamed.
Two further details separate a working streaming implementation from a robust one. Cancellation should propagate: when a user navigates away or stops a generation, the client disconnect must abort the upstream model call, or you continue paying for tokens nobody will read, and on a self-hosted server you continue occupying a batch slot. Detecting the disconnect and cancelling upstream is a small amount of code that directly reduces cost on any product where users interrupt. And structured output interacts awkwardly with streaming, since a partial JSON object is not valid JSON, so clients either wait for the complete object, defeating the purpose, or use an incremental parser that can render a partially complete structure. The usual resolution is to stream the prose the user reads and deliver structured fields as discrete events once complete.
| Metric | What It Measures | Governed By | Target Shape |
|---|---|---|---|
| Time to first token | Silence before the first visible output | Retrieval, prompt assembly, prefill, network | Under roughly one second for interactive use |
| Inter-token latency | Pace at which text appears | Serving stack, batch pressure, model size | Comfortably faster than reading speed |
| Total duration | End-to-end wall clock | Output length multiplied by inter-token latency | Secondary; matters for cost and timeouts |
| Time to first useful content | When the answer, not the preamble, appears | Prompt design and output ordering | Put the answer before the explanation |
Worked Example: A Streaming Endpoint With Guardrails and Cancellation
The implementation below keeps a small sliding buffer so a guardrail can act before a violation reaches the user, signals mid-stream failure in-band because the status code is already committed, and propagates client disconnects upstream so cancelled generations stop costing money.
async def stream_answer(request, principal):
ttft_start = now()
chunks = await retrieve(request.question, principal) # counted in TTFT
prompt = build_prompt(chunks, request.question)
async def events():
buffer, first = '', True
try:
async for token in llm.stream(prompt, max_tokens=800, timeout=30):
if first:
metrics.observe('llm.ttft_ms', now() - ttft_start) # the SLO
first = False
# Sliding buffer: hold a little back so a guardrail can act
# BEFORE text reaches the user. Streaming removes the chance
# to validate a complete response, so validate a window.
buffer += token
if len(buffer) > 120:
emit, buffer = buffer[:-120], buffer[-120:]
if violation := guardrails.scan(emit):
yield sse('error', {'code': 'content_blocked',
'reason': violation.code})
return # terminate the stream
yield sse('token', {'text': emit})
yield sse('token', {'text': buffer})
# Structured fields are delivered as ONE complete event - partial
# JSON is not valid JSON, so it is never streamed piecemeal.
yield sse('citations', {'ids': [c.id for c in chunks]})
yield sse('done', {'finish_reason': 'stop'})
except asyncio.CancelledError:
# The user navigated away. Abort upstream or we keep paying for
# tokens nobody will read and keep holding a batch slot.
await llm.abort()
metrics.increment('llm.cancelled')
raise
except Exception as e:
# HTTP 200 was already committed with the first byte. The only
# way to report failure now is IN-BAND.
log.error(e)
yield sse('error', {'code': 'generation_failed'})
return EventSourceResponse(events(), headers={
'Cache-Control': 'no-cache',
'X-Accel-Buffering': 'no', # or your proxy silently un-streams this
})Real-World Examples
- Jakob Nielsen's response-time limits, drawn from human-computer interaction research and published in his 1993 book Usability Engineering, identify roughly 0.1 seconds as feeling instantaneous, roughly 1 second as the limit for uninterrupted flow of thought, and roughly 10 seconds as the limit of attention without feedback. Those thresholds explain precisely why a multi-second LLM response is unacceptable without streaming and entirely acceptable with it, despite identical total duration.
- Server-Sent Events became the de facto transport for LLM streaming across the major provider APIs and client SDKs, chosen over WebSockets because generation is unidirectional, SSE runs over ordinary HTTP with automatic reconnection semantics, and it traverses standard proxy infrastructure. The recurring production failure is an intermediate proxy or gateway configured to buffer responses, which converts a correctly implemented streaming endpoint into a non-streaming one with no error raised anywhere.
Review Questions & Answers
Because perception is governed by the length of unexplained silence rather than by completion time. Established response-time research places roughly one second as the point beyond which a user's flow of thought is interrupted and roughly ten seconds as the limit of attention without feedback, so a six-second blank wait reads as a broken system. Streaming moves the first visible evidence of progress to under a second while leaving completion unchanged, and it also lets the user begin reading immediately, so the remaining generation overlaps with the time they would have spent reading anyway.
It removes the ability to inspect a complete response before any of it reaches the user, which affects three things. Schema validation cannot run on the full output, so structural checks must be incremental and the stream terminated on violation. Guardrails cannot scan a finished response, so they run over a small sliding buffer held back from the user, trading a little latency for the ability to block content before it is shown. And error reporting cannot use an HTTP status code, since 200 was committed with the first byte, so failures must be signalled in-band as an error event the client understands. Content that genuinely must be fully validated before being seen should not be streamed at all.
Because streaming requires every intermediate component to forward chunks as they arrive, and several standard components buffer by default. A reverse proxy, an API gateway, a content-compression middleware, or a serverless platform with response buffering will accumulate the full response before sending it, which produces a correct but non-streaming result with no error anywhere. Local development typically has none of those components in the path, which is why the behavior differs. The first places to check are proxy buffering settings, any compression layer applied to the response, and platform-level limits on response streaming and execution duration.
Because generation continues after the client disconnects unless something stops it. Output tokens are the expensive currency, so a user who navigates away after two seconds of a fifteen-second generation leaves you paying for thirteen seconds of tokens nobody will ever read. On self-hosted infrastructure it is worse than the money, because the abandoned request continues occupying a batch slot and KV cache memory that a waiting request could be using, reducing effective throughput. Detecting the disconnect and aborting the upstream call is a small amount of code that pays back directly on any product where users interrupt or refine their requests.