Catalog My learning Blog
gRPC and Protobuf Tutorial: Fast Microservice APIs · Phase 3 · Lesson 15 · 4 min read

gRPC Deadlines, Timeouts and Cancellation

On this page
  1. Why calls without deadlines cause outages
  2. Deadlines vs timeouts
  3. A 500 ms deadline in action
  4. Setting deadlines across a chain of services
  5. Cancellation and context propagation
  6. Practical deadline defaults
  7. Check yourself

Why calls without deadlines cause outages

Service A calls B, which calls C. C's database stalls. Without deadlines, A waits, B waits, C waits, each holding threads, connections and memory. Under load, this is how a slow dependency becomes an outage everywhere (see the distributed systems course on cascading failures).

Deadlines vs timeouts

A timeout is local: "I'll wait 2 seconds". A gRPC deadline is an absolute point in time that propagates: the client sends the remaining time in the grpc-timeout header, and the server's context inherits it. If the server calls another service with the same context, that call gets whatever time is left. Every service in the chain knows when the original caller will give up.

Think of a pizza order promised in 30 minutes. If the kitchen takes 25, the driver knows they have 5 minutes left, not a fresh 30. A plain timeout gives every step its own fresh 30 minutes, so the chain can keep working long after the customer has given up; a deadline passes the remaining time along.

A 500 ms deadline in action

The course's server simulates a slow lookup that respects cancellation:

Code
if req.GetOrderId() == "slow" {
	select {
	case <-time.After(3 * time.Second):           // the slow work
	case <-ctx.Done():                             // the caller gave up
		log.Printf("GetOrder(slow): caller gave up: %v", ctx.Err())
		return nil, status.FromContextError(ctx.Err()).Err()
	}
}

The client allows 500 ms:

Code
ctx, cancel := context.WithTimeout(auth, 500*time.Millisecond)
defer cancel()
_, err = client.GetOrder(ctx, &shopv1.GetOrderRequest{OrderId: "slow"})
YAML
client:  deadline:       code=DeadlineExceeded after 500ms
server:  GetOrder(slow): caller gave up: context canceled
         unary /shop.v1.OrderService/GetOrder code=Canceled took=500ms

The client got DEADLINE_EXCEEDED at 500 ms, and the server stopped its work at the same moment instead of continuing for 3 seconds. The server saw context canceled rather than deadline exceeded: the client's cancellation (sent when its deadline expired) reached the server just before the server's own copy of the deadline fired. Either way, the handler's job is the same: stop and return.

The caller's deadline travels in the grpc-timeout header; each downstream call inherits the remaining time, and when it passes, every service in the chain stops work.
The caller's deadline travels in the grpc-timeout header; each downstream call inherits the remaining time, and when it passes, every service in the chain stops work.

Setting deadlines across a chain of services

  • Set one at the edge based on the user's tolerance: an API request might get 2 seconds.
  • Pass the incoming context to every downstream call, so they inherit the remaining time.
  • Optionally tighten it per call: context.WithTimeout(ctx, 300*time.Millisecond) takes the earlier of the two deadlines.
  • Leave room for the work after the call (rendering, other calls).
  • Don't reset deadlines by creating a fresh context.Background() deep in the call chain; that breaks propagation.

Cancellation and context propagation

Cancelling a context aborts the call immediately: when a user closes the page, when one of several parallel calls fails and the others aren't needed, or when a stream should stop (the lab's server stream returned code=Canceled after the client cancelled, Day 8).

Servers must honor it. Pass the context to database drivers and HTTP clients so they abort too, and check ctx.Done() in long loops. A handler that ignores cancellation keeps consuming resources for answers nobody will read.

Practical deadline defaults

  • No call without a deadline. Many teams enforce this with a client interceptor that adds a default if none is set.
  • Choose deadlines from latency data: a little above the dependency's p99, not an arbitrary 30 seconds.
  • Long streams get long deadlines (or none) plus keepalives and heartbeats (Day 8).
  • Retries must fit inside the deadline; a retry policy (Day 22) shares the same time budget.

Check yourself

It travels with the call in the grpc-timeout header, and downstream services inherit the remaining time through their context. When it expires, the whole chain of calls stops together, instead of downstream work continuing after the caller gave up.

So database queries are cancelled when the caller's deadline passes or it disconnects, instead of continuing to use connections and CPU for a response nobody will receive. Under load, that wasted work is what turns slowness into an outage.

Creating a new context.Background() for downstream calls instead of passing the incoming context. The downstream call then has no deadline and keeps running after the original caller has given up.

The edge sets an overall deadline from the user's tolerance, and each hop uses the remaining time, optionally reserving a little for its own work.

The handler's context is cancelled, so ctx.Done() closes and ctx.Err() returns context.Canceled. Any downstream calls using that context stop too, which is why passing the context everywhere matters.