Catalog My learning Blog
gRPC and Protobuf Tutorial: Fast Microservice APIs · Phase 4 · Lesson 22 · 3 min read

Idempotent gRPC Methods and Safe Retries

On this page
  1. Why gRPC retries can create duplicates
  2. Idempotent CreateOrder with an idempotency key
  3. Which gRPC methods are naturally idempotent
  4. Automatic retries with a retry policy
  5. Check yourself

Why gRPC retries can create duplicates

A client calls CreateOrder. The server creates the order, then the connection drops before the response arrives. The client sees UNAVAILABLE or DEADLINE_EXCEEDED and can't know whether the order exists. Retrying might create a second order; not retrying might lose the first.

Idempotent CreateOrder with an idempotency key

The client generates a unique key per logical operation and sends it on every attempt. The server remembers which keys it has processed:

Protobuf
message CreateOrderRequest {
  string idempotency_key = 1;   // generated once by the client, reused on every retry
  string customer_id = 2;
  repeated Order.LineItem items = 3;
}
message CreateOrderResponse { Order order = 1; bool replayed = 2; }
Go
func (s *server) CreateOrder(ctx context.Context, req *shopv1.CreateOrderRequest) (*shopv1.CreateOrderResponse, error) {
	if req.GetIdempotencyKey() == "" {
		return nil, status.Error(codes.InvalidArgument, "idempotency_key is required")
	}
	s.mu.Lock()
	defer s.mu.Unlock()
	if o, ok := s.byKey[req.GetIdempotencyKey()]; ok {
		return &shopv1.CreateOrderResponse{Order: o, Replayed: true}, nil   // repeat: return the original
	}
	// ... create the order, then remember it under the key
	s.byKey[req.GetIdempotencyKey()] = o
	return &shopv1.CreateOrderResponse{Order: o}, nil
}

The same request sent twice from the lab client:

Code
create try 1:   id=ord_1 total=4998 replayed=false
create try 2:   id=ord_1 total=4998 replayed=true

One order, whatever happens on the network. In production, store the key and result in the same database transaction as the order, with a unique constraint, and expire keys after a window longer than any retry period (see the distributed systems course).

The client attaches an idempotency key; the server stores the result under that key, so retries after a lost response return the original result instead of acting twice.
The client attaches an idempotency key; the server stores the result under that key, so retries after a lost response return the original result instead of acting twice.

Which gRPC methods are naturally idempotent

  • Reads (GetOrder, ListOrders): always safe to retry.
  • Set-style writes ("set status to SHIPPED", "replace the address"): repeating gives the same result.
  • Create, charge, increment, send: not idempotent without a key.

Mark idempotent methods in the .proto so tools and clients know:

Code
rpc GetOrder(GetOrderRequest) returns (GetOrderResponse) {
  option idempotency_level = NO_SIDE_EFFECTS;
}

(IDEMPOTENT is the other level, for writes that are safe to repeat.)

Automatic retries with a retry policy

gRPC clients can retry transparently through service config. A backend that fails twice with UNAVAILABLE and then succeeds, called with and without a retry policy:

JSON
{"methodConfig": [{
  "name": [{"service": "shop.v1.OrderService", "method": "GetOrder"}],
  "retryPolicy": {
    "maxAttempts": 4,
    "initialBackoff": "0.1s",
    "maxBackoff": "1s",
    "backoffMultiplier": 2,
    "retryableStatusCodes": ["UNAVAILABLE"]
  }
}]}
Code
no retry policy  result=Unavailable, server saw 1 attempts
retry policy     result=OK, server saw 3 attempts

Rules for retry policies:

  • Only retry idempotent methods (or ones protected by keys).
  • Only retry transient codes: UNAVAILABLE almost always; DEADLINE_EXCEEDED only for idempotent calls, and only if time remains in the overall deadline.
  • Backoff with jitter (built into the policy) so clients don't retry in sync.
  • Retry throttling (retryThrottling in service config) stops retries when many calls fail, so retries don't amplify an outage.
  • Hedging (sending duplicate requests to cut tail latency) is only for idempotent reads.

Check yourself

The server recognizes repeats by the key: if it has already processed that key, it returns the stored result. A new key on retry looks like a new operation, so the server would create a duplicate order or charge.

UNAVAILABLE, and DEADLINE_EXCEEDED only for idempotent methods with time left in the overall deadline. Retrying INVALID_ARGUMENT or PERMISSION_DENIED never helps.

Retries multiplying load during a widespread failure, which would make the outage worse. The throttle disables retries when too many calls are failing and re-enables them as successes return.

Reads, deletes by id and sets that write a complete value, like SetPrice(product, 10), since repeating them leaves the same result. Creates and increments change state each time, so they need idempotency keys to be safely retried.

In the same database transaction as the operation's effect. A crash can then never leave an order created without its idempotency record, or a record without the order, either of which would break the next retry.