Idempotent gRPC Methods and Safe Retries
On this page
Why gRPC retries can create duplicates
A client calls CreateOrder. The server creates the order, then the connection drops before the response arrives. The client sees UNAVAILABLE or DEADLINE_EXCEEDED and can't know whether the order exists. Retrying might create a second order; not retrying might lose the first.
Idempotent CreateOrder with an idempotency key
The client generates a unique key per logical operation and sends it on every attempt. The server remembers which keys it has processed:
message CreateOrderRequest {
string idempotency_key = 1; // generated once by the client, reused on every retry
string customer_id = 2;
repeated Order.LineItem items = 3;
}
message CreateOrderResponse { Order order = 1; bool replayed = 2; }func (s *server) CreateOrder(ctx context.Context, req *shopv1.CreateOrderRequest) (*shopv1.CreateOrderResponse, error) {
if req.GetIdempotencyKey() == "" {
return nil, status.Error(codes.InvalidArgument, "idempotency_key is required")
}
s.mu.Lock()
defer s.mu.Unlock()
if o, ok := s.byKey[req.GetIdempotencyKey()]; ok {
return &shopv1.CreateOrderResponse{Order: o, Replayed: true}, nil // repeat: return the original
}
// ... create the order, then remember it under the key
s.byKey[req.GetIdempotencyKey()] = o
return &shopv1.CreateOrderResponse{Order: o}, nil
}The same request sent twice from the lab client:
create try 1: id=ord_1 total=4998 replayed=false
create try 2: id=ord_1 total=4998 replayed=trueOne order, whatever happens on the network. In production, store the key and result in the same database transaction as the order, with a unique constraint, and expire keys after a window longer than any retry period (see the distributed systems course).
Which gRPC methods are naturally idempotent
- Reads (
GetOrder,ListOrders): always safe to retry. - Set-style writes ("set status to SHIPPED", "replace the address"): repeating gives the same result.
- Create, charge, increment, send: not idempotent without a key.
Mark idempotent methods in the .proto so tools and clients know:
rpc GetOrder(GetOrderRequest) returns (GetOrderResponse) {
option idempotency_level = NO_SIDE_EFFECTS;
}(IDEMPOTENT is the other level, for writes that are safe to repeat.)
Automatic retries with a retry policy
gRPC clients can retry transparently through service config. A backend that fails twice with UNAVAILABLE and then succeeds, called with and without a retry policy:
{"methodConfig": [{
"name": [{"service": "shop.v1.OrderService", "method": "GetOrder"}],
"retryPolicy": {
"maxAttempts": 4,
"initialBackoff": "0.1s",
"maxBackoff": "1s",
"backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"]
}
}]}no retry policy result=Unavailable, server saw 1 attempts
retry policy result=OK, server saw 3 attemptsRules for retry policies:
- Only retry idempotent methods (or ones protected by keys).
- Only retry transient codes:
UNAVAILABLEalmost always;DEADLINE_EXCEEDEDonly for idempotent calls, and only if time remains in the overall deadline. - Backoff with jitter (built into the policy) so clients don't retry in sync.
- Retry throttling (
retryThrottlingin service config) stops retries when many calls fail, so retries don't amplify an outage. - Hedging (sending duplicate requests to cut tail latency) is only for idempotent reads.
Check yourself
The server recognizes repeats by the key: if it has already processed that key, it returns the stored result. A new key on retry looks like a new operation, so the server would create a duplicate order or charge.
UNAVAILABLE, and DEADLINE_EXCEEDED only for idempotent methods with time left in the overall deadline. Retrying INVALID_ARGUMENT or PERMISSION_DENIED never helps.
Retries multiplying load during a widespread failure, which would make the outage worse. The throttle disables retries when too many calls are failing and re-enables them as successes return.
Reads, deletes by id and sets that write a complete value, like SetPrice(product, 10), since repeating them leaves the same result. Creates and increments change state each time, so they need idempotency keys to be safely retried.
In the same database transaction as the operation's effect. A crash can then never leave an order created without its idempotency record, or a record without the order, either of which would break the next retry.