gRPC Load Balancing Explained
On this page
Why gRPC load balancing sends everything to one backend
A team scales a gRPC service from one pod to three behind a Kubernetes Service. Graphs show one pod at 90% CPU and two nearly idle. Nothing's broken: it's how gRPC connections work.
A gRPC client opens one HTTP/2 connection and sends every call over it as a separate stream. A Layer 4 load balancer (a Kubernetes ClusterIP Service, an AWS NLB) balances connections, not calls. One connection lands on one backend, so all calls from that client go there, for as long as the connection lives.
pick_first vs round_robin, measured
Three local backends, one client connection, 30 calls, with each of grpc-go's two built-in policies:
pick_first 30 calls on one connection -> map[backend-a:30]
round_robin 30 calls on one connection -> map[backend-a:11 backend-b:10 backend-c:9]pick_first(the default) connects to the first working address and sends everything there.round_robinconnects to every address the resolver returns and rotates calls across them.
Switching is a service config setting:
conn, err := grpc.NewClient("dns:///orders.default.svc.cluster.local:50051",
grpc.WithTransportCredentials(creds),
grpc.WithDefaultServiceConfig(`{"loadBalancingConfig": [{"round_robin": {}}]}`))Headless Services so round_robin sees every pod
Round robin is only as good as the address list. In Kubernetes, a normal ClusterIP Service resolves to one virtual IP, so the client still sees one backend. Use a headless Service (clusterIP: None): DNS then returns every pod IP, and dns:/// resolution gives round_robin all of them. Clients re-resolve when connections fail; set MaxConnectionAge on servers so clients periodically reconnect and pick up new pods.
Client-side vs proxy vs service mesh load balancing
| Approach | How | Trade-offs |
|---|---|---|
| Client-side (round_robin, or xDS) | The client knows all backends and picks per call | No extra hop; every client language must support it; discovery needed |
| Layer 7 proxy (Envoy, NGINX, cloud HTTP/2 load balancers) | A proxy terminates HTTP/2 and balances each call | Works for any client; one extra hop and component |
| Service mesh (Istio, Linkerd) | A sidecar proxy next to each pod balances per call | Transparent to apps; adds mTLS and telemetry; operational cost |
xDS lets grpc-go and other libraries get balancing configuration from a control plane (as Envoy does), enabling weighted, locality-aware and least-request policies without sidecars.
More gRPC load balancing tips
- Server-side
MaxConnectionAge(with a grace period) makes long-lived connections cycle, so new backends get traffic after scale-up. - Long streams stay on one backend regardless of policy; plan capacity per stream (Day 8).
- Health checking removes bad backends from the rotation (Day 20).
- Measure per-pod load after any change; uneven graphs are the clue.
Check yourself
It balances TCP connections, and a gRPC client sends all its calls over one long-lived connection to a single pod.
pick_first sent all 30 calls to one backend, the first address that connected. round_robin spread them 11, 10 and 9 across the three, because it keeps a connection to each and rotates calls.
A headless Service has no ClusterIP, so DNS returns every pod IP. The client can then connect to all of them and balance calls with round_robin, re-resolving as pods change.
When many languages are involved, or you want retries, mTLS and metrics without changing clients. Envoy-based meshes balance per request.
Long-lived connections stay on old pods after scaling. A maximum age makes clients reconnect periodically and spread to new pods.