Catalog My learning Blog
gRPC and Protobuf Tutorial: Fast Microservice APIs · Phase 3 · Lesson 18 · 3 min read

gRPC Load Balancing Explained

On this page
  1. Why gRPC load balancing sends everything to one backend
  2. pick_first vs round_robin, measured
  3. Headless Services so round_robin sees every pod
  4. Client-side vs proxy vs service mesh load balancing
  5. More gRPC load balancing tips
  6. Check yourself

Why gRPC load balancing sends everything to one backend

A team scales a gRPC service from one pod to three behind a Kubernetes Service. Graphs show one pod at 90% CPU and two nearly idle. Nothing's broken: it's how gRPC connections work.

A gRPC client opens one HTTP/2 connection and sends every call over it as a separate stream. A Layer 4 load balancer (a Kubernetes ClusterIP Service, an AWS NLB) balances connections, not calls. One connection lands on one backend, so all calls from that client go there, for as long as the connection lives.

pick_first vs round_robin, measured

Three local backends, one client connection, 30 calls, with each of grpc-go's two built-in policies:

Code
pick_first   30 calls on one connection -> map[backend-a:30]
round_robin  30 calls on one connection -> map[backend-a:11 backend-b:10 backend-c:9]
  • pick_first (the default) connects to the first working address and sends everything there.
  • round_robin connects to every address the resolver returns and rotates calls across them.

Switching is a service config setting:

Code
conn, err := grpc.NewClient("dns:///orders.default.svc.cluster.local:50051",
	grpc.WithTransportCredentials(creds),
	grpc.WithDefaultServiceConfig(`{"loadBalancingConfig": [{"round_robin": {}}]}`))
A Layer 4 balancer pins a client's single HTTP/2 connection to one backend; per-call balancing in the client or a Layer 7 proxy spreads calls across all backends.
A Layer 4 balancer pins a client's single HTTP/2 connection to one backend; per-call balancing in the client or a Layer 7 proxy spreads calls across all backends.

Headless Services so round_robin sees every pod

Round robin is only as good as the address list. In Kubernetes, a normal ClusterIP Service resolves to one virtual IP, so the client still sees one backend. Use a headless Service (clusterIP: None): DNS then returns every pod IP, and dns:/// resolution gives round_robin all of them. Clients re-resolve when connections fail; set MaxConnectionAge on servers so clients periodically reconnect and pick up new pods.

Client-side vs proxy vs service mesh load balancing

ApproachHowTrade-offs
Client-side (round_robin, or xDS)The client knows all backends and picks per callNo extra hop; every client language must support it; discovery needed
Layer 7 proxy (Envoy, NGINX, cloud HTTP/2 load balancers)A proxy terminates HTTP/2 and balances each callWorks for any client; one extra hop and component
Service mesh (Istio, Linkerd)A sidecar proxy next to each pod balances per callTransparent to apps; adds mTLS and telemetry; operational cost

xDS lets grpc-go and other libraries get balancing configuration from a control plane (as Envoy does), enabling weighted, locality-aware and least-request policies without sidecars.

More gRPC load balancing tips

  • Server-side MaxConnectionAge (with a grace period) makes long-lived connections cycle, so new backends get traffic after scale-up.
  • Long streams stay on one backend regardless of policy; plan capacity per stream (Day 8).
  • Health checking removes bad backends from the rotation (Day 20).
  • Measure per-pod load after any change; uneven graphs are the clue.

Check yourself

It balances TCP connections, and a gRPC client sends all its calls over one long-lived connection to a single pod.

pick_first sent all 30 calls to one backend, the first address that connected. round_robin spread them 11, 10 and 9 across the three, because it keeps a connection to each and rotates calls.

A headless Service has no ClusterIP, so DNS returns every pod IP. The client can then connect to all of them and balance calls with round_robin, re-resolving as pods change.

When many languages are involved, or you want retries, mTLS and metrics without changing clients. Envoy-based meshes balance per request.

Long-lived connections stay on old pods after scaling. A maximum age makes clients reconnect periodically and spread to new pods.