Service Discovery in Distributed Systems
On this page
Why service discovery exists
With three servers that never change, you can put their IPs in a config file. With autoscaling, deploys, containers and failed instances replaced automatically, addresses change many times a day. Service discovery answers one question at runtime: where are the healthy instances of the orders service right now?
The service registry
Discovery systems keep a service registry: a live list of instances and their addresses.
- Registration: instances add themselves when they start (self-registration), or the platform registers them (Kubernetes, ECS, Nomad).
- Health: instances send heartbeats or the registry runs health checks; failing instances are removed (Day 20 of the distributed systems course covers the detection trade-offs).
- Deregistration: instances remove themselves on clean shutdown, and expire on crashes.
Consul, etcd, ZooKeeper and Eureka are common registries; Kubernetes uses its own API server backed by etcd.
Client-side vs server-side discovery
| Client-side discovery | Server-side discovery | |
|---|---|---|
| How | The caller queries the registry and picks an instance itself | The caller sends to a stable address; a load balancer or proxy picks the instance |
| Load balancing | In the client library | In the load balancer |
| Pros | No extra hop; smart client-side balancing | Clients stay simple and language-agnostic |
| Cons | Every language needs the library | An extra component to run and scale |
| Examples | Netflix Eureka with Ribbon, gRPC client-side balancing | Kubernetes Services, AWS load balancers, service meshes |
Kubernetes: DNS plus a virtual IP
Every Kubernetes Service gets a stable DNS name and a virtual IP. From any pod:
$ nslookup orders.default.svc.cluster.local
Name: orders.default.svc.cluster.local
Address: 10.96.24.17The name follows <service>.<namespace>.svc.cluster.local, and within the same namespace just orders works. 10.96.24.17 is a ClusterIP: no pod has it. kube-proxy (or an eBPF data plane like Cilium) programs every node to send traffic for that IP to one of the ready pods behind the Service. Pods are listed in EndpointSlices, and only pods passing their readiness probe receive traffic.
A headless Service (clusterIP: None) skips the virtual IP: DNS returns the pod IPs directly, which is what databases and gRPC clients doing their own balancing need.
Service discovery pitfalls
- DNS caching: clients that cache DNS answers too long, or connection pools that never reconnect, keep calling instances that are gone. Keep TTLs short and recycle connections.
- Long-lived connections defeat balancing: HTTP/2 and gRPC reuse one connection, so a ClusterIP balances per connection, not per request. All traffic can pile onto one pod. Use client-side balancing over a headless Service, or a mesh that balances per request.
- Health checks that check too much remove every instance when one shared dependency is slow.
- A service mesh (Istio, Linkerd) adds a proxy next to each instance that handles discovery, per-request balancing, retries and mutual TLS, at the cost of extra components.
Check yourself
In client-side discovery the caller queries the registry and picks an instance; in server-side discovery it calls a stable address and a load balancer picks.
A virtual IP for a Service that no pod owns; each node routes traffic for it to one of the Service's ready pods.
gRPC keeps one long-lived HTTP/2 connection and sends all requests over it, while a ClusterIP balances connections, not requests, so that one connection pins all traffic to one pod. Use client-side load balancing with a headless Service, or a service mesh.
A Service with no ClusterIP. DNS returns the pods' own IPs, so clients can connect to them directly and balance requests themselves.
A registry listing dead instances keeps sending traffic into failures. Health checks or heartbeats with short expiry remove unhealthy instances quickly, so clients only discover instances that can serve.