Elasticsearch Replicas and Node Failure Recovery
On this page
What happens when a node dies
A three-node cluster loses a node. Each primary that lived there has a replica on another node, so the master promotes those replicas to primaries within seconds. No data is lost and searches continue. Health turns yellow: every primary is available, but some shards now have fewer copies than configured. Then the cluster rebuilds the missing replicas on the remaining nodes.
Why shard recovery waits: delayed allocation
When a node leaves, the cluster doesn't immediately copy gigabytes of data. It waits for index.unassigned.node_left.delayed_timeout (1 minute by default), in case the node is just restarting. If it comes back, its existing shard copies are reused and only recent changes are synced. Raise the delay during planned maintenance; better still, for rolling restarts, set cluster.routing.allocation.enable to primaries first so the cluster doesn't start moving shards around.
Diagnosing unassigned shards with allocation explain
On the course's single-node lab, an index with one replica per shard:
{'status': 'yellow', 'active_primary_shards': 3, 'active_shards': 3, 'unassigned_shards': 3}
index shard prirep state node
orders 0 p STARTED node-1
orders 0 r UNASSIGNED
...Ask the cluster why:
GET /_cluster/allocation/explain
{ "index": "orders", "shard": 0, "primary": false }node-1 same_shard - a copy of this shard is already allocated to this nodeThe explain API names the decider that said no. Common reasons:
| Decider | Meaning | Fix |
|---|---|---|
same_shard | Replica would sit on the same node as its primary | Add nodes, or reduce replicas |
disk_threshold | Node above the disk watermark | Free space, add nodes, delete old indices |
awareness / filter | Allocation rules exclude the node | Check zone awareness and allocation filters |
max_retry | Allocation failed repeatedly (corrupt shard, mapping issue) | Fix the cause, then POST /_cluster/reroute?retry_failed=true |
Red health means a primary is unassigned: data in that shard is unavailable. If no copy survives anywhere, the remedy is restoring from a snapshot (Day 27).
How many replicas to run
- One replica is the usual minimum in production: any single node can fail.
- More replicas add search throughput (each copy can serve queries) and survive more failures, at the cost of disk and indexing work.
- Zone awareness: with
cluster.routing.allocation.awareness.attributes: zoneand anode.attr.zoneon each node, primaries and replicas land in different availability zones, so a zone outage doesn't take both copies. - Three master-eligible nodes (in three zones) keep the cluster managed when one fails (Day 2).
- Replicas protect against hardware loss, not against mistakes: a deleted index is deleted on every copy. That's what snapshots are for.
How writes reach replicas
An index request goes to the primary, which forwards it to replicas in parallel and acknowledges once in-sync copies have it. wait_for_active_shards can require a number of copies to be available before writing, a durability guard during degraded periods.
Check yourself
Replicas of those primaries on other nodes are promoted to primaries, so all data remains available for reads and writes. Only redundancy is reduced, since the lost copies haven't been rebuilt yet, which yellow signals.
The node may be restarting for a patch or reboot. index.unassigned.node_left.delayed_timeout, one minute by default, avoids copying large amounts of data to other nodes when the original will be back shortly.
No. Deleting an index removes every copy, primaries and replicas alike. Only snapshots taken before the deletion can recover the data.
Why a specific shard is unassigned or can't move to a node, with the reason per node, such as disk watermarks, allocation filters, too many copies on one node or no valid node. It turns guessing into a direct answer.
The primary shard indexes the document, then forwards the operation to all in-sync replicas in parallel, and acknowledges the client once they've confirmed. A replica that fails is removed from the in-sync set so writes don't stall.