Catalog My learning Blog
Elasticsearch Tutorial: How Full-Text Search Works · Phase 4 · Lesson 23 · 3 min read

Elasticsearch Replicas and Node Failure Recovery

On this page
  1. What happens when a node dies
  2. Why shard recovery waits: delayed allocation
  3. Diagnosing unassigned shards with allocation explain
  4. How many replicas to run
  5. How writes reach replicas
  6. Check yourself

What happens when a node dies

A three-node cluster loses a node. Each primary that lived there has a replica on another node, so the master promotes those replicas to primaries within seconds. No data is lost and searches continue. Health turns yellow: every primary is available, but some shards now have fewer copies than configured. Then the cluster rebuilds the missing replicas on the remaining nodes.

Why shard recovery waits: delayed allocation

When a node leaves, the cluster doesn't immediately copy gigabytes of data. It waits for index.unassigned.node_left.delayed_timeout (1 minute by default), in case the node is just restarting. If it comes back, its existing shard copies are reused and only recent changes are synced. Raise the delay during planned maintenance; better still, for rolling restarts, set cluster.routing.allocation.enable to primaries first so the cluster doesn't start moving shards around.

When a node fails, replicas of its primaries are promoted, the cluster turns yellow, and after a delay new replicas are built on remaining nodes until it returns to green.
When a node fails, replicas of its primaries are promoted, the cluster turns yellow, and after a delay new replicas are built on remaining nodes until it returns to green.

Diagnosing unassigned shards with allocation explain

On the course's single-node lab, an index with one replica per shard:

Code
{'status': 'yellow', 'active_primary_shards': 3, 'active_shards': 3, 'unassigned_shards': 3}

index  shard prirep state      node
orders 0     p      STARTED    node-1
orders 0     r      UNASSIGNED
...

Ask the cluster why:

HTTP
GET /_cluster/allocation/explain
{ "index": "orders", "shard": 0, "primary": false }
Code
node-1 same_shard - a copy of this shard is already allocated to this node

The explain API names the decider that said no. Common reasons:

DeciderMeaningFix
same_shardReplica would sit on the same node as its primaryAdd nodes, or reduce replicas
disk_thresholdNode above the disk watermarkFree space, add nodes, delete old indices
awareness / filterAllocation rules exclude the nodeCheck zone awareness and allocation filters
max_retryAllocation failed repeatedly (corrupt shard, mapping issue)Fix the cause, then POST /_cluster/reroute?retry_failed=true

Red health means a primary is unassigned: data in that shard is unavailable. If no copy survives anywhere, the remedy is restoring from a snapshot (Day 27).

How many replicas to run

  • One replica is the usual minimum in production: any single node can fail.
  • More replicas add search throughput (each copy can serve queries) and survive more failures, at the cost of disk and indexing work.
  • Zone awareness: with cluster.routing.allocation.awareness.attributes: zone and a node.attr.zone on each node, primaries and replicas land in different availability zones, so a zone outage doesn't take both copies.
  • Three master-eligible nodes (in three zones) keep the cluster managed when one fails (Day 2).
  • Replicas protect against hardware loss, not against mistakes: a deleted index is deleted on every copy. That's what snapshots are for.

How writes reach replicas

An index request goes to the primary, which forwards it to replicas in parallel and acknowledges once in-sync copies have it. wait_for_active_shards can require a number of copies to be available before writing, a durability guard during degraded periods.

Check yourself

Replicas of those primaries on other nodes are promoted to primaries, so all data remains available for reads and writes. Only redundancy is reduced, since the lost copies haven't been rebuilt yet, which yellow signals.

The node may be restarting for a patch or reboot. index.unassigned.node_left.delayed_timeout, one minute by default, avoids copying large amounts of data to other nodes when the original will be back shortly.

No. Deleting an index removes every copy, primaries and replicas alike. Only snapshots taken before the deletion can recover the data.

Why a specific shard is unassigned or can't move to a node, with the reason per node, such as disk watermarks, allocation filters, too many copies on one node or no valid node. It turns guessing into a direct answer.

The primary shard indexes the document, then forwards the operation to all in-sync replicas in parallel, and acknowledges the client once they've confirmed. A replica that fails is removed from the in-sync set so writes don't stall.