Catalog My learning Blog
Elasticsearch Tutorial: How Full-Text Search Works · Phase 4 · Lesson 25 · 3 min read

Elasticsearch Cluster Health Monitoring

On this page
  1. Why green isn't the same as healthy
  2. Elasticsearch metrics that matter
  3. Diagnosing a slow cluster
  4. Elasticsearch monitoring tools
  5. Elasticsearch alerts worth having
  6. Check yourself

Why green isn't the same as healthy

Cluster health only says whether shards are assigned (Day 2). A green cluster can have one node at 90% heap with constant garbage collection, a disk about to hit its watermark, or a search thread pool rejecting requests. Monitor the resources underneath.

Elasticsearch metrics that matter

SignalAPIWorry when
Cluster status, unassigned shardsGET /_cluster/healthYellow for long, or any red
Heap usedGET /_nodes/stats/jvmSustained above about 75 to 85%, or frequent long GC
Disk freeGET /_nodes/stats/fs, _cat/allocationApproaching the 85% low watermark
Thread pool queues and rejectionsGET /_cat/thread_pool/search,write?vAny growing rejected count
Search and indexing latencyGET /_nodes/stats/indices (query and index time totals)Rising per-operation time
Pending cluster tasksGET /_cluster/pending_tasksA growing queue: the master is overloaded
Circuit breaker tripsGET /_nodes/stats/breakerAny trips: requests rejected to avoid running out of memory

The lab node's heap and disk, from node stats:

HTTP
GET /_nodes/stats/jvm,fs
Code
node-1 heap_used_percent=17 disk_free=50%

Fine for a lab. In production, track these per node over time; one hot node in an otherwise calm cluster usually means uneven shard placement or one expensive query pattern.

Cluster health covers shard assignment; node-level heap, disk, thread pools and latency show whether the cluster can keep up, and slow logs point to the queries responsible.
Cluster health covers shard assignment; node-level heap, disk, thread pools and latency show whether the cluster can keep up, and slow logs point to the queries responsible.

Diagnosing a slow cluster

Searches got slow at 14:00 and the cluster is green.

  1. Which nodes? GET /_nodes/stats/jvm shows one node at 88% heap, the others near 50%.
  2. What's on it? GET /_cat/shards?v&s=store:desc shows it holds several of the largest shards: placement is uneven.
  3. What's running? GET /_tasks?actions=*search&detailed lists long-running searches; GET /_nodes/hot_threads shows what threads are busy.
  4. Which queries? Enable the search slow log on the index (index.search.slowlog.threshold.query.warn: 2s), and look for deep aggregations on high-cardinality fields, leading wildcards or huge size values.
  5. Fix: rebalance or split the large shards, fix the query, add capacity. For sustained thread pool rejections on writes, slow down bulk clients (back off on 429) rather than raising queue sizes.

Elasticsearch monitoring tools

  • Stack Monitoring in Kibana, or Elastic Agent shipping metrics to a separate monitoring cluster (don't monitor a cluster only from inside itself).
  • Prometheus via an Elasticsearch exporter, with Grafana dashboards and alert rules (see the observability course).
  • AutoOps (Elastic Cloud) and similar services analyze clusters and suggest fixes.

Elasticsearch alerts worth having

  • Health red (page) or yellow for over 15 minutes.
  • Disk above the low watermark on any node.
  • Heap above 85% for 10 minutes.
  • Any thread pool rejections on search or write.
  • p99 search latency above your target, measured from the application's side.

Check yourself

Green only means shards are assigned; heap, disk, thread pools and latency can all be problems while every shard is placed.

The node can't keep up and is refusing work; clients must back off and the cause (load, uneven shards, expensive queries) needs fixing.

If the production cluster is struggling or down, monitoring stored inside it fails at exactly the moment you need it, and the monitoring load adds to the problem.

Cluster status not green, disk above the low watermark, high heap and long GC pauses, thread pool rejections, and rising search or indexing latency.

It records searches and indexing operations that exceed configured time thresholds, with the query and the shard. That shows exactly which queries need optimizing, instead of guessing from cluster-wide latency.