Catalog My learning Blog
Elasticsearch Tutorial: How Full-Text Search Works · Phase 4 · Lesson 24 · 3 min read

Elasticsearch ILM and Hot-Warm-Cold Architecture

On this page
  1. Why aging data needs lifecycle management
  2. Data streams and rollover
  3. An ILM policy: hot, warm, cold and delete
  4. Attaching ILM with an index template at the right priority
  5. Hot, warm, cold and frozen data tiers
  6. A simpler alternative: data stream lifecycle
  7. Check yourself

Why aging data needs lifecycle management

Today's logs are searched constantly during incidents. Last month's are queried occasionally. Last year's are kept for compliance. Storing all of them on fast SSDs with full replicas wastes money. Index lifecycle management (ILM) moves data through stages automatically.

Data streams and rollover

A data stream is a name you write to (logs-app-checkout) backed by a series of hidden indices. When the current backing index reaches a size or age, rollover creates a new one; older ones are never written to again, which makes them easy to shrink, move and delete.

HTTP
POST /logs-app-checkout/_doc
{ "@timestamp": "2026-10-11T12:00:00Z", "level": "error", "message": "payment gateway timeout" }

In the lab, the document landed in .ds-logs-app-checkout-2026.10.11-000001, the first backing index.

An ILM policy: hot, warm, cold and delete

HTTP
PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
      "warm":   { "min_age": "7d",  "actions": { "shrink": { "number_of_shards": 1 },
                                                  "forcemerge": { "max_num_segments": 1 } } },
      "cold":   { "min_age": "30d", "actions": { "set_priority": { "priority": 0 } } },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}
  • Hot: actively written and searched; roll over at 50 GB per primary shard or one day.
  • Warm (after 7 days): read-only; shrink to one shard and merge segments to save resources.
  • Cold (after 30 days): rarely searched; lowest recovery priority.
  • Delete (after 90 days).

min_age counts from rollover, not from when the data was written. Phases move indices to matching data tiers automatically (data_hot, data_warm, data_cold nodes) when those exist.

New data is written to hot nodes; rollover starts new backing indices, and ILM moves older ones to warm, cold and frozen tiers on cheaper hardware before deleting them.
New data is written to hot nodes; rollover starts new backing indices, and ILM moves older ones to warm, cold and frozen tiers on cheaper hardware before deleting them.

Attaching ILM with an index template at the right priority

HTTP
PUT /_index_template/logs-app
{
  "index_patterns": ["logs-app-*"],
  "priority": 200,
  "data_stream": {},
  "template": { "settings": { "index.lifecycle.name": "logs-policy" } }
}

Testing this without priority produced a data stream managed by Elasticsearch's built-in logs policy instead: the built-in template for logs-*-* has priority 100 (Day 19). With priority 200, the explain API confirmed the right policy:

Code
.ds-logs-app-checkout-2026.10.11-000001 policy= logs-policy phase= hot

Check GET /<index>/_ilm/explain whenever you set up a policy.

Hot, warm, cold and frozen data tiers

TierHardwareData
HotFast SSDs, more CPUCurrent writes and frequent searches
WarmLarger, cheaper disksRead-only, occasionally searched
ColdCheaper still; can use searchable snapshots with fewer replicasRarely searched
FrozenSearchable snapshots in object storage, cached locally on demandVery rarely searched; slow but very cheap

Searchable snapshots (cold and frozen) are a paid feature; hot-warm tiers with ILM work on the free license.

A simpler alternative: data stream lifecycle

For data streams that only need rollover and retention, the newer data stream lifecycle sets retention directly ("data_retention": "90d") and manages rollover automatically, without a separate ILM policy.

Check yourself

Older backing indices are no longer written to, so they can be shrunk, moved to cheaper tiers and deleted safely.

From the index's rollover, not from the time each document was written. Documents in an index that rolled over today are treated as day 0, even if some were written a week ago.

A built-in template matching logs-*-* had a higher priority than the custom template's default of 0, and only the highest-priority matching template applies. Its lifecycle policy won until the custom template's priority was raised.

A named stream for append-only time series data, backed by hidden indices that roll over automatically, with writes always going to the newest one.

Data stays searchable directly from snapshots in object storage, with only the parts you query cached locally on demand. It costs very little to keep years of data, but searches are much slower, so it suits rarely queried history.