Elasticsearch ILM and Hot-Warm-Cold Architecture
On this page
Why aging data needs lifecycle management
Today's logs are searched constantly during incidents. Last month's are queried occasionally. Last year's are kept for compliance. Storing all of them on fast SSDs with full replicas wastes money. Index lifecycle management (ILM) moves data through stages automatically.
Data streams and rollover
A data stream is a name you write to (logs-app-checkout) backed by a series of hidden indices. When the current backing index reaches a size or age, rollover creates a new one; older ones are never written to again, which makes them easy to shrink, move and delete.
POST /logs-app-checkout/_doc
{ "@timestamp": "2026-10-11T12:00:00Z", "level": "error", "message": "payment gateway timeout" }In the lab, the document landed in .ds-logs-app-checkout-2026.10.11-000001, the first backing index.
An ILM policy: hot, warm, cold and delete
PUT /_ilm/policy/logs-policy
{
"policy": {
"phases": {
"hot": { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "1d" } } },
"warm": { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 },
"forcemerge": { "max_num_segments": 1 } } },
"cold": { "min_age": "30d", "actions": { "set_priority": { "priority": 0 } } },
"delete": { "min_age": "90d", "actions": { "delete": {} } }
}
}
}- Hot: actively written and searched; roll over at 50 GB per primary shard or one day.
- Warm (after 7 days): read-only; shrink to one shard and merge segments to save resources.
- Cold (after 30 days): rarely searched; lowest recovery priority.
- Delete (after 90 days).
min_age counts from rollover, not from when the data was written. Phases move indices to matching data tiers automatically (data_hot, data_warm, data_cold nodes) when those exist.
Attaching ILM with an index template at the right priority
PUT /_index_template/logs-app
{
"index_patterns": ["logs-app-*"],
"priority": 200,
"data_stream": {},
"template": { "settings": { "index.lifecycle.name": "logs-policy" } }
}Testing this without priority produced a data stream managed by Elasticsearch's built-in logs policy instead: the built-in template for logs-*-* has priority 100 (Day 19). With priority 200, the explain API confirmed the right policy:
.ds-logs-app-checkout-2026.10.11-000001 policy= logs-policy phase= hotCheck GET /<index>/_ilm/explain whenever you set up a policy.
Hot, warm, cold and frozen data tiers
| Tier | Hardware | Data |
|---|---|---|
| Hot | Fast SSDs, more CPU | Current writes and frequent searches |
| Warm | Larger, cheaper disks | Read-only, occasionally searched |
| Cold | Cheaper still; can use searchable snapshots with fewer replicas | Rarely searched |
| Frozen | Searchable snapshots in object storage, cached locally on demand | Very rarely searched; slow but very cheap |
Searchable snapshots (cold and frozen) are a paid feature; hot-warm tiers with ILM work on the free license.
A simpler alternative: data stream lifecycle
For data streams that only need rollover and retention, the newer data stream lifecycle sets retention directly ("data_retention": "90d") and manages rollover automatically, without a separate ILM policy.
Check yourself
Older backing indices are no longer written to, so they can be shrunk, moved to cheaper tiers and deleted safely.
From the index's rollover, not from the time each document was written. Documents in an index that rolled over today are treated as day 0, even if some were written a week ago.
A built-in template matching logs-*-* had a higher priority than the custom template's default of 0, and only the highest-priority matching template applies. Its lifecycle policy won until the custom template's priority was raised.
A named stream for append-only time series data, backed by hidden indices that roll over automatically, with writes always going to the newest one.
Data stays searchable directly from snapshots in object storage, with only the parts you query cached locally on demand. It costs very little to keep years of data, but searches are much slower, so it suits rarely queried history.