Elasticsearch in Production: Common Pitfalls
On this page
The same production mistakes, again and again
Elasticsearch is forgiving at first: index some JSON and search works. The costs of early shortcuts appear months later, as reindexing projects, slow clusters and surprised security reviews. Every pitfall below showed up in this course's lab or is among the most common causes of production trouble.
Pitfalls seen in this course
| Pitfall | What happened | Prevention |
|---|---|---|
| Wrong field type | A text SKU couldn't be looked up exactly (Day 19) | Explicit mappings; keyword for identifiers; "dynamic": "strict" (Day 5) |
| term on text fields | An exact title search returned nothing (Day 8) | match for text, term for keyword |
| Stemming versus fuzziness | A fuzzy search for "wireles mose" found nothing on a stemmed field (Day 11) | Fuzzy and autocomplete on lightly analyzed subfields |
| Object arrays cross-matching | "Ana gave 5 stars" matched wrongly (Day 17) | nested where conditions must apply within one object |
| best_fields on split names | "Smith Jones" outranked "John Smith" (Day 15) | cross_fields for entities spread across fields |
| Deep from/size | Page 1,000 hit the 10,000 result limit (Day 13) | search_after with a point in time |
| Replicas on one node | A yellow cluster with unassigned replicas (Day 2) | Enough nodes, or zero replicas only in development |
| Template priority | A data stream got the built-in policy (Day 24) | Set priority above 100 for logs-*-* style names, and check _ilm/explain |
| Paid-only features | RRF and field- and document-level security were refused on the Basic license (Days 21, 26) | Check the subscription matrix before designing around a feature |
Other common Elasticsearch production pitfalls
- Oversharding: thousands of tiny shards from daily indices with many shards (Day 22).
- Heap set wrong: above 31 GB, or more than half of RAM, starving the file cache.
- Mapping explosion: dynamic keys (user IDs as field names) creating thousands of fields; use
flattenedor a key-value structure. - Using Elasticsearch as the primary database: no transactions, near real-time visibility, and indices you may need to rebuild. Keep the source of truth elsewhere (Day 18).
- Ignoring bulk errors: a 200 response with
"errors": truesilently loses documents (Day 7). - No snapshots, or untested restores (Day 27).
- Security off "just for now", or ports exposed to the internet (Day 26).
- Unbounded queries from users: leading wildcards, huge
size, deep aggregations on high-cardinality fields; set limits andsearch.allow_expensive_queries: falseif users can send raw queries.
Elasticsearch pre-production checklist
Mapping and data
[ ] Explicit mappings, "dynamic": "strict" for application data
[ ] text vs keyword reviewed for every field; multi-fields where needed
[ ] nested only where cross-object matching matters
[ ] Database remains the source of truth; reindex path documented and timed
Queries
[ ] Yes/no conditions in filter context
[ ] Deep pagination uses search_after; page depth capped in the UI
[ ] Relevance tuned against a judgment list (_rank_eval)
Capacity
[ ] Shard size 10-50 GB; rollover for time-based data
[ ] Heap <= 50% of RAM and below ~31 GB; SSDs for hot data
[ ] Load-tested with realistic data and queries
Resilience
[ ] 3 master-eligible nodes across zones; >= 1 replica; zone awareness
[ ] SLM snapshots to a locked-down repository; restore tested and timed
[ ] Aliases in front of every index clients use
Security and operations
[ ] Security on, TLS everywhere, no public ports
[ ] Least-privilege roles and per-application API keys
[ ] Monitoring from a separate cluster; alerts on health, heap, disk, rejections
[ ] Slow logs enabled with sensible thresholdsCheck yourself
It lacks multi-document transactions, is near real-time rather than immediately consistent, and indices sometimes must be rebuilt for mapping changes. A database should hold the authoritative data so the index can always be recreated from it.
Stemming changes misspelled words too, so the query term ends up more than the allowed edit distance from the stored terms. Run fuzzy queries against a lightly analyzed subfield instead.
Some capabilities, such as reciprocal rank fusion and document- or field-level security, are refused on the free Basic license. Finding out after building means redesigning or paying.
Dynamic mapping's first guesses become permanent for that index, and fixing a wrong type later means reindexing everything. A template applies the intended mapping to every new index, including rolled-over ones.
Explicit mappings, sensible shard sizes, security enabled, snapshots tested with a restore, monitoring with alerts, aliases in front of indices, and queries load-tested at realistic volume. Each item prevents a common production incident.