Elasticsearch CRUD and the Bulk API
On this page
Elasticsearch CRUD: index, get, update, delete
Every document lives in an index under an _id. The REST API maps cleanly to CRUD:
PUT /products/_doc/101 // create or replace with a chosen id
{ "name": "Wireless Mouse", "category": "electronics", "price": 24.99 }
POST /products/_doc // create with a generated id
{ "name": "USB-C Hub", "category": "electronics", "price": 39.99 }
GET /products/_doc/101 // read by id (real-time, even before a refresh)
POST /products/_update/101 // partial update
{ "doc": { "price": 22.99 } }
DELETE /products/_doc/101Use PUT /products/_create/101 when the document must not already exist: it fails with a conflict instead of overwriting.
Why updates rewrite the whole document
Lucene segments are immutable (Day 3), so an "update" fetches the stored _source, applies the change, indexes a complete new version and marks the old one deleted. Two consequences:
- Frequent small updates to large documents are expensive.
- Fields that change constantly (view counters, stock levels updated every second) are often better kept elsewhere or batched.
Scripted updates change values in place on the server:
POST /products/_update/101
{ "script": { "source": "ctx._source.sales += params.n", "params": { "n": 1 } } }The same idea works across many documents with _update_by_query. In the lab, renaming an author on every matching article took one request:
POST /articles/_update_by_query?refresh=true
{ "query": { "term": { "author": "ana" } },
"script": { "source": "ctx._source.author = params.name", "params": { "name": "ana.k" } } }updated 2 took_ms 62Optimistic concurrency with if_seq_no
Two processes read a product, change it and write it back; the second write silently overwrites the first. Every document carries a _seq_no and _primary_term; write only if they're unchanged:
PUT /products/_doc/101?if_seq_no=12&if_primary_term=1
{ "name": "Wireless Mouse", "price": 21.99 }If someone else wrote in between, the request fails with 409 Conflict, and you re-read and retry. _update with retry_on_conflict does this for you on partial updates.
The bulk API and partial failures
Indexing one document per request wastes time on round trips. The bulk API takes newline-delimited JSON: an action line, then (for index and create) the document:
{ "index": { "_id": "101" } }
{ "name": "Wireless Mouse", "category": "electronics", "price": 24.99 }
{ "index": { "_id": "102" } }
{ "name": "Wireless Keyboard", "category": "electronics", "price": 49.99 }
{ "delete": { "_id": "999" } }curl -s -X POST 'localhost:9200/products/_bulk' \
-H 'Content-Type: application/x-ndjson' --data-binary @products.ndjsonLoading the course's product file returned errors: False items: 12.
A bulk request can partly fail. The response's top-level errors flag is true if any item failed, and each item has its own status. Always check it: a mapping conflict on one document doesn't stop the others, and ignoring the flag silently loses data. Client libraries (the official Python, JavaScript, Java and .NET clients) include bulk helpers that batch, retry and report failures.
Practical sizing: batches of a few megabytes (often 1,000 to 5,000 documents), several in parallel, and back off when you get 429 Too Many Requests.
Refresh and when documents become searchable
New documents become searchable after the next refresh, every second by default (GET by id sees them immediately).
?refresh=trueforces a refresh: fine in tests, expensive under load.?refresh=wait_forwaits for the next scheduled refresh: use it when a user must see their own write.- For large initial loads, set
"refresh_interval": "-1"and"number_of_replicas": 0, then restore both afterwards.
Check yourself
Segments are immutable, so every update indexes a full new version of the document and marks the old one deleted.
Read the document's _seq_no and _primary_term, and write with if_seq_no and if_primary_term. If someone else changed it in between, the write fails with 409 Conflict and you can re-read and retry.
Not necessarily. The 200 only means the request was processed; check the top-level errors flag and each item's status, since individual items can fail with mapping errors or rejections while others succeed.
Typically 5 to 15 MB per request; benchmark to find the sweet spot. Very large requests use more memory and can cause rejections.
index creates or replaces the document; create fails if a document with that id already exists, which prevents accidental overwrites.