Catalog My learning Blog
Elasticsearch Tutorial: How Full-Text Search Works · Phase 1 · Lesson 6 · 3 min read

Elasticsearch Analyzers and Tokenizers Explained

On this page
  1. What an analyzer does
  2. Character filters, tokenizers and token filters
  3. Standard vs english analyzer with the analyze API
  4. The trade-off stemming brings
  5. Building a custom analyzer with synonyms
  6. Language analyzers, edge n-grams and normalizers
  7. Check yourself

What an analyzer does

When a document is indexed, its text fields go through an analyzer that produces terms. When a query runs, the query text goes through the same analyzer. A match happens when the terms line up. Change the analyzer and you change what users can find.

Character filters, tokenizers and token filters

  1. Character filters clean raw text: strip HTML, map characters (& to and).
  2. Tokenizer splits text into tokens: on whitespace and punctuation, into n-grams, or by language rules.
  3. Token filters modify tokens: lowercase, remove stop words, stem (running to run), add synonyms, fold accents (café to cafe).

Standard vs english analyzer with the analyze API

The same sentence through two built-in analyzers on the lab cluster:

HTTP
POST /_analyze
{ "analyzer": "standard", "text": "The Quick-Running Foxes!" }
Code
['the', 'quick', 'running', 'foxes']
HTTP
POST /_analyze
{ "analyzer": "english", "text": "The Quick-Running Foxes!" }
Code
['quick', 'run', 'fox']

The standard analyzer splits on word boundaries and lowercases. The english analyzer also removes stop words (the) and stems, so running matches run and foxes matches fox. In the lab, a search for laptops in English-analyzed product descriptions found "Budget Laptop", while the standard-analyzed article titles kept databases and database apart.

Text passes through character filters, a tokenizer and token filters to become index terms; queries are analyzed the same way so that equivalent words produce the same terms.
Text passes through character filters, a tokenizer and token filters to become index terms; queries are analyzed the same way so that equivalent words produce the same terms.

The trade-off stemming brings

Stemming improves recall but changes words in surprising ways:

Code
english analyzer: "wireless mouse wireles mose"  ->  ['wireless', 'mous', 'wirel', 'mose']

mouse becomes mous, and a misspelled wireles becomes wirel. Day 11 shows how this breaks fuzzy matching on a stemmed field. A common solution: index the field twice with a multi-field, once stemmed and once with the standard analyzer, and search both.

Building a custom analyzer with synonyms

Product search that folds accents, handles synonyms and stems:

HTTP
PUT /shop
{
  "settings": {
    "analysis": {
      "filter": {
        "shop_synonyms": { "type": "synonym_graph", "synonyms": ["laptop, notebook", "tv, television"] }
      },
      "analyzer": {
        "shop_index": {
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "porter_stem"]
        },
        "shop_search": {
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "shop_synonyms", "porter_stem"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": { "type": "text", "analyzer": "shop_index", "search_analyzer": "shop_search" }
    }
  }
}

Synonyms applied at search time (search_analyzer) can be updated without reindexing, using a synonyms set and reloading analyzers. Test any analyzer against an index with POST /shop/_analyze {"analyzer": "shop_search", "text": "Café Notebook"}.

Language analyzers, edge n-grams and normalizers

  • Language analyzers (english, french, german, and about 30 more) and the ICU plugin for Unicode-aware tokenizing.
  • edge_ngram for prefix matching (autocomplete), and search_as_you_type fields that build these for you.
  • Normalizers for keyword fields: lowercase and fold accents on exact values, so Ana and ana match as keywords.

Check yourself

Matching compares terms. If a query produces different terms than indexing did, for example running at search time against run in the index, documents that should match won't. Match queries use the field's search analyzer automatically.

It split on punctuation, lowercased, removed the stop word the and stemmed the rest, producing quick, run and fox. That's why searches for fox or running now match.

Search-time synonyms can be changed by updating a synonyms set or reloading the analyzer, without reindexing every document. Index-time synonyms are baked into the index and need a full reindex to change.

An analyzer for keyword fields that can lowercase or fold accents without splitting into tokens, so exact matching becomes case-insensitive.

Indexing prefixes of each word, like l, la, lap, so search-as-you-type matches partial words quickly, at the cost of a larger index.