Catalog My learning Blog
LLM System Design Course: RAG, Vector Databases, Cost · Phase 3 · Lesson 14 · 8 min read

How to Choose an Embedding Model

On this page
  1. Learning Objective
  2. Concept Explanation
  3. Worked Example: Choosing a Model on Your Own Corpus, Not a Leaderboard
  4. Real-World Examples
  5. Review Questions & Answers

Learning Objective

Select an embedding model against the dimensions that actually matter - retrieval quality on your own corpus, dimensionality, maximum input length, language coverage, hosting model, and cost - and explain how dimension truncation and quantization trade storage and latency against recall.

Concept Explanation

Embedding model choice is usually made by picking whatever tops a public leaderboard, and that is the wrong procedure for a specific reason: leaderboard scores are averaged across many tasks and datasets, none of which is your corpus or your query distribution. A model that excels at scientific abstract retrieval may be mediocre on customer support tickets full of product names and abbreviations. The correct procedure is to build a small labelled evaluation set from your own domain, perhaps a hundred to three hundred real questions each paired with the chunk that genuinely answers it, and measure recall at k for two or three candidate models on your own data. This takes a day and routinely contradicts the leaderboard ordering.

Dimensionality is the parameter with the most direct systems consequence, because it multiplies straight through storage, index memory, and per-comparison distance cost. Higher dimensions can represent finer distinctions, but the returns diminish sharply and the cost does not: going from 384 to 1536 dimensions quadruples memory for a retrieval quality improvement that is often in the low single digits of percentage points on a specific corpus. Matryoshka Representation Learning, published in 2022, changed the shape of this tradeoff by training models so that meaning is concentrated in the leading dimensions, which means a 1536-dimension vector can be truncated to 512 or 256 and renormalized while retaining most of its quality. Several current commercial embedding models expose this as a dimensions parameter, and it turns dimensionality into a tunable rather than a permanent commitment, though the quality cost still has to be measured on your data rather than assumed.

Quantization attacks the same cost from a different angle by reducing the precision of each component rather than the number of components. Scalar quantization from 32-bit floats to 8-bit integers cuts memory by four with a small and usually recoverable quality loss. Binary quantization, reducing each dimension to a single bit and using Hamming distance for comparison, cuts memory by thirty-two and makes distance computation extraordinarily fast, at a quality cost that is real rather than negligible. The standard pattern that makes aggressive quantization viable is rescoring: search the compressed index to retrieve a generously sized candidate set, then recompute exact distances for those candidates against full-precision vectors kept on slower storage, and reorder. You get most of the memory saving and most of the accuracy, and it composes naturally with the re-ranking stage you already have.

Beyond quality and size, four constraints decide model choice in practice and are easy to overlook until they bite. Maximum input length caps your chunk size, and a model accepting 512 tokens forces smaller chunks than one accepting 8,000 regardless of what your chunking analysis preferred. Language coverage is binary rather than gradual: a model trained predominantly on English will produce poor and sometimes actively misleading similarities on other languages, and multilingual models additionally let a query in one language retrieve documents in another, which is either a feature or a surprise depending on whether you designed for it. Asymmetric models, trained with distinct query and document encoders or requiring a task prefix on one side, will underperform noticeably if you embed both sides identically, and this is a silent misconfiguration rather than an error. And hosting model - API or self-hosted - determines whether your corpus content leaves your infrastructure, which is frequently the decisive constraint in regulated environments regardless of quality.

Finally, remember from the embeddings lesson that changing this model is a full re-embed and index rebuild. That makes it one of the stickiest decisions in the system, and it argues for two habits. First, evaluate seriously before committing, because the cost of being wrong is paid in a migration rather than a config change. Second, build the migration path anyway: store the model identifier with every vector, keep the re-embedding job as maintained code rather than a one-off script, and be able to run a shadow index on a new model against production queries so the next decision can be made with measurements instead of arguments.

ConfigurationMemory per Million VectorsRelative Retrieval QualityNotes
1536 dimensions, float326.1 GBBaselineThe default nobody measured against alternatives
768 dimensions, float323.1 GBTypically a small dropTruncated from a Matryoshka-trained model and renormalized
1536 dimensions, int81.5 GBSmall drop, largely recoverableRescore the top candidates on full-precision vectors
768 dimensions, int80.8 GBModerate dropCombines both levers; measure before adopting
1536 dimensions, binary0.2 GBReal drop without rescoringViable only with a generous candidate set and rescoring

Worked Example: Choosing a Model on Your Own Corpus, Not a Leaderboard

This evaluation is a day of work and frequently overturns the obvious choice. Note that it measures recall at the candidate-set size the pipeline actually uses, not at ten, because a downstream re-ranker means position within the candidate set is not what matters.

Code
# 1. Build the eval set from REAL traffic, not synthetic questions.
#    ~200 (question, correct_chunk_id) pairs, labelled by someone who
#    knows the domain. This asset outlives every model you will try.
eval_set = load_labelled_pairs('eval/retrieval_v1.jsonl')

CANDIDATES = [
    ('embed-large',  1536, 8192, 'api',        0.13),   # $ per 1M tokens
    ('embed-small',   768, 8192, 'api',        0.02),
    ('bge-m3-local', 1024, 8192, 'self-host',  0.00),   # GPU cost instead
]

for name, dims, max_in, hosting, price in CANDIDATES:
    for truncate_to in [dims, 512, 256]:          # Matryoshka truncation
        index = build_index(corpus, model=name, dims=truncate_to)
        r50 = recall_at_k(index, eval_set, k=50)  # k = CANDIDATE SET size,
                                                  # not 10 - a reranker follows
        report(name, truncate_to, hosting,
               recall50=r50,
               memory_gb=len(corpus) * truncate_to * 4 / 1e9,
               ingest_cost=corpus_tokens * price / 1e6)

# Typical shape of the result, and why the leaderboard is not the answer:
#   embed-large @1536 : recall@50 0.947   6.1 GB   $$$
#   embed-large @512  : recall@50 0.941   2.0 GB   $$$    <- 0.6% for 3x memory
#   embed-small @768  : recall@50 0.938   3.1 GB   $      <- often the winner
#   bge-m3-local@1024 : recall@50 0.944   4.1 GB   GPU    <- wins if data
#                                                             cannot leave
# Decide on the recall/memory/cost/hosting surface, not on one number.
The embedding configuration tradeoff surface: dimensionality reduction through Matryoshka truncation and precision reduction through scalar and binary quantization plotted against memory and retrieval quality, with a rescoring stage recovering accuracy from a compressed index.
The embedding configuration tradeoff surface: dimensionality reduction through Matryoshka truncation and precision reduction through scalar and binary quantization plotted against memory and retrieval quality, with a rescoring stage recovering accuracy from a compressed index.

Real-World Examples

  • Matryoshka Representation Learning, published in 2022, trains embeddings so that the most important information is concentrated in the leading dimensions, allowing a single model to serve multiple embedding sizes by truncation. OpenAI's text-embedding-3 models expose exactly this as a dimensions parameter, which turns what used to be a permanent architectural commitment into a per-deployment tuning decision.
  • Cohere shipped int8 and binary embedding support for its Embed v3 models, reporting order-of-magnitude reductions in the memory required to hold a large corpus in a vector index, with the standard mitigation of retrieving a larger candidate set from the compressed index and rescoring those candidates against higher-precision vectors to recover ranking quality.
Best PracticeNever choose an embedding model from a public leaderboard alone. Leaderboard scores average across tasks and corpora that are not yours, and a model that ranks third overall regularly wins on a specific domain at a fraction of the cost and memory. Build a labelled evaluation set of a few hundred real question and chunk pairs from your own traffic, measure recall at your actual candidate-set size, and decide on the combined recall, memory, cost, and hosting surface.

Review Questions & Answers

An ordinary embedding distributes information across all dimensions with no particular ordering, so removing the trailing half discards an arbitrary and significant share of the representation. Matryoshka training adds an objective that forces the leading dimensions to carry a self-sufficient representation on their own, so prefixes of the full vector are themselves valid embeddings of decreasing fidelity. Truncating and renormalizing therefore produces a usable smaller vector rather than a damaged one, which is what makes dimensionality a tunable deployment parameter rather than a permanent commitment. The quality cost is still real and still has to be measured on the target corpus.

The compressed index is used only to produce a candidate set, which it can do quickly and with a small memory footprint, and its ranking errors are tolerable because the set is deliberately larger than the number of results needed. Exact distances are then recomputed for those candidates against full-precision vectors held on cheaper storage, and the candidates are reordered, which restores most of the lost ranking accuracy. The cost is a second lookup and the storage of the full-precision vectors somewhere, plus a slightly larger candidate set, all of which is far cheaper than keeping full-precision vectors resident in the index itself.

Some models are trained with distinct treatment for queries and documents, either through separate encoders or through a required task prefix on one side, because a short question and a long passage have different statistical shapes. If both sides are embedded identically the vectors are still produced, the similarities are still numerically valid, and no error is raised anywhere, but the query lands in a slightly wrong region of the space and retrieval quality degrades in a way that looks like a general model weakness. The only way to catch it is to read the model documentation and to verify against a labelled evaluation set that the configured usage actually outperforms the alternative.

Store the model identifier on every vector and filter on it at query time so two models can coexist in one table without ever being compared. Keep the re-embedding job as maintained, tested code rather than a one-off script, since it will be run again. Maintain the labelled evaluation set as a durable asset so a candidate model can be compared against the incumbent on the same basis. And support running a shadow index populated by a new model against real production queries, so the comparison is made on live traffic before any cutover. With those in place the migration is a scheduled backfill, and without them it is an outage-shaped project.