What Are Embeddings? Vectors Explained
On this page
Learning Objective
Explain what an embedding vector is, what "semantic similarity" concretely means in terms of cosine distance in a high-dimensional space, why embeddings from different models are mutually incomparable, and what embeddings reliably capture versus what they systematically miss.
Concept Explanation
An embedding is a fixed-length array of floating point numbers produced by running text through a model whose training objective pushed semantically related inputs toward nearby points in the output space. A typical dimensionality is a few hundred to a few thousand: 384 for a small sentence-transformer, 768 or 1024 for mid-size models, 1536 or 3072 for the larger commercial embedding models. The individual numbers are not interpretable. There is no dimension 412 that means "is about finance". The only thing that has meaning is the geometric relationship between vectors, which is why every operation you will perform on embeddings is a distance or similarity computation rather than an inspection of components.
Cosine similarity is the standard measure, and it deliberately ignores magnitude by measuring only the angle between two vectors. This matters because vector magnitude in most embedding models correlates with things you do not care about, such as text length and token frequency, while direction carries the semantic content. Many production embedding models emit unit-normalized vectors already, in which case cosine similarity and the dot product are identical and Euclidean distance becomes a monotonic function of both, which is why you will see vector databases offer all three metrics and why choosing the wrong one for a given model is a subtle and common misconfiguration. The important discipline is to use the distance metric the model was trained with, which its documentation states.
The single most important operational property of embeddings is that they are only comparable within one model. Vectors from two different embedding models occupy entirely unrelated coordinate systems, even at identical dimensionality, so a cosine similarity between them is arithmetic that produces a number with no meaning. The direct consequence is that changing your embedding model is not a configuration change, it is a full corpus re-embedding and index rebuild, and any system that mixes vectors from two model versions in one index is silently broken in a way no exception will ever reveal. Version the embedding model explicitly alongside the index, store the model identifier with every vector, and treat a model upgrade as a migration with a dual-write or shadow-index period.
It is equally important to be precise about what embeddings capture. They are strong at topical and paraphrase similarity: "how do I reset my password" and "I forgot my login credentials" land close together despite sharing almost no words, which is exactly why they beat keyword search for natural-language questions. They are notably weaker at several things engineers keep expecting from them. Negation is poorly represented, so "the drug is safe for children" and "the drug is not safe for children" can be uncomfortably close. Exact identifiers such as error codes, SKUs, part numbers, and rare proper nouns are often better matched by literal keyword search, because an embedding compresses them into a general semantic neighborhood rather than preserving the exact string. Numerical and temporal reasoning is essentially absent: nothing in the geometry encodes that 2019 is before 2024. These gaps are the entire justification for hybrid retrieval, covered later in this course.
Finally, embeddings are computed over a bounded amount of text, and that bound is usually far smaller than the generation model's context window. This is the mechanical reason chunking exists as a design problem at all: you cannot embed a 200-page manual as one vector and expect useful retrieval, both because the model will truncate it and because a single vector averaging 200 pages of content is close to nothing in particular. Every embedding is a lossy compression of its input into a fixed number of dimensions, and the longer the input, the more that compression blurs. Understanding embeddings as lossy fixed-size summaries, rather than as some exact semantic address, is what makes the chunking and retrieval tradeoffs in the next phase of this course make sense.
| Embeddings Handle Well | Why | Embeddings Handle Poorly | Mitigation |
|---|---|---|---|
| Paraphrase and topical similarity | Training objective pulls semantically related text together | Negation and polarity flips | Re-ranking with a cross-encoder that reads both texts jointly |
| Cross-vocabulary matching | No shared words required for a close match | Exact identifiers, SKUs, error codes | Hybrid retrieval with BM25 keyword scoring alongside vectors |
| Multilingual matching in multilingual models | Shared semantic space across languages | Numeric and temporal comparison | Metadata filters on real typed columns, not the vector |
Worked Example: Comparing Two Vectors, and the Migration a Model Change Forces
The arithmetic is short enough to be worth seeing once, and the second half shows why versioning the model alongside the index is not optional.
import numpy as np
def cosine(a, b):
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
q = embed('how do I reset my password') # model: text-embed-v2, dim 1536
d1 = embed('I forgot my login credentials') # -> cosine(q, d1) ~ 0.86 strong, no shared words
d2 = embed('password rotation policy for staff')# -> cosine(q, d2) ~ 0.71 topical, wrong intent
d3 = embed('how to reset a circuit breaker') # -> cosine(q, d3) ~ 0.42 lexical overlap, unrelated
# The failure that no exception will ever report:
old = embed_with('text-embed-v1', doc) # dim 1536
new = embed_with('text-embed-v2', doc) # dim 1536, ENTIRELY different space
cosine(old, new) # ~0.1 - a meaningless number, not a "low similarity"
# So the schema carries the model identity, and search filters on it:
# CREATE TABLE chunks (
# id BIGSERIAL PRIMARY KEY,
# doc_id BIGINT NOT NULL,
# embed_model TEXT NOT NULL, -- never mix models in one result set
# embedding VECTOR(1536) NOT NULL
# );
# SELECT id FROM chunks
# WHERE embed_model = 'text-embed-v2'
# ORDER BY embedding <=> $1 LIMIT 10;Real-World Examples
- Spotify open-sourced Annoy (Approximate Nearest Neighbors Oh Yeah), the library it built to serve music recommendations by finding nearby items in an embedding space, and it remains one of the clearest public examples that embedding-based similarity search was a production recommendation technique years before the current wave of LLM applications made vector search a mainstream infrastructure category.
- The Massive Text Embedding Benchmark (MTEB), released by Hugging Face researchers in 2022 and maintained as a public leaderboard, evaluates embedding models across retrieval, clustering, classification, and reranking tasks and repeatedly shows that no single model wins everywhere, which is direct evidence for treating embedding model choice as a task-specific engineering decision rather than picking whichever model tops a general leaderboard.
Review Questions & Answers
Cosine similarity measures only the angle between vectors and ignores magnitude, which is desirable because magnitude in most embedding models correlates with incidental properties like text length and token frequency rather than with meaning, while direction carries the semantic content. The distinction stops mattering when the model emits unit-normalized vectors, since for unit vectors cosine similarity and dot product are identical and Euclidean distance is a monotonic transformation of both, meaning all three produce the same ranking. The rule is to use whatever metric the model documentation specifies, because a mismatch degrades ranking quality silently.
Each embedding model defines its own coordinate system, and there is no meaningful correspondence between the spaces of two different models even when the dimensionality matches. A query embedded with the new model compared against a corpus embedded with the old one produces numerically valid but semantically meaningless similarities, and nothing in the system raises an error. Changing models therefore requires re-embedding the entire corpus and rebuilding the index, typically with a shadow index or dual-write period so that queries and stored vectors are never served from different model versions at the same time.
Embedding training objectives reward topical and paraphrase proximity, and a negated sentence shares nearly all of its topical content with its affirmative counterpart, so the two land close together in the space. Exact identifiers such as error codes and part numbers are compressed into a general semantic neighborhood rather than preserved as exact strings, so a near-miss identifier can score higher than the exact one. Both gaps argue for hybrid retrieval, where lexical scoring such as BM25 runs alongside vector search to catch exact-token matches, and for a re-ranking stage where a cross-encoder reads the query and candidate together and can therefore represent the relationship between them rather than just their separate positions.
An embedding is a lossy compression of its input into a fixed number of dimensions, so the amount of information it can retain does not grow with input length. A vector summarizing 200 pages is an average over many unrelated topics and ends up close to nothing in particular and moderately close to almost everything, which destroys the discriminative power that ranking depends on. Chunking exists to keep each vector focused on a small enough span of text that its position in the space actually corresponds to a specific idea, which is a separate concern from any hard input length limit the embedding model also imposes.