Embeddings & Vector Search

The search lesson in the high-level module solved a specific problem: find documents containing these words. Inverted indexes do that brilliantly.

They cannot do this: a user searches "how do I stop my laptop overheating", and the best article is titled "Reducing thermal throttling on portable devices". Not one word matches. Keyword search returns nothing useful.

The article means the same thing. Matching by meaning rather than by words is what embeddings are for.

What an embedding is

An embedding model converts text into a list of numbers — a vector, typically several hundred to a few thousand dimensions.

The vector is not arbitrary. The model is trained so that texts with similar meaning produce nearby vectors. "Dog" lands near "puppy", further from "canine legislation", and far from "quarterly revenue".

You can now do something that was previously impossible: measure semantic similarity as distance in space. Finding related text becomes finding nearby points.

"laptop overheating"     -> [0.21, -0.88, 0.13, ...]
"thermal throttling"     -> [0.19, -0.85, 0.15, ...]   <- close
"quarterly revenue"      -> [-0.62, 0.31, 0.77, ...]   <- far

Cosine similarity is the usual distance measure. It compares the angle between vectors rather than their magnitude, which suits text because a long document and a short one about the same topic should count as similar.

Embeddings work on more than text — images, audio and video embed too, which is what enables searching photos with a written description.

Why exact search does not scale

Given a query vector, the exact answer is simple: compute the distance to every stored vector and keep the closest.

With 10,000 documents that is fine. With 50 million, each search compares against 50 million vectors of maybe 1,000 dimensions each — billions of arithmetic operations for a single query. At any real query rate this is hopeless.

And you cannot index your way out with the structures you know. A B-tree sorts on one dimension; the spatial indexing lesson showed that even two dimensions need special handling. At a thousand dimensions, the geometry stops cooperating entirely — a phenomenon usually called the curse of dimensionality, where distances between points become increasingly similar and partitioning stops helping.

Approximate nearest neighbour

The escape is to give up exactness.

ANN algorithms find almost the closest vectors, very fast. You accept perhaps 95–99% of the true results in exchange for searching in milliseconds instead of seconds.

This trade is almost always correct, and it is worth understanding why: you are feeding these results to a model that will read several of them. Missing the fifth-best passage out of ten changes the final answer far less than a two-second delay changes the product.

Two families dominate.

HNSW builds a layered graph. Upper layers hold few nodes with long-range links; lower layers are dense and local. A search starts at the top, moves greedily towards the query, and descends — like finding an address by taking the motorway, then main roads, then streets. Excellent recall and speed; the cost is memory, since the graph lives in RAM.

IVF clusters vectors and searches only the nearest few clusters. Cheaper on memory, and quality depends on how many clusters you probe — a tunable dial between speed and recall.

Both are frequently combined with quantization to compress vectors, trading a little accuracy for a large reduction in memory.

Do you need a vector database?

The honest answer is often no, and it is worth resisting the default.

Postgres with pgvector handles hundreds of thousands to a few million vectors comfortably. That is enough for most internal search, documentation assistants and support tools. You keep transactions, joins, existing backups and one system to operate.

A dedicated vector database earns its place at tens of millions of vectors and above, where you need purpose-built index tuning, distributed sharding of the index, and high query throughput.

The strong argument for starting with Postgres is metadata filtering. Real queries are rarely pure similarity — they are "similar to this, and owned by this tenant, and not archived, and newer than January". Combining vector similarity with structured predicates is natural where your relational data already lives, and a recurring source of friction when your vectors live somewhere else.

Operational realities

Embeddings must be regenerated when you change models. Vectors from different models are not comparable — mixing them silently produces nonsense rather than an error. Changing your embedding model means re-embedding your whole corpus, so treat the choice as semi-permanent and record which model produced each vector.

Dimensionality is a cost decision. More dimensions generally means better quality and proportionally more memory and slower search. Many modern models support truncating vectors to shorter lengths with modest quality loss, which is a cheap lever worth testing.

Keeping the index in sync is the same problem as the search lesson: avoid dual writes, drive updates from a change stream, and expect eventual consistency.

Cost has two parts. Embedding is a one-off charge per document plus a small charge per query; storage and serving is continuous. At scale the memory bill for a large in-memory index usually dominates the embedding bill.

Check your understanding

Common mistakes

Using vector search alone. It handles paraphrase well and exact identifiers badly. Combine with keyword search.

Mixing embeddings from different models. No error, just meaningless results.

Choosing a vector database before measuring. Postgres covers a surprising amount of ground with far less operational cost.

Ignoring recall. Speed tuning silently degrades quality, and nothing in your dashboards will show it.

Embedding enormous chunks. A whole document compresses to one vector that represents everything and matches nothing precisely. Chunking is covered in the retrieval lesson.

Treating similarity scores as absolute. A score of 0.8 is not "80% relevant" — thresholds must be calibrated per model and per corpus.

Practice

Key takeaways

• Embeddings turn text into vectors positioned so that similar meaning is nearby, making semantic search a distance problem.
• Semantic similarity is not intent — negation, identifiers and exact terms are handled poorly, which is why hybrid retrieval exists.
• Exact nearest-neighbour search is infeasible at scale, and high dimensionality defeats conventional indexes.
• ANN trades a few percent of recall for orders of magnitude in speed, which is the right trade when a model reads the results anyway.
• HNSW gives excellent recall at high memory cost; IVF is cheaper with a tunable dial. Watch recall@k, not just latency.
• Postgres with pgvector is enough for most systems; dedicated stores earn their place in the tens of millions.
• Filtered search is the requirement most likely to catch you out — post-filtering starves small tenants.
• Changing embedding model means re-embedding everything; vectors from different models are incomparable.

Further reading

• Jeff Johnson, Matthijs Douze and Hervé Jégou, "Billion-scale similarity search with GPUs" (IEEE Big Data, 2019) — the FAISS paper.
• Yu Malkov and Dmitry Yashunin, "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs" (IEEE TPAMI, 2018).
• Gordon Cormack, Charles Clarke and Stefan Buettcher, "Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods" (SIGIR 2009).
• Nils Reimers and Iryna Gurevych, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks" (EMNLP 2019).
• Omar Khattab and Matei Zaharia, "ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT" (SIGIR 2020).