Vector Search After the Hype: What We Got Right and What We Overbuilt

For a couple of years the answer to every retrieval question was the same answer. Someone would describe a search problem — a help centre, a set of internal documents, a product catalogue — and the design would arrive already decided: embed everything, put it in a vector database, retrieve the nearest neighbours.

Often the corpus was a few hundred documents and the previous system had been a keyword index that worked.

Enough of those have now been built, operated and quietly repaired that the picture has settled. Vector search is genuinely useful and it was oversold in a specific, correctable way. It's worth being precise about which parts held up.

What embeddings are good at, stated narrowly

Embeddings retrieve on meaning rather than on tokens. A user asking how to stop being charged every month finds the page titled "Cancelling your subscription", even though the two share almost no words. Description-to-thing, paraphrase, the question asked in the user's vocabulary rather than the author's — this is a real capability keyword search never had, and it's why the enthusiasm was warranted.

And what they're bad at, which surprised people

Exact tokens. Order numbers, error codes, SKUs, function names, configuration keys, version strings, people's surnames. An embedding works by smoothing meaning across a neighbourhood, and an identifier has no meaning to smooth — so a query for a specific error code returns documents that are broadly about errors, confidently and uselessly. A keyword index would have found the one document that contains the string.

Rarity is the underlying issue. Classical scoring rewards terms that are rare in the corpus, which is exactly the signal that makes an identifier findable. Dense retrieval has no equivalent notion, so the distinguishing term gets averaged away by the ordinary words around it. Negation suffers similarly: "invoices not yet paid" sits close to "invoices paid" in the embedding space, because they are on the same topic.

Hybrid isn't a compromise, it's the default

Running keyword and vector retrieval together and fusing the results is not a hedge for teams that couldn't decide. It's the configuration that wins, because the two methods fail on different queries — the identifier lookup that vectors miss is trivial for the keyword index, and the vocabulary-mismatch query that keyword search misses is exactly what vectors were built for.

Fusing by rank rather than by score is the practical detail worth knowing. The two systems produce numbers on incomparable scales that also drift as the corpus changes, so any attempt to blend the raw scores becomes a tuning exercise that has to be redone. Combining positions in each result list needs no calibration and holds up as the data moves.

Then rerank the fused candidates with a model that reads the query and passage together rather than comparing two independent vectors. That step is usually a larger quality improvement than switching embedding models, and it's the one most often skipped because it's an extra hop.

Chunking dominates model choice

Teams spend weeks comparing embedding models and an afternoon on how the text is split. The ordering should be reversed.

No embedding model recovers from a chunk that cuts the answer in half at a boundary, because the passage containing the answer no longer exists as a retrievable unit. And no model recovers from a chunk so large that its vector averages five unrelated topics into something bland enough to be moderately close to everything and clearly close to nothing.

What actually helps is unglamorous. Split along the document's own structure — headings, sections, list boundaries, whole code blocks — rather than at a fixed character count that lands mid-sentence. Prepend the document title and section path into the chunk's own text so a passage is self-describing, since a paragraph beginning "this is not supported in that configuration" is unretrievable without knowing what "this" is. Overlap adjacent chunks so an answer straddling a boundary appears intact in at least one. And store a pointer to the surrounding region so you can retrieve precisely but pass the model a wider window than the fragment you matched on.

If your corpus is tables, code or logs rather than prose, chunking isn't part of the problem — it is the problem, and generic splitters will not do it.

Do you actually need a vector store?

At modest corpus sizes, searching every vector is a matrix multiplication and it finishes in the time it takes to parse the request. Approximate indexes exist to avoid comparing against everything; when everything is not very much, they're solving a problem you don't have while adding an index to build, tune and rebuild.

Keeping vectors in the database that already holds the records buys something the standalone store struggles with: filtering. Real queries are rarely "find similar text" — they're "find similar text in documents this user may read, from this workspace, not archived". Alongside the row, that's a predicate. In a separate system it's a choice between fetching far more neighbours than you need and filtering afterwards, or replicating your permission model into the index and keeping it in sync.

A dedicated vector store earns its place at genuine scale — where the index no longer fits comfortably alongside the transactional workload, where query volume justifies scaling retrieval independently, or where you need to tune the recall-versus-latency trade-off precisely because retrieval quality is the product. Those are real situations. They're just not the situation most retrieval features are in.

The measurement we should have started with

The habit that separates retrieval systems that improve from ones that merely change is a fixed set of real queries with known correct passages, scored on one question: was the right passage in the candidate set at all?

Everything downstream depends on that number and cannot repair it. A model handed a set of passages that don't contain the answer will produce something fluent and wrong, and no amount of prompt work fixes a retrieval miss. Without this evaluation set, every retrieval change — a new model, a new chunk size, a reranker — is judged by trying three queries you already remember, which is how systems get worse while everyone feels good about it.

Where it clearly earns its place

Vague queries where the user doesn't know the vocabulary. Cross-lingual retrieval, where the query and the document share no words by construction. Deduplication and clustering of near-identical records. "More like this" recommendation over items with no useful text overlap. Multi-modal search, where there is no keyword index to fall back to.

That's a strong list. It's also a narrower one than "the way search works now", and the correction is mostly a demotion in status: vector search stopped being an architecture and became an index type. Treat it like an index — one retrieval signal among several, chosen because it beat the alternatives on your own queries — and it does its job well.