What is Vector Index?
A vector index is the data structure that makes similarity search over stored embeddings fast, by organizing vectors so a query can skip most of the collection. Without one, a search must compare the query against every vector. The index type chosen sets the achievable balance between query speed, memory footprint, accuracy, and update cost.
Index families differ in how they prune the search space. Flat indexes store vectors as a plain list and scan everything, which is exact but slow beyond modest sizes. Cluster based indexes group vectors and probe only the nearest groups. Graph based indexes link each vector to close neighbors and navigate toward the query. Quantized indexes compress vectors into short codes so far more of them fit in memory.
Each family has distinct operational behavior. Flat indexes need no build step and handle constant updates well. Cluster based indexes require training on a sample before use, and their quality degrades if the data distribution shifts far from that sample. Graph indexes build incrementally but grow memory with link count. Quantized indexes save memory at a measurable accuracy cost that must be checked against real queries.
Indexes rarely stand alone in production. They are usually paired with metadata filters, so a query can be restricted to one tenant, language, or date range, and the interaction between filtering and index traversal matters: filtering after the search can return too few results, while filtering during traversal costs more per candidate but keeps result counts stable.
Choosing an index is an empirical exercise, not a matter of picking the newest option. The usual method is to hold out a set of real queries with known relevant documents, then measure recall, latency at a high percentile, memory use, and rebuild time for a few candidate configurations. Collections also change over time, so the winning configuration should be rechecked as the corpus grows.
Key points
- Turns similarity search from a full scan into a partial one
- Flat, cluster, graph, and quantized families trade off differently
- Interacts with metadata filtering in ways that affect result counts
- Choice should be measured on real queries, not assumed
- Rebuild and update cost matter as much as query latency
In practice
A team starts with a flat index over eighty thousand policy paragraphs, which is exact and fast enough. After a year the collection reaches three million paragraphs and median query time crosses a second. They switch to a graph index, measure recall against the old exact results on two hundred saved queries, confirm it stays above ninety-seven percent, and accept the roughly forty times speedup.