Retrieval is where most production AI systems spend their engineering effort, and it is taught almost entirely at the level of library calls. This chapter is the layer underneath: what the index is doing, what the tuning parameters cost, and which of the popular practices survive measurement.

The single most useful fact here is that recall is not linear in effort. Moving from 0.80 to 0.95 recall costs roughly 31% more latency; moving from 0.95 to 0.99 costs three to five times. That elbow is where essentially every vector-search cost decision lives, and the right question in a design review is not ‘can we get higher recall’ but ‘does our task actually need the last four points?’

• HNSW — a multi-layer navigable small-world graph traversed greedily. Best recall-per-millisecond when it fits in RAM, which is the constraint: the graph plus the vectors must be resident, giving a practical ceiling around 100 to 200 million vectors per node. Its two most important parameters are frozen at build time — M, the edges per node, and efConstruction. Only efSearch is tunable per query. And the graph alone, before any vectors, costs roughly 4 × M × N × 1.1 bytes — about 7 GB of edges at M=32 and 50 million rows.
• IVF-PQ — partition the space into cells, then product-quantise the residuals. Memory footprint is tiny because you store codes rather than vectors, at the cost of recall lost to compression. The right choice when the corpus will not fit in RAM and you can tolerate the loss.
• DiskANN / Vamana — the graph lives on SSD with compressed vectors in RAM for routing and full vectors read from disk on demand. Around 90% less RAM than HNSW, bounded by SSD IOPS rather than by memory. Reported results include billion-vector corpora at 95% recall in single-digit milliseconds on one node, which is the number that made single-node billion-scale retrieval practical.
• ScaNN — anisotropic vector quantisation, where the quantisation loss is weighted by its effect on the inner product rather than on reconstruction accuracy. That is the conceptually cleanest idea in the family: you are not trying to reconstruct the vector, you are trying to preserve the ranking.

BM25 and dense retrieval fail differently. Lexical search nails rare tokens, product codes, identifiers and exact quotes; dense retrieval handles paraphrase and synonymy. Combining them helps because the errors are orthogonal, which is also why the gain is largest out-of-domain.

Reciprocal rank fusion ignores scores entirely and fuses ranks: score(d) = Σr 1 / (k + rankr(d)). The original suggestion was k = 60; a published grid search on a standard benchmark suite found k = 20 optimal for their setup, so treat k as a tunable rather than a constant.

Measured on a standard benchmark suite: Reciprocal Rank Fusion (RRF) over lexical plus a sparse neural retriever gave about +1.4% over the neural retriever alone and +18% over lexical alone, while a properly tuned linear combination gave +6% and +24% respectively.

So the honest position is: a calibrated linear combination beats RRF, and RRF needs no calibration. Score distributions from a lexical and a dense retriever are not commensurable, and normalising them is fragile and corpus-specific. RRF sidesteps that by throwing away score magnitude. Choose RRF when you cannot afford to maintain a calibration, and a tuned combination when you can.

A cross-encoder jointly encodes the query and the document, so no document embedding can be cached and the cost is one forward pass per candidate. The canonical illustration: scoring 10,000 pairs takes hours with a cross-encoder against seconds to embed them with a bi-encoder. Hence the standard two-stage architecture — retrieve 50 to 100 cheaply, rerank those.

The measured contribution is enormously variable, and that variability is the honest headline: reported lifts range from around eighteen points of NDCG in one pipeline to about two points for 200 milliseconds of added latency in another. That is two orders of magnitude of disagreement.

The resolving insight: a reranker's lift is a property of how bad your first-stage retrieval is. If your embedding model is well matched to the domain, reranking adds little. If it is a generic model on specialist text, reranking rescues it. So the answer to “should we add a reranker” is always “measure it on our corpus”, and anyone quoting a general number is quoting their corpus rather than yours.

ColBERT-style late interaction stores one vector per token and scores with MaxSim: for each query token take the maximum dot product over all document tokens, then sum. This preserves token-level matching evidence that pooling into a single vector destroys, which is why it is strong out-of-domain and on long documents.

The cost is storage, and the arithmetic is stark. A million documents at one 768-dimension vector each is about 3 GB. The same million at 200 tokens each with 128-dimension token vectors is about 102 GB — or roughly 6 GB with aggressive residual compression, which is what makes it practical. Centroid-based candidate pruning brings query latency into the tens of milliseconds at scale with no measured quality loss.

Where this is most clearly winning in 2026 is visual document retrieval — scoring page images directly rather than running OCR first — because the token-level machinery maps naturally onto image patches, and it removes an entire lossy preprocessing stage.

This is the most consequential architectural question in applied AI right now, and there are three independent lines of evidence that the advertised window is not the usable window.

• Effective context length. A benchmark that defines effective length as the longest input at which a model stays above a quality threshold found that although every model tested claims at least 32K, only about half hold up at 32K — and a model advertising 128K frequently has an effective length around half of that.
• Associative rather than lexical retrieval. A benchmark that rewrites needle-in-a-haystack so the needle and the question share minimal wording — forcing association rather than string matching — found most tested models falling below 50% accuracy at quite modest lengths. Models that ace vanilla needle-in-a-haystack fail this, which means high needle-in-a-haystack scores are close to meaningless as evidence of long-context competence.
• Context rot. Measured degradation with position and distractor density. One reported example has a frontier model falling from over 99% at short context to around 70% by 32K, with many models dropping below half their own short-context baseline at that length.

The cost side reinforces it: attention is quadratic in prefill, and a cached million-token session is on the order of a hundred gigabytes of KV state, which is the arithmetic from chapter 1 applied at scale.