Corpus Embedding & Indexing: 2026 Production Playbook

Decision: For most RAG pipelines: recursive 512-token chunks [4], text-embedding-3-small or Qwen3-Embedding-0.6B as your starting model [3], HNSW index up to ~100M vectors, IVF or IVF-PQ above that [7], and BM25+dense hybrid with RRF fusion when exact-keyword recall matters [9]. Chunking quality constrains retrieval accuracy more than model choice [4]. Every embedder swap forces a full re-index — re-embed only on model switches or chunking restructures [16].

Embedding Models

The MTEB quality curve flattens past 768 dimensions for most tasks — going from 1536 to 3072 dims adds marginal recall at roughly 6× the storage cost [3]. Open-weight models now match or exceed closed APIs on MTEB v2; the self-hosting crossover is ~500M–1B tokens/month [2]. Models marked † support Matryoshka Representation Learning (MRL): truncating to 256 dims retains 93–95% of retrieval quality at 6× lower storage [3]. Note that MTEB scores are not directly comparable across leaderboard versions; benchmark on your own corpus before committing [1].

Sources: [1], [2], [3]. MTEB (ret) = retrieval sub-task; (avg) = overall MTEB average.

At comparable scale, self-hosting BGE-M3 costs ~$500/month vs ~$13,000/month for OpenAI’s API [2]. For multilingual and cross-modal retrieval, Gemini Embedding 2 is the strongest all-rounder [1].

Chunking Strategies

Best-vs-worst strategy on the same corpus produces a ~9% recall gap [4]. A January 2026 systematic analysis (arXiv:2601.14123) [20] found chunk overlap adds no measurable benefit and raises indexing cost — treat as tunable rather than a mandatory default. Optimal chunk size by query type: factoid/lookup → 64–256 tokens; analytical/narrative → 512–1024 tokens; quality degrades sharply beyond ~2,500 tokens [4].

Late chunking [5] [6]: embed all document tokens first, then split the token-embedding sequence before mean pooling. The transformer sees full context before chunking, so coreferences (“it”, “they”, “the city”) resolve correctly. Gains of 1.8–1.9% nDCG@10 over naive chunking on BEIR; gains grow with document length [5]. Requires a long-context embedding model (e.g., jina-embeddings-v3).

Contextual Retrieval (Anthropic): an LLM prepends a 50–100 token context summary to each chunk before embedding. Alone cuts retrieval failures by 35%; combined with cross-encoder reranking, −67% [15]. Cost is ~$1.02/M document tokens with prompt caching [4].

ANN Index Algorithms

Use HNSW by default for pure similarity search without heavy filtering [7]. Switch to IVF when filtered searches are common or memory is constrained. Use IVF-PQ at billion-scale: 4–8× memory reduction at ~5% recall cost [18]. For IVF, set nlist ≈ √n and tune nprobe (1–16) at query time without rebuilding the index [7].

Vector Databases

Sources: [8], [17]. Stars fetched June 2026.

Scale selection [8]: under 10M vectors, any option works; 10M–1B narrows to Qdrant, Weaviate, Milvus, or managed Pinecone; above 1B → Milvus distributed or Vespa. Qdrant and Weaviate maintain low latency with complex filters; others can see 2–3× slowdowns. By 2026, hybrid search (BM25+dense) is table stakes — Weaviate, Qdrant, and Vespa ship it natively; Pinecone added it; pgvector requires manual composition [8].

Advanced Retrieval Pipeline

The production pattern for maximum recall [9] [22]:

BM25 (sparse) + dense ANN → top-1000 candidates
  → Reciprocal Rank Fusion (RRF)
  → cross-encoder rerank → top-100
  → LLM generate with citations

Why hybrid. BM25 handles exact-match and rare-term queries that break dense retrieval; dense vectors handle semantic paraphrase and synonyms that break BM25 [19]. Reciprocal Rank Fusion (RRF) sidesteps score-incompatibility between the two signals by working purely on rank position, not raw scores [9]. One team reported +48% RAG accuracy using cascading BM25+FAISS+cross-encoder reranking in production [19].

Cross-encoder reranking. Unlike bi-encoders (which embed query and document independently), a cross-encoder processes the query+document pair jointly, letting attention directly model their interaction. The “retrieve top-1000 → rerank top-100” pattern stays within p99 latency budgets for interactive search [22].

Production Optimization

Batching. GPUs thrive on large batches (256–512 sequences); CPUs are optimal at 16–64 [10]. A key finding: in a naive embedding function, GPU inference accounts for only ~10% of total compute — 90% is CPU-side tokenization [10]. Disaggregating tokenization from inference (pipeline parallelism) and running multiple model replicas per GPU produced a 16× throughput improvement, sustaining 230k tokens/sec on A10g GPUs [10]. Self-hosting via Hugging Face Text Embeddings Inference (TEI) on a GPU instance is straightforward for teams crossing the ~500M-token/month threshold [21].

Vector quantization. Three tiers [11] [12]:

Binary quantization achieves 32× compression and 80% faster queries but requires a full-vector rescoring pass to recover precision [11]. Combined INT8 + dimensionality reduction can cut storage 4–8× with under 1% MTEB score drop [12].