RAG that actually ships: what changes past 10k docs

Building a RAG system that works on 50 documents is a weekend project. Building one that works on 10,000 documents is an engineering discipline. The difference isn't scale — it's the failure modes that only appear when your data gets messy, your queries get diverse, and your users get creative.

Chunking strategy matters more than you think

At small scale, naive chunking works fine. Split by paragraphs, embed each chunk, retrieve the top five. Done. At 10k documents, you discover that some documents have paragraphs that are 2,000 words long, some have paragraphs that are one sentence, and some have content that only makes sense when you read the surrounding context.

The fix: implement chunking strategies that respect document structure. Headers, sections, and semantic boundaries. Not just character counts.

Hybrid search is not optional

Pure vector search misses exact matches. Pure keyword search misses semantic relationships. At scale, you need both. Implement hybrid search with reciprocal rank fusion from day one, even if pure vector search seems "good enough" at 10k documents. It won't be at 100k.

The eval harness

You can't improve what you can't measure. Build an evaluation harness early. Curate 50-100 representative queries with known-good answers. Run them automatically. Track precision, recall, and relevance scores over time.

Without this, you're flying blind. Every change to your chunking strategy, embedding model, or retrieval parameters is a guess. With it, you can iterate confidently.

The quiet checklist

Before you ship past 10k documents: validate your chunking strategy on diverse document types. Implement hybrid search with rank fusion. Build an eval harness with representative queries. Set up monitoring for retrieval quality. Plan for index updates, not just initial ingestion.

RAG at scale isn't about the fanciest embedding model. It's about the discipline to handle the boring edge cases that your users will inevitably find.