The illusion of RAG being production-ready is a classic problem in enterprise development. At the prototype stage, working with a small volume of curated documents, the system demonstrates impressive accuracy. However, once integrated with an actual corporate knowledge base—where duplicates, contradictions, and obsolete policies have accumulated for years—the quality of answers rapidly degrades. In most cases, the culprit is not the language model and its notorious hallucinations, but architectural missteps in data preparation and retrieval.
Core Architecture: Classic vs. Agentic
Attempting to deploy a complex agentic pipeline right from the start is a common mistake. Agentic loops, where the model independently decides on a retrieval strategy and performs multiple iterations, increase response latency, inflate token costs, and complicate debugging.
Experience shows that about 80% of user queries are trivial and can be resolved using a classic single-pass RAG. The optimal approach relies on a hybrid model:
- A lightweight classifier-router assesses the complexity of the incoming query.
- Standard questions follow a direct, deterministic path.
- Complex multi-part queries are directed to an agentic loop with evaluation of intermediate results.
Secrets of Data Preparation and Indexing
No advanced generation algorithm can fix the output if the data preparation stage was done improperly.
Chunking Pitfalls
Splitting documents requires accounting for language nuances and domain specifics. Overly small chunks (e.g., around 100 tokens) break semantic relationships, turning technical manuals or formulas into disconnected noise. Optimal guidelines for text chunking include:
- FAQ-style databases: around 200 tokens (the question and answer should reside within a single chunk).
- Coherent articles in Russian: around 1,000 tokens on average, with an overlap of about 10% to preserve context across block boundaries.
- Complex layouts (tables, cross-cutting diagrams): semantic boundary segmentation using an LLM, despite higher indexing costs.
Choosing a Search Engine
Relying solely on semantic vectors is risky: dense vector search can miss specific terminology, part numbers, or personal names. The best stability is achieved by combining two approaches:
| Index Type | Mechanism | Strengths |
|---|---|---|
| Semantic | Dense vectors (OpenAI text-embedding-3-large for cloud or BGE-M3 for on-premise) | Understands tone, synonyms, and overall query context. |
| Full-text | Sparse vectors (BM25 algorithm) | Exact matching of terminology, codes, and keywords. |
| Graph | Entity and relationship extraction via LLM | Effective for multi-hop queries, but extremely expensive and complex to maintain. |
Complex graph structures (“the path through Mordor”) should only be introduced once the potential of combining vector and full-text search is completely exhausted.
Retrieval Pipeline and Top-N Traps
A typical fault-tolerant pipeline includes rewriting the initial user turn (taking into account the last k history messages), hybrid retrieval, and subsequent list merging using the Reciprocal Rank Fusion (RRF) algorithm. The selected top candidates (e.g., 10 chunks) pass through a cross-encoder reranker, which filters out irrelevant fragments and feeds the top 3–5 chunks to the generator.
However, fixed Top-N retrieval introduces hidden vulnerabilities:
- Heuristic threshold: A chunk containing the critical answer might rank at N+1 and fail to enter the context.
- Duplicate clustering: Semantic search fills the top results with semantically similar chunks without evaluating their marginal informativeness.
- Broad context blindness: The system cannot answer high-level summary questions (“What is this documentation about?”) because it operates strictly on fragments.
Diagnostics and Quality Control in Production
When an assistant begins producing incorrect answers, isolate the issue strictly step-by-step: first, verify whether the necessary chunk was included in the reranker's output. If the chunk is present, look for the issue in the prompt and temperature (which should be close to zero for factual accuracy). If the document is missing from the retrieved set, tune the retrieval engine, clean query noise, or isolate data domains.
Splitting storage into domain-specific partitions (e.g., dedicated indexes for HR, IT, and accounting) prevents context bleed and boosts accuracy. For automated regression testing, use a combination of real user logs and LLM-as-a-judge methods with a graded scale from 1 to 10, whereas simple user thumbs up/down provide virtually no actionable data for engineering root-cause analysis.