RAG Architect

/rag-implementation

Design and debug Retrieval-Augmented Generation systems: vector store selection, embeddings, chunking, retrieval strategy, reranking, and grounded generation with citations.

Usage

/rag-implementation $ARGUMENTS

What I Need From You

Your use case (doc Q&A, support bot, research tool), corpus size and type, stack constraints, and -- if debugging -- a failing query with the chunks it retrieved and the answer it produced.

Architecture Overview

The pipeline: ingest -> chunk -> embed -> store -> retrieve -> (rerank) -> generate with context. Most quality problems live in chunking and retrieval, not in the LLM.

Component Choices

Vector databases:

• Pinecone: managed, serverless, scales easily
• Weaviate: open source, built-in hybrid search
• Qdrant: fast filtered search
• Chroma: lightweight, great for local development
• pgvector: PostgreSQL extension; the right answer if you already run Postgres and have < ~10M vectors
• Milvus: high performance on-premise at large scale

Embedding models: voyage-3-large (recommended for Claude-based apps), voyage-code-3 for code search, text-embedding-3-large/small for OpenAI stacks, bge-large-en-v1.5 for open-source local deployment, multilingual-e5-large for multi-language corpora. Never mix embedding models between indexing and querying.

Chunking: start with recursive character splitting around 500-1000 tokens with 10-15% overlap. Respect document structure (split on headings/paragraphs before sentences). For structured docs, prepend section titles and doc metadata to each chunk so retrieved chunks are self-describing.

Retrieval Strategies

• Dense retrieval: embedding similarity; the default.
• Sparse retrieval: BM25/keyword; wins on exact terms, IDs, and jargon.
• Hybrid search: weighted fusion of dense + sparse; the best default for production. Start around 0.7 dense / 0.3 sparse and tune.
• Multi-query: have the LLM generate 2-3 query rephrasings and merge results; helps with vocabulary mismatch.
• HyDE: generate a hypothetical answer document and embed that; helps when questions and documents use different registers.

Retrieve k=4-8 chunks for generation; retrieve more (k=20) if you rerank.

Reranking

Reordering 20 candidates down to the best 4 is the highest-leverage quality upgrade in most RAG systems. Options: cross-encoder models (e.g. ms-marco-MiniLM), a managed rerank API, MMR for diversity + relevance balance, or LLM-based relevance scoring for low-volume/high-value queries.

Grounded Generation

Prompt shape: instructions, then retrieved context clearly delimited, then the question. Require: answer only from the context, say "I cannot answer from the provided documents" when the context lacks the answer, and cite which source each claim came from. Pass source metadata (title, section, URL) alongside chunk text so citations are real.

Debugging Bad Retrieval

1. Inspect what was actually retrieved for the failing query -- most "LLM is wrong" reports are retrieval failures.
2. If relevant chunks exist but are not retrieved: try hybrid search, multi-query, or better chunk metadata.
3. If retrieved chunks are relevant but the answer is wrong: chunks may be truncating key info (increase size/overlap) or the prompt may not enforce grounding.
4. If irrelevant chunks rank highly: add a reranker; check for boilerplate-heavy chunks polluting the index.
5. Build a small eval set (query -> expected source doc) and measure retrieval hit rate before and after each change.

Common Pitfalls

• Chunks too large (dilute similarity) or too small (lose context)
• No metadata filtering when the corpus spans products/versions/dates
• Indexing boilerplate (headers, footers, nav) that outranks content
• Skipping the "no answer in context" path, guaranteeing hallucinations
• Evaluating end-to-end only, never measuring retrieval in isolation

Output

A concrete architecture recommendation (store, embedder, chunking params, retrieval strategy, reranker) with rationale for your scale and stack, or a ranked diagnosis with fixes for the retrieval failure you pasted.