Why RAG dominates AI system design
Retrieval-Augmented Generation (RAG) is the pattern behind "ChatGPT over our docs," support bots with a knowledge base, and internal copilots. Instead of stuffing the company wiki into model weights, you retrieve the right passages at query time and augment the LLM prompt so the answer is grounded in your corpus.
In interviews this is the default answer for knowledge Q&A. Fine-tuning still has a place (style, domain language), but RAG is how you ship updatable facts with citations. This deep dive covers the full system — not just "call a vector DB."
Parse · chunk · embed · index.
ANN · hybrid · filters · rerank.
Grounded prompt · citations.
Recall@k · faithfulness.
Why not just a bigger context window?
Long-context models tempt you to paste everything. That fails in production:
- Cost & latency — tokens scale with every request; corpora are GBs.
- Attention dilution — "lost in the middle"; relevant facts get ignored.
- Permissions — not every user can see every doc; retrieval must respect ACL.
- Freshness — docs change hourly; you need reindex, not retrain.
RAG keeps the prompt small and relevant. Long context still helps (more chunks, longer history) — it doesn't replace retrieval at company scale.
Numbers that stick in interviews: a 50k-page Confluence export might be 200M tokens — impossible to send per query. Even a 128k context window holds ~300 pages; your corpus is thousands. Retrieval finds the needle; the window holds the hay around the needle.
Ingestion: from sources to index
Sources
Wikis, Notion, Confluence, PDFs, tickets, code, Slack exports. Normalize to text + structured metadata. Store the canonical blob in object storage; the vector index holds chunks + pointers.
Parsing
Strip boilerplate (nav, footers). Preserve headings and tables when possible — structure helps chunking. OCR for scans; language detection for multilingual corpora.
Metadata you must keep
| Field | Why |
|---|---|
source_id / URL | Citations and deep links |
title, section path | UI + ranking signals |
updated_at | Freshness boost / staleness |
acl / tenant | Filtered retrieval |
embedding_model | Avoid mixing incompatible vectors |
Triggers
Nightly batch is fine for wikis. For support policies, prefer CDC / webhooks on publish. Deletes must tombstone vectors — soft-delete then compact.
Idempotency and exactly-once ingest
Ingest workers will retry. Every document write needs a stable content_hash or version id: unchanged hash → skip re-embed; changed hash → upsert chunks and tombstone old chunk ids. Use idempotent upsert keys (source_id + chunk index + embedding_model_version). Without this, a flaky worker doubles index size and spend.
Index lag as an SLO
Define index lag: time from publish/delete in the source system to searchable/reflected in the vector index. Support teams feel "the bot lied" when lag is 6 hours but the UI says "updated today." Measure p95 lag; alert when it exceeds SLA (e.g. 15 min for policies, 24 h for wikis).
Chunking strategy
Chunking is the most underestimated RAG design choice. Embeddings describe a passage, not a whole book. Bad chunks → bad retrieval forever.
- Fixed-size + overlap — simple; good baseline.
- Structure-aware — split on Markdown headers / HTML sections first.
- Semantic chunking — split when topic shifts; more complex, sometimes better.
- Parent-child — retrieve small child chunks, expand to parent section for the prompt (precision + context).
Embeddings and the index
Use the same embedding model at ingest and query time. Mixing models silently destroys quality. Version the model id in metadata; plan a dual-index migration when you upgrade (see Vector databases).
- Index — HNSW/IVF in pgvector, ES, Pinecone, etc.
- Dims — 384–1536 typical; more dims ≠ always better for your domain.
- Filters — apply ACL / product / locale at query time (pre-filter when selective).
- Sparse + dense — hybrid BM25 + vectors for SKUs, error codes, proper nouns.
Authoritative content still lives in Postgres/S3. The vector store is an index, not the source of truth — same discipline as Elasticsearch. Rebuild the index from canonical storage; never treat "vector DB is down" as "docs are gone."
Embedding model migration
Upgrading embeddings is a data migration, not a config flip. Run dual indexes (old + new model version), shadow-query the new index, compare Recall@k on golden queries, cut over with a feature flag, then decommission. Mixing vectors from two models in one index silently destroys quality — enforce embedding_model in metadata and query filters.
ACL at retrieval, not in the prompt
Never rely on the LLM to "ignore" docs the user cannot access. Apply tenant_id / ACL filters in the retrieval layer before chunks enter the prompt. For regulated tenants, use separate indexes or namespaces — filter leakage is a security incident, not a quality bug.
Hybrid retrieval in practice
Dense vectors catch paraphrases ("standing desk stipend" ≈ "ergonomic furniture allowance"). Sparse BM25 catches exact tokens (SKU-4421, error code E_AUTH_403, product names). Reciprocal Rank Fusion (RRF) merges ranked lists without calibrating scores: for each doc, score = Σ 1/(k + rank_i). Start with k=60; tune on golden queries.
Online query path
1. Query understanding
Optional but high leverage: rewrite follow-ups with conversation history ("that" → full question), multi-query expansion, or HyDE (generate a hypothetical answer, embed that). Keep it behind a flag — adds latency and failure modes.
2. Retrieve
Embed the query → ANN top-k (often 20–100) with metadata filters. Optionally fuse with BM25 (RRF — reciprocal rank fusion is a common merge).
3. Rerank
Cross-encoder or LLM reranker scores (query, chunk) pairs. Cut to top 5–10 for the prompt. Same two-stage idea as recsys: recall then precision.
4. Generate
Pack chunks with ids into the prompt; instruct grounded answering and citations. Details in Prompt engineering.
5. Post-process
Verify citation ids exist in retrieved set; strip invented sources; optionally run a faithfulness check; stream tokens to the client.
Production architecture
- API gateway — auth, rate limits, abuse, tenant isolation.
- Orchestrator — owns retrieval plan, prompt templates, tool calls, retries.
- Vector DB — similarity index with filters.
- Primary DB / object store — canonical docs + ACL.
- LLM provider — with fallback model and budget caps.
- Workers + queue — parse/chunk/embed; reindex; eval jobs.
- Observability — trace retrieve → rerank → generate; log chunk ids, latency, tokens, prompt version.
Caching
Cache embeddings for repeated queries; cache full answers only for identical, non-personalized questions (careful with ACL). Semantic caches can collide — prefer exact-hash for safety-critical tenants.
Multi-tenancy
Filter by tenant_id on every retrieve. Prefer separate indexes/namespaces for hard isolation when required. Never rely on the LLM to "not mention" other tenants' data.
Resilience
Circuit-break the LLM and embedding providers; fall back to cached answers or keyword-only search when the vector path is degraded. Queue ingest when embed API is throttling — don't block publish webhooks synchronously on the hot path. See Caching strategies and Message queues for the same patterns you use elsewhere.
Deletion and compliance
GDPR/CCPA deletes must propagate: tombstone in source DB → delete or invalidate chunks in the index → verify retrieval no longer returns them. Log deletion jobs with audit ids. RAG systems forget slowly if ingest doesn't handle deletes as first-class events.
Reference latency budget
| Stage | p50 target | p95 target | Notes |
|---|---|---|---|
| Auth + routing | 5–15 ms | 40 ms | Gateway |
| Query rewrite (optional) | 80–200 ms | 400 ms | Small model |
| Embed query | 30–80 ms | 150 ms | Cache repeats |
| ANN + filters | 20–60 ms | 120 ms | HNSW in RAM |
| Rerank top-40→8 | 80–200 ms | 400 ms | Skip for FAQ route |
| LLM first token | 200–500 ms | 1.2 s | Stream UX |
| End-to-end FAQ | <1.5 s | <3 s | No agent |
Advanced patterns (use when evals demand)
- Hybrid search — dense + sparse for IDs and jargon.
- Multi-hop / iterative retrieval — retrieve → partial answer → new queries (agentic RAG). Cap hops.
- Graph RAG — entities/relations for structured domains; heavier ops.
- Corrective RAG — if retrieved context is weak, rewrite query or web-search fallback.
- Late chunking / contextual embeddings — enrich chunk text with document title before embed.
Failure modes
| Symptom | Likely cause | Fix | Example |
|---|---|---|---|
| Confident wrong answer | Hallucination or ignored context | Grounded prompt, citations, faithfulness check | Bot cites chunk-9 but chunk-9 says opposite — post-process rejects |
| "I don't know" but doc exists | Retrieval miss | Chunking, hybrid, HyDE, more k | Policy uses term "SKU-4421"; user says "blue widget" — hybrid BM25 saves it |
| Answer mixes products | No ACL / filter | Metadata filters, tenant indexes | Enterprise user sees Startup pricing doc — filter leak |
| Outdated policy | Stale index | Reindex pipeline, freshness boost | Refund window changed to 14 days; index still has 30 — index lag alert |
| Slow p95 | Large k, big context, slow embed | Rerank funnel, cache, smaller model | top-100 + no rerank = 40k prompt tokens every query |
| Spiky cost | Rerank + long context on every query | Router, tier models, cap prompt tokens | Simple "hours of operation" hits full rerank path |
| Wrong after deploy | Embedding or prompt change untested | Shadow index, golden-set gate, rollback flag | New embed model drops Recall@10 from 0.82 → 0.61 |
Evaluation
Build a golden set: questions → acceptable source docs → (optional) reference answers. Measure:
- Retrieval — Recall@k, MRR, whether the gold doc appears in top-k.
- Answer — faithfulness to context, relevance, citation accuracy, correct refusals.
- Ops — latency, $/query, index lag (time from publish to searchable).
Change one lever at a time (chunk size vs reranker vs prompt). LLM-as-judge helps but calibrate with humans for high-stakes domains.
End-to-end worked example: Notion-style docs Q&A
Design brief: 10k employees, 2M Notion pages + PDFs, multi-workspace ACL, p95 < 3s for FAQ, citations required.
- Ingest — Notion webhook → queue → parse blocks → heading-aware 500-token chunks → embed
text-embed-3-small→ upsert to pgvector withworkspace_id,page_id,acl_group_ids[]. - Online — gateway auth → expand ACL groups for user → rewrite follow-ups → embed → ANN top-40 with ACL pre-filter → cross-encoder top-8 → grounded prompt → stream + citation ids.
- Eval — 200 golden Qs from support tickets; gate deploys on Recall@10 ≥ 0.78 and citation accuracy ≥ 0.95.
- Ops — index lag SLO 15 min for policy pages; nightly full reconcile from Notion export for drift.
- What we skip on day 1 — GraphRAG, multi-hop agents, semantic answer cache across users.
Failure drill
Incident: answers cite deleted pages. Root cause: soft-delete in Notion without tombstone in index. Fix: delete webhook → tombstone chunks within 60s; weekly reconcile job compares source ids vs index. Add alert: retrieval hit rate on deleted_ids > 0.
Principal engineer lens
RAG at scale is a search + data pipeline + LLM feature problem, not an ML research project. The decisions that separate staff from principal:
- Separate indexes from truth — rebuild path exists; DR tested.
- ACL before prompt — retrieval layer enforces tenancy; audits prove no cross-tenant hits.
- Version everything — embedding model, chunker, prompt template, reranker — in traces for rollback.
- Two metric classes — retrieval vs generation; never optimize one proxy (BLEU, vibes) for both.
- Incremental complexity — hybrid and rerank when golden-set misses justify the latency/cost tax.
- Platform boundary — shared ingest + index service; product teams own prompts and eval suites.
In your interview
Classic prompts: "Design a knowledge base Q&A," "Design Notion AI," "Design an internal docs chatbot," "Design customer support with a help center."
What to say out loud
"I'd split offline ingest from online Q&A. Docs land in S3/Postgres; workers chunk ~500 tokens with overlap, embed with a fixed model version, and upsert into a vector index with ACL metadata. At query time the orchestrator embeds the question, retrieves top-50 with tenant filters, optionally reranks to top-8, packs a grounded prompt that requires citations, and streams the answer. We evaluate Recall@k and faithfulness on a golden set, and we reindex on publish — not by fine-tuning."
- Draw online vs offline paths explicitly.
- Name chunking and ACL before naming Pinecone.
- Separate vector index from source of truth.
- Give numbers: top-k, chunk size, latency budget, rough token cost.
- Escalate: hybrid → rerank → multi-hop only with a reason.
Cost and performance levers
Production checklist
- Canonical docs in S3/DB; vector index is derived and rebuildable.
- Same embedding model at ingest and query; version in metadata.
- ACL filters applied before chunks enter the prompt.
- Idempotent ingest with content_hash; deletes tombstone vectors.
- Golden set with Recall@k + faithfulness; change one lever at a time.
- Index lag SLO + provider circuit breakers + prompt version in traces.
- Hybrid + rerank only after naming the failure they fix.
RAG ops addendum
Failure modes to mention
Call out at least one dependency failure (DB down, cache stampede, queue lag, region outage) and your mitigation (timeouts, retries with jitter, degraded mode, circuit breaker).
Interview Q&A by level
Practice saying these out loud for RAG. Interviewers grade clarity and judgment more than buzzwords.
Match depth to the bar: define → trade off → operate. Don't dump principal answers in an entry-level screen.
Wrapping up
RAG is a retrieval system glued to a generator. Most quality lives in chunking, indexing, filters, and reranking; prompts and model choice finish the job. Keep ingest async, treat vectors as an index, ground every answer, and measure retrieval separately from generation.
Continue with Agentic architectures, Agentic patterns, Agentic frameworks, Prompt engineering, and Key technologies.