RAG architecture (retrieval-augmented generation) is how most production AI assistants stay grounded in your documents instead of inventing policies.
Large language models are fluent. They are not automatically correct about your product, handbook, or tickets. With retrieval-augmented generation, you retrieve relevant passages from a knowledge base, then generate an answer using that context - reducing hallucinations and keeping private data out of fine-tuning pipelines.
This is the RAG architecture pattern we use when shipping business chatbots and AI features: chunking, embeddings, hybrid retrieval, reranking, guardrails, and evals - practical enough for an MVP, strict enough for production.
What Is Retrieval-Augmented Generation (RAG)?
RAG is
- Index your content → embed chunks → retrieve top-k → generate with citations
- A pattern, not a single product
- Compatible with OpenAI, Anthropic, open-weight models, or a mix
RAG is not
- Fine-tuning (different tool - useful for style/format, weaker for "what changed in the handbook last Tuesday")
- A guarantee of zero hallucinations (bad retrieval still produces confident nonsense)
- "Dump the PDF into ChatGPT"
If your content changes weekly, prefer RAG (or RAG + light fine-tuning later). If you need a brand voice on generic tasks, fine-tuning or system prompts may be enough.
Production RAG Architecture (Reference Stack)
A production stack usually has these layers:
| Layer | Job | Common choices |
|---|---|---|
| Ingestion | Load docs, APIs, Confluence, S3, tickets | Custom ETL, LlamaIndex loaders, Airbyte |
| Chunking | Split into retrievable units | 300-800 tokens, overlap, structure-aware splits |
| Embeddings | Vectorize chunks | text-embedding-3-large, Cohere, open models |
| Vector store | Similarity search (+ filters) | pgvector, Pinecone, OpenSearch, Weaviate |
| Retrieval | Top-k + metadata filters | Hybrid BM25 + dense, permission filters |
| Rerank | Improve precision | Cohere Rerank, cross-encoders |
| Generation | Answer with context | GPT-4.1 / Claude / open models |
| Guardrails | PII, topic allowlists, tool gates | Policy prompts + validators |
| Evals | Measure quality over time | Golden Q&A sets, LangSmith, custom harness |
| Observability | Cost, latency, failures | Traces, token budgets, alert on empty retrieval |
You do not need every box on day one. You do need ingestion → retrieve → generate → log.
Chunking: Where Most RAG Systems Fail
Garbage chunks produce garbage answers.
Prefer
- Split on headings, sections, and semantic boundaries
- Keep enough context in each chunk to stand alone
- Store metadata:
source_url,product,locale,updated_at,acl
Avoid
- Blind 500-character slices through tables and code
- Mixing unrelated policies into one mega-chunk
- Indexing nav chrome, cookie banners, and duplicate footers
For APIs and engineering docs, chunk by endpoint or module. For policies, chunk by section with the policy title repeated in every piece.
How to Improve RAG Retrieval Quality
Dense-only search misses exact IDs, error codes, and SKUs. Keyword-only misses paraphrases.
Production default for RAG architecture:
- Hybrid retrieve - BM25/keyword + vector similarity
- Metadata filters - tenant, role, product line, language
- Rerank top 20 → keep 4-8 for the prompt
- Citations - return source IDs the UI can link
Always enforce permissions at retrieval time. Do not retrieve a chunk the user could not open in the source system, then "hope" the model refuses.
Prompting the LLM in a RAG System
A durable pattern:
- System: role, refusal rules, citation format
- Retrieved passages labeled
[1],[2], ... - User question
- Instruction: answer only from passages; say "I don't know" if unsupported
Keep context under your model's effective window. More chunks ≠ better answers after a point - noise rises and cost climbs. For napkin token math while you design prompts, use Myrafy's free AI token estimator and prompt linter.
Evals Beat Vibes
Ship a golden set of 30-100 questions with expected sources and acceptable answers. Score:
- Retrieval hit rate (was the right doc in top-k?)
- Answer faithfulness (supported by context?)
- Refusal quality (does it decline out-of-scope?)
- Latency and cost per query
Without evals, every prompt tweak is superstition. With evals, you can upgrade models and chunkers safely.
Cost and Latency Levers
| Lever | Effect |
|---|---|
| Smaller embedding + cached embeddings | Lower index and reindex cost |
| Rerank + fewer generator tokens | Often cheaper than stuffing 20 chunks |
| Route easy FAQs to a cheap model | Reserve frontier models for hard cases |
| Cache frequent queries | Huge win for support bots |
| Streaming | Better UX even when TTFT is fixed |
For broader AI feature budgeting, see OpenAI chatbots for business and custom software costs in 2026.
Security Checklist
- Encrypt vector stores and backups
- Tenant isolation in multi-customer SaaS
- Redact secrets before embedding
- Log prompts/responses with retention limits
- Human escalation path for high-risk domains (legal, medical, finance)
RAG does not remove compliance work. It concentrates it in retrieval and logging.
MVP Path (2-4 Weeks)
- Pick one corpus (help center or internal runbooks)
- Structure-aware chunk + embed into pgvector or a managed store
- Hybrid retrieve + simple prompt with citations
- Admin UI to reindex; basic eval set of 40 questions
- Monitor empty retrieval and high token spend
Add agents and tools after retrieval quality is boringly good. Tool-using agents on top of bad RAG amplify mistakes.
FAQ: RAG Architecture
RAG vs fine-tuning - which should I use? Use RAG when answers must track changing documents. Use fine-tuning for style, format, or domain phrasing. Many systems use both.
Do I need a specialized vector database? Not always. pgvector on Postgres is enough for many MVPs. Move to Pinecone/Weaviate/OpenSearch when scale or ops demands it.
Why do RAG bots still hallucinate? Usually weak chunking, no hybrid search, no rerank, or prompts that allow answering without citations. Fix retrieval before swapping models.
The Bottom Line
Production RAG architecture is a data and retrieval problem first, a model problem second. Invest in chunking, hybrid search, permissions, citations, and evals. The LLM is the last mile - not the architecture.
Building a grounded AI assistant on your docs or product data? See production AI or contact us for an estimate.