RAG Architecture Explained: How to Build a Production-Ready Enterprise AI Chatbot

RAG architecture connects a language model to external knowledge through a retrieval pipeline. It ingests and indexes documents, finds information relevant to a user’s question, and supplies authorised evidence to the model before it generates an answer. A production-ready enterprise RAG chatbot also needs identity controls, document permissions, source citations, evaluation, monitoring and reliable content updates. Businesses use this architecture to make internal knowledge easier to access while retaining control over which information each user can retrieve.

Key takeaways

• Treat ingestion and answering as separate pipelines with shared document identity and versioning.
• Evaluate retrieval before trying to repair every failure with a larger model.
• Enforce document access before evidence reaches rerankers or language models.
• Citations make answers inspectable; they do not prove that an answer is correct.
• Include permission changes, deletion, recovery and operating cost in the first architecture review.

Contents

1. What is RAG?
2. How the architecture works
3. Core components
4. Build an enterprise chatbot
5. Advanced patterns
6. RAG, fine-tuning and long context
7. Security and governance
8. Evaluation and monitoring
9. Costs and optimisation
10. Common mistakes
11. Industry use cases
12. Deployment checklist
13. How Prominent can help
14. Frequently asked questions

What Is Retrieval-Augmented Generation (RAG)?

Retrieval-augmented generation means searching external information and using the results to help a language model answer. Retrieval supplies evidence; generation explains it in language the user can understand.

A standalone LLM answers from its learned parameters and supplied conversation. It does not automatically know your latest purchasing policy, product manual or customer contract. RAG supplies selected information at request time without retraining the model for every document update.

Consider an employee asking which approval is needed for a purchase. The assistant retrieves the current policy for that employee’s business unit, explains the approval route and links to the relevant section. If the evidence is missing or contradictory, it should say so and route the question to the policy owner.

RAG can improve freshness and traceability, but only as far as the underlying content and retrieval allow. It cannot make an obsolete policy correct, guarantee that every relevant passage is found, or eliminate hallucinations. It also creates another system containing potentially sensitive information that must be governed.

For broader business applications, see Generative AI for Business: Practical Use Cases Beyond Chatbots.

How Does RAG Architecture Work?

A typical enterprise RAG pipeline has an indexing path and an answering path. Keeping them separate lets teams update knowledge without tying document processing to interactive response times.

Figure 1. Indexing prepares searchable evidence. Query-time controls determine which evidence a particular user may receive. Neither pipeline grants access merely because a model requests it.

Offline indexing: prepare trusted, searchable evidence

1. Ingest documents: connect approved repositories, record ownership and capture change events or watermarks.
2. Clean and preprocess: remove extraction noise and repeated headers without deleting important qualifications, units or dates.
3. Parse and chunk: preserve headings, table structure and page references; divide content into passages that retain enough context to be understood.
4. Generate embeddings: represent each passage numerically using a selected embedding deployment.
5. Index and store: retain text, vectors, source identifiers, versions, effective dates, tenant IDs and permission metadata. Keep originals in controlled object storage.

Make processing idempotent: replaying an event should not multiply chunks. Record failed items for retry and reconcile index contents against source records. A successful ingestion job is not evidence that every item was indexed successfully.

Online answering: retrieve, constrain and explain

6. Process the question: validate length, identify the conversation and resolve the authenticated user’s access scope.
7. Rewrite when useful: turn an ambiguous follow-up into a standalone query. Preserve names, identifiers and user intent; rewriting is optional.
8. Run hybrid retrieval: combine keyword matching with vector similarity when the workload benefits from both.
9. Apply authorisation and metadata constraints: the retrieval request must carry server-derived tenant and permission filters. This is part of retrieval, not a later prompt instruction.
10. Rerank authorised candidates: use a second relevance stage when measured quality gains justify its latency and cost.
11. Assemble context: deduplicate passages, retain provenance and fit selected evidence into an explicit token budget.
12. Generate a response: ask the model to answer from that evidence, identify uncertainty and abstain where necessary.
13. Validate citations: map citation IDs to retrieved passages, reject invented references and test whether cited evidence supports material claims.
14. Evaluate and monitor: capture privacy-safe traces, quality signals, failures and costs, then feed reviewed findings into controlled improvements.

Azure AI Search combines keyword and vector result rankings through reciprocal rank fusion. Keyword matching is especially useful for exact product codes and names; vector similarity helps with paraphrases. Microsoft’s hybrid search documentation explains the mechanism. Whether it improves your chatbot remains an evaluation question.

Planning an internal knowledge assistant? Discuss your enterprise RAG architecture with Prominent, including your repositories, users and access requirements.

Core Components of a Production-Ready RAG System

Choose components around document types, access boundaries and operating requirements. A specialised vector database is one option; an existing search service or database with suitable vector capabilities may be sufficient.

The practical design question is not how many services can be added. It is whether each failure can be located, explained and corrected without weakening access controls.

How to Build an Enterprise RAG Chatbot Step by Step

The reference stack is Python, FastAPI, Azure Blob Storage, Azure AI Search, Azure-hosted language and embedding models, Microsoft Entra ID and Azure Monitor/Application Insights. LangChain or LangGraph is optional, not a prerequisite.

The downloadable reference code, including app.py and ingest.py, provides educational implementation examples. They require resource provisioning and identity integration and have not been executed against an Azure subscription. They are not a production certification.

1. Define a bounded knowledge task

Choose one corpus and a clear audience: for example, approved IT procedures for employees. Define supported questions, excluded topics, freshness expectations and an escalation path. Collect representative questions before selecting chunk sizes or models.

2. Provision and configure the stack

Create storage, search, model deployments and an application host in approved locations. Use infrastructure as code. Separate the ingestion identity, which can write documents, from the serving identity, which should only read search data and invoke approved models.

Use configuration for resource endpoints, index names and model deployment names. Use managed identity in Azure rather than embedding secrets in source code. Microsoft documents keyless Azure model access and the v1 API. Check endpoint, token audience and model feature support for your resource before deployment.

import os
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import OpenAI

credential = DefaultAzureCredential()
model_client = OpenAI(
    base_url=os.environ["AZURE_OPENAI_BASE_URL"],  # .../openai/v1/
    api_key=get_bearer_token_provider(
        credential, "https://ai.azure.com/.default"
    ),
    timeout=30.0,
    max_retries=1,
)

Azure requests use the deployment name in model. Do not assume a model name copied from a tutorial exists in your subscription. Select models using your own quality, latency and cost tests; see our model-selection guide.

3. Build an ingestion contract

Parse originals into records containing document_id, version, tenant_id, allowed_principals, title, source_url, page and text. Generate chunk IDs deterministically from the document version and passage position. Preserve a manifest of active chunks for replacement and deletion.

Create an index with a string key, searchable text, retrievable provenance, filterable tenant and permission fields, and a vector field whose dimensions match the embedding output. Configure the vector search profile and, if used, semantic ranking separately. The sample ingestion script assumes that index already exists.

Inspect difficult PDFs and tables manually. Start with a structure-aware chunking baseline, then compare alternatives on labelled questions. There is no universally correct chunk size.

4. Implement permission-aware retrieval

Resolve user identity from a verified Entra token. Derive application tenant membership and current group access on the server; a token’s directory tenant is not automatically your application’s customer boundary. Handle group-overage cases explicitly.

from azure.search.documents.models import VectorizedQuery

results = search_client.search(
    search_text=question,
    vector_queries=[VectorizedQuery(
        vector=query_vector, fields="content_vector",
        k_nearest_neighbors=50,
    )],
    filter=server_generated_access_filter,
    vector_filter_mode="preFilter",
    select=["id", "content", "title", "source_url", "page"],
    top=6,
)

Here, 50 candidates and six returned passages are illustrative configuration choices, not recommended universal defaults. Verify the SearchClient interface against your locked SDK version. The example does not enable a separate reranker; evaluate one as an additional controlled change.

5. Generate bounded, inspectable answers

Assign short IDs to retrieved passages. Ask the model for claims with source IDs and an abstention flag. Validate the response schema and ensure every cited ID belongs to the authorised retrieval set. A valid ID is only a reference-integrity check: supporting evidence still needs evaluation.

Budget context using the chosen model’s tokenizer. Keep the original question, relevant conversation history and mandatory instructions within the model’s input limit, with room reserved for output. Avoid passing entire conversations or documents by default.

6. Deploy with operational controls

Run the FastAPI service behind authenticated ingress with request limits, explicit timeouts and bounded concurrency. Separate background workers from chat serving. Add health probes, dependency alerts and privacy-safe telemetry. Use staged releases, quality gates and a tested rollback for application, prompt and index versions.

The sample intentionally rejects requests when trusted identity context is absent. Add the real authentication integration before exposing it. See our cloud AI operations discussion for the broader infrastructure decisions.

Advanced RAG Architecture Patterns

Start with a predictable retrieve-then-generate baseline. Add complexity to address an observed failure, not because a pattern is fashionable. These patterns overlap rather than forming mutually exclusive products.

Here, hybrid refers to keyword-plus-vector retrieval; some frameworks use the term for mixed deterministic and agentic workflows. LangChain’s retrieval overview distinguishes fixed and agent-controlled retrieval. Amazon Bedrock’s retrieval documentation also illustrates managed retrieval and generation options. Product implementations and guarantees differ.

For agentic systems, authorise every tool call independently and bound the number of steps, elapsed time and spending. Our AI agents article explains the business context.

RAG vs Fine-Tuning vs Long-Context LLMs

RAG supplies external knowledge; fine-tuning changes model behaviour; long-context prompting supplies more material directly. They can be combined.

For exact live account balances, query an authorised transactional API rather than searching yesterday’s exported PDF. For deterministic calculations, use validated code. Choosing RAG where ordinary search or a workflow would suffice adds avoidable operating risk.

Enterprise RAG Security and Governance

A secure RAG chatbot enforces access outside the model. Once confidential text reaches an unauthorised model context or client, a prompt saying “do not reveal it” is not an adequate control.

Authenticate callers, validate token issuer, audience, signature and expiry, and resolve their authorised scope. At retrieval time, filter by tenant and permitted principals. Apply equivalent checks to document previews, parent expansion, citation URLs and follow-up tools. Azure’s security filter pattern describes application-managed result trimming; it does not populate or maintain your permissions for you.

Fail closed when identity, policy resolution or access metadata is missing. Do not fall back to unrestricted search. Revocations need an explicit propagation target and a rapid deny mechanism where eventual indexing is too slow. Cache keys must include relevant access scope and content/policy versions; permission changes must invalidate affected entries.

Treat retrieved documents as untrusted data, including instructions hidden in PDFs, tables or images. Restrict tools, validate outputs, escape displayed content and require approval for consequential actions. The OWASP prompt injection guidance explains why RAG alone is not a defence against malicious instructions.

Encrypt transport and storage, minimise sensitive fields, review model-provider processing terms and restrict network paths where required. Log identities, source versions and decisions with access controls; avoid unrestricted prompt/response logging. Define retention and deletion across originals, chunks, embeddings, caches, backups and traces.

Maintain provenance so teams can investigate which source supported an answer. Test restoration without resurrecting revoked content. These controls support governance; they do not establish legal compliance by themselves. Our AI security guide covers the wider application risks.

RAG Evaluation Metrics and Production Monitoring

Evaluate search and generated answers separately. A fluent answer can hide a retrieval failure, while a relevant passage can still be misinterpreted.

Define whether relevance labels apply to chunks or documents; mixing them makes metrics misleading. Groundedness is not truth: an answer can accurately repeat an outdated source. Citation coverage also matters—correct citations on two claims do not justify three uncited claims.

Build a representative dataset spanning user roles, departments, languages, document types, versions and unanswerable questions. Include adversarial documents and denied-access cases. Separate tuning examples from a held-out release set. Have domain reviewers label expected evidence and acceptable answers; record disagreements.

Run candidate versions on the same corpus snapshot and evaluation set. Compare quality, latency and costs by slice, not only an overall average. Calibrate automated judges against human review. Microsoft’s RAG evaluator guidance and OpenAI’s evaluation best practices provide complementary evaluation approaches.

Monitor retrieval misses, abstentions, stale sources, citation failures, authorisation denials and ingestion lag alongside service health. Version prompts, parsers, models and indexes so regressions can be traced. See MLOps vs. Traditional DevOps for release discipline.

RAG Implementation Costs and Performance Optimization

Enterprise RAG cost includes initial engineering and recurring operation. A token estimate alone excludes much of the system.

Model monthly cost as:

Monthly total = fixed infrastructure + ingestion/parsing + embedding
              + storage + search/reranking + model inference
              + monitoring/network/security services + maintenance

Inference = (input tokens / 1,000,000 × input rate)
          + (output tokens / 1,000,000 × output rate)

Illustrative workload, not a quote: assume 100,000 documents averaging 2,000 source tokens, 10% changed monthly, 30,000 answered questions monthly, 4,000 input tokens and 500 output tokens per answer. Initial source text is 200 million tokens; monthly changed source text is 20 million tokens. Chunk overlap, metadata and retries increase actual embedding volume and must be measured separately.

Answer generation uses 120 million input and 15 million output tokens monthly under those assumptions. If current per-million input and output rates are I and O, generation is 120I + 15O. If measured changed-text embedding volume is B million tokens at rate E, monthly embedding cost is BE. Add query embeddings, rewriting, evaluation and reranking separately; the scenario excludes those auxiliary model calls.

Obtain regional service prices from the Azure pricing calculator, recording currency, region, deployment type, date, capacity and assumptions. Search may be capacity-based rather than a simple price per query. Include replicas, partitions, peak concurrency, logs, OCR, backups and recovery capacity. No vendor price is assumed here.

Reduce cost by correcting retrieval before increasing model size, bounding output, deduplicating ingestion and re-embedding only changed chunks. Cache only where access isolation and freshness are preserved. Test smaller models and reranking trade-offs. Track latency and quality together: faster answers are not valuable if they omit essential evidence.

Common RAG Architecture Mistakes

Azure’s document deletion guidance is particularly relevant to avoiding orphaned search records. Test deletion with the same care as initial ingestion.

Enterprise RAG Use Cases by Industry

These are illustrative applications, not descriptions of completed Prominent projects.

Regional deployment decisions should follow each organisation’s data-handling requirements. An international audience does not justify inventing local offices, certifications or blanket compliance claims.

Production Deployment Checklist

• Architecture: approved data flows, ownership and failure boundaries are documented.
• Data quality: difficult scans, tables, duplicates and obsolete versions are tested.
• Retrieval: held-out relevance tests cover exact terms, paraphrases and empty results.
• Security: tenant isolation, permission revocation and fail-closed behaviour pass negative tests.
• Scalability: peak concurrency, throttling and dependency outages are exercised.
• Evaluation: reviewers approve task-specific answer, citation and abstention criteria.
• Monitoring: ingestion lag, denied requests, errors, tail latency and cost have owners and alerts.
• Cost controls: token, request, retry and agent-step limits are enforced.
• Disaster recovery: source manifests, index rebuilds and restore procedures are tested against recovery objectives.
• Governance: source approvals, retention, deletion and incident responsibilities are agreed.
• Release readiness: rollback, dependency versions, user guidance and escalation are verified.

Need an independent review before launch? Request a technical consultation covering retrieval quality, access controls and operating assumptions.

How Prominent AI Services LLP Can Help

Prominent’s published capabilities include generative AI development, enterprise AI integration, conversational AI integration and Microsoft Azure engineering.

A scoped engagement can address solution architecture, data preparation, knowledge ingestion, workflow integration, deployment and ongoing optimisation. Agree the actual deliverables, evaluation criteria and responsibilities before implementation. Start with your source systems, permitted users and a representative set of questions rather than a model preference alone.

Frequently Asked Questions

What is RAG architecture?

RAG architecture retrieves external evidence and supplies it to a language model before generation. Enterprise implementations add identity, permissions, content lifecycle management, evaluation and monitoring.

How does a RAG chatbot work?

It receives a question, searches information the user is authorised to access, selects relevant passages and generates a response with source references. Missing evidence should trigger an explanation or escalation.

What is the difference between RAG and fine-tuning?

RAG supplies knowledge at query time. Fine-tuning updates model behaviour through training. They solve different problems and can be used together.

Which vector database is best for RAG?

There is no universal winner. Compare filtering, hybrid search, isolation, scale, operations, cost and integration with your existing platform using the same representative workload.

How do you build a production-ready RAG system?

Build ingestion and query pipelines, preserve provenance and permissions, evaluate retrieval and answers, then add operational limits, monitoring, recovery and staged release controls.

How much does an enterprise RAG chatbot cost?

Cost depends on engineering scope, documents, query volume, model use, search capacity and operating requirements. Estimate each component using measured workload assumptions and current regional vendor rates.

How do you secure confidential documents in RAG?

Authenticate users and enforce current document permissions before retrieval results reach generation. Isolate tenants, constrain caches and tools, and fail closed when policy cannot be resolved.

How do you reduce hallucinations in RAG?

Improve source quality, retrieval and context selection; require evidence and abstention; validate citations and evaluate answers. These measures reduce risk but do not eliminate hallucinations.

What is the difference between RAG and agentic RAG?

Fixed RAG follows a predetermined retrieval sequence. Agentic RAG lets a model choose retrieval tools or additional searches, requiring extra limits, authorisation and evaluation.

How do you evaluate RAG accuracy?

Use labelled questions and evidence to measure retrieval quality, answer correctness, groundedness and citation support. Include unanswerable questions and access-denied cases, with human review.

Can RAG work with SharePoint and Microsoft 365?

Yes, through appropriately authorised integrations that preserve permissions and content changes. Microsoft currently labels its SharePoint indexer preview; verify feature status and permission-update behaviour before relying on a particular connector in production.

When should an organization avoid using RAG?

Avoid it when approved evidence is unavailable, permissions cannot be maintained, ordinary search is sufficient, or deterministic APIs are needed for live transactions and calculations.

Conclusion: build for evidence, access and operation

A useful enterprise chatbot needs more than a model connected to documents. Design around authorised evidence, measurable quality and maintainable operation. Start with a bounded task, prove retrieval on representative questions, then expand only after security and release controls work.

Discuss Your Enterprise AI Chatbot — share your knowledge sources, intended users and questions the system must answer.