How to build a RAG system on AWS.

IWhat RAG is, and the problem it solves

RAG is a pattern, not a product. Before the model answers, you retrieve the most relevant passages from your own data and put them in the prompt. The model answers from those passages instead of from memory alone.

A foundation model only knows its training data, frozen at a cutoff date. It knows nothing about your wiki, tickets, contracts or last night's product change, and if you ask, it may invent an answer. RAG separates knowledge from reasoning. The model does the language and reasoning; the facts come from data you control and can update at any time.

Offline, you split documents into chunks, turn each chunk into a vector with an embedding model, and store the vectors. At query time, you embed the question with the same model, find the nearest chunks, optionally re-rank them, and put the best ones in the prompt with an instruction such as "answer only from the context below and cite the passage you used."

You get three things. Freshness: update a document and the next answer reflects it, with no retraining. Citations: users can check the source. Governance: you decide what is in the corpus and who can retrieve what. Fine-tuning bakes knowledge into weights, is costly to update, gives no citations and cannot enforce per-user access. Use RAG for knowledge; use fine-tuning for style or a narrow skill.

IIThe reference RAG architecture on AWS, stage by stage

Every RAG system, managed or DIY, runs the same seven stages. Most quality problems trace back to one of them.

IIITwo paths: Bedrock Knowledge Bases or DIY

Start managed. Move a stage to DIY only when a specific requirement forces it.

IVVector store options on AWS

The vector store holds your embeddings and answers nearest-neighbour queries. The right one depends on how you operate, your cost shape and your scale.

Bedrock Knowledge Bases can use Amazon OpenSearch Serverless, OpenSearch Service managed clusters, Amazon S3 Vectors, Amazon Aurora PostgreSQL (pgvector), Neptune Analytics (for GraphRAG), Pinecone, Redis Enterprise Cloud and MongoDB Atlas. A DIY pipeline can use any of these, plus stores such as Amazon MemoryDB. The current list is in the Bedrock user guide.

OpenSearch Serverless is the common default. It does vector and keyword search in one engine, so hybrid retrieval is built in, but it bills by capacity units with a baseline, which can feel high for a small corpus. Aurora PostgreSQL with pgvector suits teams already on Postgres: vectors sit next to your relational data and you filter with SQL. S3 Vectors targets large vector sets that are queried less often, at lower storage cost. Pinecone is a vector-only managed service from a third party. In-memory stores such as MemoryDB fit retrieval where latency matters most, at a higher cost per GB.

VEmbeddings, chunking and re-ranking

If the right chunk never reaches the prompt, no model can answer well. Three knobs matter most.

VIEvaluating a RAG system

A demo is not an evaluation. Use metrics that separate retrieval problems from generation problems.

Build a fixed set of real questions with correct answers and, ideally, the source passage. Run it on every change. Amazon Bedrock includes RAG evaluation that uses a model as a judge to score retrieval and answers; open-source tools such as Ragas do the same for DIY pipelines.

• Faithfulness: Does the answer follow from the retrieved context? Low faithfulness with good context is a prompt or model problem.
• Answer relevance: Does the answer address the question that was asked?
• Context precision: How many retrieved chunks were relevant? Low precision means a noisy retriever: add re-ranking or hybrid search.
• Context recall: Did retrieval find everything needed? Low recall means fix chunking, the embedding model, the candidate count or hybrid search.

In production, log every query with its retrieved chunks and answer, and have a person review a sample. Automated judges miss domain errors an expert catches at once.

VIIProduction: freshness, access control, latency and cost

Four things rarely show up in a prototype and always show up in production.

• Freshness: Knowledge Bases can sync changed documents; in DIY, re-embed on S3 events or a schedule. Set a freshness target per source and keep a timestamp in each chunk's metadata.
• Access control: Enforce it in retrieval, not after generation. Tag each chunk with user, group or tenant metadata and filter every query. Knowledge Bases supports metadata filters; DIY stores support them too. For multi-tenant SaaS, filter by tenant at least, or use separate indexes.
• Latency: Generation usually dominates, so stream the response. Keep the re-ranked context short.
• Cost: Route easy questions to a smaller model, pass fewer chunks, use prompt caching for a static system prompt, and run offline generation as batch, which AWS prices 50% below on-demand for select models. Size the vector store to the corpus, since you pay for it with or without traffic.

VIIICommon RAG apps on AWS: chatbots, document Q&A, voice

Most GenAI assistants are RAG underneath. The differences are the sources, the channel and the controls.

• Support chatbot: Retrieve from your help center and resolved tickets, ground answers with citations, add a Guardrail, and hand off to a person when the model is unsure or the customer asks.
• Document Q&A: Answer questions over contracts, policies or reports. Use hierarchical chunking for long documents, filter by who may see each document, and always show the source passage.
• Internal knowledge assistant: Connect the wiki, Confluence or SharePoint through Knowledge Bases connectors, and filter by team or group.
• Voice assistant: Add speech in front of the same pipeline: Amazon Transcribe for speech to text and Amazon Polly for text to speech, or a speech model on Bedrock. Keep answers short and stream them to cut perceived latency.

IXA step-by-step build (managed path)

The fastest credible path to a cited RAG system on AWS. A DIY build follows the same order.

• Step 1: stage the corpus in S3: Land documents in S3 and extract clean text first (Textract for scans and tables).
• Step 2: check model access: Access to many Bedrock models is on by default. The first call to a third-party model needs AWS Marketplace permissions, and Anthropic models need a one-time use-case form.
• Step 3: create the Knowledge Base: Point it at the bucket, pick an embedding model and a vector store, and start with semantic or hierarchical chunking.
• Step 4: sync and inspect: Run ingestion, then use the Retrieve API on a handful of known questions. Fix parsing and chunking before anyone sees an answer.
• Step 5: wire RetrieveAndGenerate: Instruct the model to answer only from context and cite sources. Turn on re-ranking, attach a Guardrail and show citations in the UI.
• Step 6: add access control and freshness: Tag chunks with access metadata, filter every query and turn on incremental sync.
• Step 7: evaluate and iterate: Score faithfulness, relevance, precision and recall with Bedrock RAG evaluation. Tune chunk size, candidate count and re-ranking against the numbers. Add logging and human review before scaling traffic.

XWhere the money goes in a RAG system

A RAG bill has five lines. Generation is usually the largest, then the vector store baseline. Check the AWS pricing pages for current rates.

XIWho files AWS funding for a RAG build

A partner files one kind of AWS funding. You apply for the other.

Partner-filed funding. If the build qualifies, your matched AWS partner can file POC (proof of concept) funding to test the use case, or incremental workload (IW) funding if you already run on AWS and are launching new work such as an AI assistant. If you are moving to AWS from another cloud, it can file MAP migration funding. For startups this is typically $50K to $150K. Depends on the workload; AWS approves. See AWS partner funding.

AWS Activate credits. You apply yourself at aws.amazon.com/startups/credits: Activate Founders gives $1,000 to start, up to $5,000; Activate Portfolio gives up to $200,000 with an Org ID from your VC, accelerator or angel group if it is an Activate Provider. A consulting partner cannot file Activate for you.

If your team can build RAG and you only need Activate credits, apply yourself; you do not need a partner. A partner helps when you want the build done with you, or when the work qualifies for funding only a partner can file.