Amazon Bedrock model evaluation: pick a model with evidence from your own prompts.

IWhat Amazon Bedrock model evaluation is

A managed way to measure how well a model, or a whole RAG system, does your task, so you choose with data instead of a leaderboard.

Bedrock offers many models at very different prices and speeds. Picking by a public ranking is how teams pay for a top-tier model on a task a cheaper one handles just as well, or ship a model that demos well and fails on real inputs. An evaluation runs your prompts through each candidate and scores the results the same way.

You create an evaluation job in the console or API, choose what to evaluate, supply a prompt dataset (or use a built-in one), pick metrics, and Bedrock writes scores and the generated answers to an S3 bucket you control. According to the AWS docs, jobs can target foundation models, Bedrock Marketplace models, customized and imported models, prompt routers and Provisioned Throughput models, as well as models and RAG sources outside Bedrock.

IIThe four kinds of evaluation job

They trade cost and speed against how well they capture subjective quality. Many evaluations use more than one.

IIIWhich evaluation method should you use?

Match the method to what "good" means for the task, not to what sounds most rigorous.

• Checkable right answer: programmatic: Classification, extraction and question answering with known answers. Fast, cheap and repeatable.
• Subjective and high-stakes: human: Medical, legal or brand-critical output, where a wrong call is expensive. Use your own experts with a clear rubric.
• Subjective at volume: model as a judge: Thousands of open-ended answers scored on helpfulness, coherence or relevance. Check the judge against human ratings first.
• Answers from your documents: RAG evaluation: Scores retrieval and answers separately, so you know which half to fix.
• Safety: built-in toxicity metrics plus a human spot-check: Then enforce limits at runtime with Bedrock Guardrails.

IVEvaluating a RAG system: two halves, two fixes

A bad RAG answer has one of two causes, and they need opposite fixes.

Retrieval failed: the system pulled the wrong or incomplete passages, so even a perfect model could not answer. Fix chunking, the embedding model, the number of results, metadata filters or the source documents. Generation failed: the right passages came back, but the model ignored them, added facts, or answered only part of the question. Fix the prompt ("answer only from the context"), the model, or how context is assembled.

A RAG evaluation scores retrieval (are the passages relevant, and do they cover what is needed?) separately from the answer (is it faithful to the passages, relevant and complete?). Without that split you tune the wrong half. See the Knowledge Bases guide for the pipeline being measured.

VBuilding a dataset that predicts production

An evaluation is only as good as its prompts. Easy, hand-picked prompts give a flattering score that fails on real users.

• A JSON Lines file in S3: one example per line, with the prompt and, for reference-based metrics, the expected answer. Check the AWS docs for the exact fields your job type expects.
• Prompts that look like real traffic: same phrasing, length and mess.
• The hard cases on purpose: ambiguous, edge, adversarial and out-of-scope inputs, where models differ.
• Balance across the categories you care about. A few hundred well-chosen prompts usually beat thousands of near-duplicates.
• Keep it separate from any fine-tuning data, or you measure memory, not skill.
• Save it and re-run it whenever a new model ships.

VIThe metrics, and how to choose them

Score on the two or three metrics that match how your product can fail, not on all of them.

• Accuracy: How often the output matches the expected answer. The core metric when there is a right answer.
• Robustness: Whether the output stays stable when the input has typos or is reworded. Catches models that only work on clean prompts.
• Toxicity: How likely the output is to be harmful. A gate for any user-facing product.
• Faithfulness (RAG): Whether the answer sticks to the retrieved passages. The key answer-side metric for a knowledge assistant.
• Relevance and completeness: Whether the answer, and the retrieved context, address the whole question.
• Open-ended quality: Helpfulness, coherence and tone. Scored by people or a judge model against your rubric, not by an algorithm.

VIIFrom scores to a decision: choose, then right-size

The useful outcome is usually not "the best model wins". It is "the cheapest model that clears the bar wins".

• Set the quality bar before you look at cost: For example, a minimum faithfulness score and a maximum toxicity score. Then cost cannot bias the call.
• Pick the cheapest candidate that clears it: Per-token cost and latency repeat on every request you will ever serve.
• Compare setups, not only models: A smaller model with RAG and a better prompt often beats a larger model alone. Evaluate a fine-tuned model against its base model with good prompting before you pay to host it.
• Re-run on a schedule: New models ship often. A saved dataset tells you cheaply when a newer or cheaper model clears your bar.

For live traffic, a prompt router can make a version of this choice per request. Evaluate the router itself as a candidate.

VIIIWhat an evaluation costs

Mostly the inference: every candidate answers every prompt.

On 10 October 2026 the Bedrock pricing page said: you pay for model inference at the selected model's rates; automatically generated algorithmic scores carry no extra charge; human evaluation with your own workforce costs $0.21 per completed human task; and judge model tokens bill at on-demand standard rates. Add S3 storage for datasets and results. The cost is usually small next to the saving from picking a cheaper model that is good enough.

IXWho can file AWS funding for a Bedrock build

There are two kinds of AWS money for a Bedrock build. A partner files one. You apply for the other yourself.

Partner-filed funding. An AWS consulting partner can file POC (proof of concept) funding to test a Bedrock workload, and model selection with evidence is a normal part of that test. If you already run on AWS and are adding new AI work, the partner can file incremental workload (IW) funding instead. For startups this is often $50K to $150K. Depends on the workload; AWS approves. The work has to be real, and the partner has to be the one doing it.

AWS Activate, which you apply for yourself. Activate Founders gives $1,000 to start, up to $5,000, with no investor needed. Activate Portfolio gives up to $200,000 if your VC, accelerator or angel group is an Activate Provider and gives you its Org ID. A consulting partner cannot apply for Activate on your behalf. AWS made Activate credits usable for third-party models on Bedrock in April 2024.

For most startups without a VC, the partner route is the larger of the two. CloudRoute matches you with one vetted AWS partner for free. Partners pay CloudRoute a monthly membership plus 10% of AWS-approved funding.