Amazon Bedrock batch inference: when and how to use it.

IWhat Amazon Bedrock batch inference is

Batch inference trades latency for a lower price. A large share of real GenAI work fits its shape better than the real-time path.

On on-demand you send one request, wait for one response and pay the full rate. That is right for anything a person waits on. But much GenAI work is not latency-sensitive: classifying ten million tickets overnight, embedding a corpus, enriching a dataset, scoring an evaluation set, generating synthetic data. You need the whole job done by a deadline, cheaply.

With batch, you put all requests into one job (a file of inputs in S3). Bedrock processes them in the background and writes results to S3. AWS prices batch at 50% lower than on-demand for select models (see the Bedrock pricing page). The output is the same as on-demand; only delivery and price differ.

The mental model: on-demand is a conversation, batch is a queue. There is no model-quality trade-off and no reserved-capacity commitment. You move a workload that does not need real time onto the batch path, and its token bill roughly halves. See the batch inference user guide.

IIHow to submit a batch job: JSONL in S3, output to S3

A file of inputs in S3, a file of outputs in S3, and one API call to start the job.

• 1. Format inputs as JSONL in S3: One JSON object per line. Each line has a record ID and a modelInput object shaped like the body that model takes on a normal InvokeModel call. Very large jobs are split across several input files.
• 2. Pick an output S3 location: Choose a prefix for results. Keep input and output prefixes (ideally buckets) separate so a job never reads its own output.
• 3. Grant an IAM service role: Bedrock assumes a role to read the input and write the output. Its trust policy must allow the Bedrock batch service, and its permissions must cover the exact S3 paths. This is the step that most often fails.
• 4. Call CreateModelInvocationJob: From the console, CLI or an SDK, pass the model ID, input S3 URI, output S3 URI, role ARN and a job name. Bedrock returns a job ARN.
• 5. Poll, then read S3: The job moves from Submitted to InProgress to Completed (or Failed or Stopped). Poll with GetModelInvocationJob or react to EventBridge events. Then read the output JSONL and join each modelOutput back to your data by record ID.

Every record is independent: there is no shared conversation, so batch is for parallel work (classify each item, embed each chunk). And it is file in, file out through S3, so it fits next to Glue, Athena or Step Functions in a data pipeline. Moving a workload from on-demand to batch is mostly writing the same request bodies to a file. Check the current docs for exact field names and job states.

IIIWhich models support batch inference

AWS lists batch support for select models from Anthropic, Meta, Mistral AI and Amazon, including embedding models used for corpus backfills.

Batch covers many high-volume text models (Anthropic Claude, Amazon Nova, Llama, Mistral) and embedding models (Amazon Titan Text Embeddings, Cohere Embed). Not every model supports it, and coverage changes, so confirm your model in the current docs before you design a job.

For batch, pick the cheapest model that clears the quality bar, because batch is where volume is highest. A nightly job over hundreds of millions of tokens rarely needs a frontier model if a small model (Nova Micro or Lite, a Claude Haiku model, a small Llama or Mistral) does the job. The rate difference, times batch volume, then halved, is the gap between a cheap job and an expensive one.

For embeddings, batch is the natural home for the initial backfill of a corpus and for re-embedding when content or the embedding model changes.

IVWhen to use batch inference, and when not to

Use batch when the work is high-volume and nobody is waiting on any single answer.

When not to use batch: anything a person or a synchronous system is waiting on, such as chat, agent loops, autocomplete or in-path moderation. Those belong on on-demand, often with prompt caching. The test: is something waiting on this answer right now? If not, use batch.

• Bulk classification and tagging: Categorizing, routing or moderating a backlog of tickets, listings, documents or user content. Usually best on a small model.
• Embeddings backfill: Embedding a whole corpus to build or rebuild a vector index for RAG or search, and re-embedding after changes.
• Dataset enrichment: Adding model-generated fields (summaries, entities, sentiment, translations) to a dataset as a scheduled job.
• Model evaluations: Running one or more models over a large evaluation set. Batch makes it cheap to evaluate often.
• Synthetic data generation: Producing large volumes of examples for training, testing or red-teaming.
• Offline content generation: Generating descriptions, variants or summaries ahead of time and serving them later.

VQuotas, limits and turnaround

Batch has no real-time completion promise. Plan for predictability, not speed.

Turnaround. A job runs as capacity allows. Small jobs can finish in minutes; very large ones can take many hours. Build the deadline into your schedule: submit the nightly job with plenty of margin before anyone needs the results.

Quotas. Bedrock limits concurrent batch jobs, input file size and records per job, by account, model and Region. Split large datasets into several files or jobs. Many limits can be raised through Service Quotas; check the current values rather than hard-coding them.

Failures. Single records can fail (bad input, over a limit) without failing the job. Reconcile inputs against outputs by record ID and resubmit failures. Treat batch as a data pipeline with reconciliation, not a fire-and-forget call.

VIThe cost math vs on-demand: a worked example

A representative example you can reproduce for your own job.

The job. Summarize and tag 2,000,000 documents a month, each about 1,500 input tokens and 250 output tokens. That is 3 billion input tokens and 500 million output tokens a month.

On-demand on Amazon Nova Lite (representative rates of $0.06 per 1M input and $0.24 per 1M output): 3,000 × $0.06 = $180 input plus 500 × $0.24 = $120 output, about $300 a month. On batch, about $150 a month.

Model choice matters more. The same job on a Sonnet-class model at $3 and $15 per 1M is about $16,500 a month on-demand, or about $8,250 on batch. Right-size the model first, then halve it with batch.

These are illustrations; rates change by model and Region. Confirm current prices on the AWS Bedrock pricing page.

VIICombining batch with other Bedrock cost levers

Route each path of a product to its cheapest suitable mode instead of picking one global setting.

The main ways to pay on Bedrock are on-demand (per token, real time), batch (lower price, asynchronous), Provisioned Throughput (reserved capacity by the hour) and prompt caching (a discount on repeated input on the real-time path). One product often uses several.

Batch plus model right-sizing is the strongest pairing: the savings multiply.

Batch vs prompt caching depends on traffic shape. Caching discounts a repeated prefix on interactive traffic; batch discounts offline bulk work.

Batch vs Provisioned Throughput is about latency and predictability. Provisioned reserves capacity for steady real-time volume (and is one way to serve fine-tuned models). Batch suits periodic, asynchronous bulk work where you would rather pay per token than reserve capacity.

VIIIWho can fund a batch pipeline build

Batch and on-demand inference, embeddings and the S3 and pipeline services around them all draw down AWS credits automatically.

Partner-filed funding: a vetted AWS partner can file POC (proof of concept) funding to prove a new GenAI use case (often the first embeddings backfill or evaluation run), or incremental workload (IW) funding if you already run on AWS and this is new work. For startups this is typically $50K to $150K. Depends on the workload; AWS approves.

AWS Activate: you apply yourself. Activate Founders gives self-funded startups $1,000 to start, up to $5,000. Activate Portfolio gives up to $200,000 with the Org ID from your VC, accelerator or angel group, if it is an Activate Provider. A consulting partner cannot file Activate for you.

CloudRoute matches you with one vetted AWS partner who can build the batch pipeline (S3 layout, job submission, reconciliation, model right-sizing) and check which funding fits. Startups pay CloudRoute nothing. Partners pay CloudRoute a monthly membership plus 10% of AWS-approved funding.