How to deploy an open-source LLM on AWS — Bedrock, SageMaker JumpStart, or self-host (2026).

IThe three ways to run an open LLM on AWS

Most guides describe one path and present it as "the" way. The honest picture is that AWS gives you three distinct deployment surfaces for an open-weight model, on a spectrum from least-ops/least-control to most-ops/most-control. Picking the right one before you start saves weeks.

By "open-source LLM" — more precisely, an open-weight model — we mean a model whose weights you can download and run yourself: Llama (Meta), Mistral / Mixtral, Qwen, DeepSeek, Gemma, Falcon, and the long tail of fine-tunes built on top of them. Unlike a closed API model, you are responsible for where and how it runs — which is exactly the choice this page is about. The three surfaces below all run inside AWS, under one bill, one IAM model, and (for the self-managed options) your own VPC.

Path 1 — Amazon Bedrock (fully managed, pay per token). Several open-weight models — notably the Llama family and Mistral models — are available directly in the Bedrock catalog. You call them through the same API as Claude or Nova, you provision nothing, and you pay per input/output token. If your model (or a close-enough one) is in the catalog, this is the shortest path to production. For an open-weight model that is not in the catalog, Bedrock also offers Custom Model Import, which lets you bring the weights of a supported architecture (Llama-class and several other common families) and serve them through the Bedrock API on managed infrastructure. The constraint is architecture support — your model must match an importable family.

Path 2 — Amazon SageMaker JumpStart (managed endpoint, your account, your instance). JumpStart is SageMaker's model hub: a catalog of hundreds of open and proprietary models you can deploy to a real-time endpoint in a few clicks or a few lines of the SageMaker SDK. You keep control over the instance type (GPU such as the g- and p-families, or Inferentia inf2), the model runs in your own account and VPC, and AWS manages the serving container, the health checks, and the autoscaling. This is the pragmatic middle ground: far less work than wiring up your own server, far more control than a managed token API. It is the path this page walks step-by-step in section IV.

Path 3 — Self-host on EC2 with vLLM or TGI. The maximum-control path: launch a GPU instance (or an Inferentia/Trainium inf/trn instance) yourself, run a high-throughput inference server — vLLM or Hugging Face Text Generation Inference (TGI) — load the weights, and put the whole thing behind a load balancer or Kubernetes (EKS) with your own autoscaling. You own every layer, you get the lowest unit cost when the box is well-utilized, and you carry all of the operational responsibility — sizing, scaling, patching, and keeping it busy. For steady, high-volume traffic where unit cost dominates, this is where the cheapest cost-per-token lives.

A useful way to hold the three in your head: Bedrock is "the model as an API call," SageMaker JumpStart is "a managed endpoint you own the dial on," and self-hosting is "you run the server." They are not ranked best-to-worst — each wins for a different traffic shape and team. The rest of the page is about matching your situation to the right one.

IIThe tradeoffs: effort, cost, and scaling

The three paths trade the same three things against each other — how much work it is to stand up and run, what it costs per unit of inference, and how it scales with traffic. Understanding the shape of each trade is what makes the decision obvious for your case.

The throughline across all three trades: traffic shape is the deciding variable. Steady, high, predictable traffic rewards the self-managed paths (you keep the box full and the unit cost low). Spiky, low, or unpredictable traffic rewards Bedrock (you pay nothing in the troughs). Most real products have both — a steady baseline plus bursts — which is why the pragmatic answer is so often a mix, covered in the decision section.

IIIWhich open models run where

Before choosing a path you have to know whether your specific model is even available on it. Catalog membership (Bedrock), hub coverage (JumpStart), and "anything you can download" (self-host) are three different levels of availability.

The availability picture, in plain terms. Bedrock's catalog carries selected open-weight families — the Llama models and Mistral models are the headline open options, alongside Amazon Nova/Titan, Claude, Cohere, AI21, and others. If your model is one of those, the managed-API path is open to you with zero deployment. Bedrock Custom Model Import widens this to open-weight models you bring yourself, provided the architecture is supported (Llama-class and several other common transformer families) — you upload the weights and Bedrock serves them on managed infrastructure. SageMaker JumpStart's hub is far broader: hundreds of open models — Llama, Mistral/Mixtral, Qwen, Falcon, Gemma, embeddings and vision models, and more — each deployable to an endpoint you control. And self-hosting has no catalog limit at all: if you can download the weights (from Hugging Face or elsewhere) and they run under vLLM or TGI, you can serve them, including bleeding-edge releases and your own private fine-tunes that no managed catalog has yet.

The practical consequence: your model's availability narrows the choice for you. A current Llama or Mistral model gives you all three options, so you decide purely on traffic and ops. A newer or more niche open model (a fresh Qwen or DeepSeek release, say) may not be in the Bedrock catalog yet — then it is JumpStart (if the hub has it) or self-host. A private fine-tune on a custom base, or a brand-new architecture, usually means JumpStart-with-your-own-artifact or self-host. Confirm the current Bedrock model list and JumpStart hub coverage in the consoles before committing — both expand continually.

IVStep-by-step: deploy an open LLM on SageMaker JumpStart

JumpStart is the path most teams should try first when the model is not already in the Bedrock catalog, because it gives you a production-grade managed endpoint in your own account with very little code. Here is the end-to-end flow, in the order you actually do it.

The walkthrough below assumes you want a real-time endpoint for an open-weight chat/instruct model (a Llama or Mistral instruct variant is the canonical example). The same flow applies to other JumpStart LLMs; only the model ID and the recommended instance change. Exact instance names, container versions, and quotas move — treat the specifics as representative and confirm current values in the console.

• Step 1 — Prepare the account and quotas — In your AWS account, make sure you have a SageMaker execution role with access to S3 and the right permissions, and — critically — that you have service quota for the GPU or inf2 instance type you intend to use (e.g. an ml.g5 / ml.g6 GPU instance or an ml.inf2 instance for the endpoint). New accounts often have a zero quota for large GPU instances; request an increase early because approval can take time. Open SageMaker Studio as your workbench.
• Step 2 — Find the model in JumpStart — In SageMaker Studio, open JumpStart and search for your model (e.g. the Llama or Mistral instruct variant you want). JumpStart shows the model card, the license terms (accept them — open-weight models carry usage licenses you must agree to), and a recommended instance type. You can deploy from the UI or, more reproducibly, from the SageMaker Python SDK using the model's JumpStart model ID.
• Step 3 — Choose the instance and serving config — Pick the instance type and count. For a mid-size open LLM, a single modern GPU instance is often enough; larger models need a multi-GPU instance or sharding, and Inferentia inf2 is the lower-cost option if your model has a supported Neuron path. Set the inference container (JumpStart wires up an optimized one — frequently a TGI- or vLLM-based image — automatically) and the number of model replicas per instance if applicable.
• Step 4 — Deploy the endpoint — Deploy. From the SDK this is essentially a JumpStartModel(...).deploy(...) call with your instance type and initial instance count; from the UI it is a button. SageMaker provisions the instance(s), pulls the optimized container, downloads the weights, and brings up a real-time HTTPS endpoint in your account. Initial deployment typically takes several minutes (model download + container start).
• Step 5 — Invoke and validate — Call the endpoint with the SageMaker runtime (invoke_endpoint) or your application SDK, sending a chat/completion payload in the model's expected format. Validate that outputs are correct, then measure latency — time-to-first-token and inter-token latency under realistic prompt sizes — and confirm it meets your application's requirements.
• Step 6 — Configure autoscaling — Attach Application Auto Scaling to the endpoint: set a target metric (e.g. invocations-per-instance or a concurrency/utilization target), a minimum and maximum instance count, and scale-in/scale-out cooldowns. This is what turns a single fixed instance into an endpoint that grows and shrinks with traffic. Decide here whether you want a warm minimum (lower latency, higher floor cost) or scale-to-near-zero (lower cost, cold-start risk) — covered in section VI.
• Step 7 — Add observability and guardrails — Wire the endpoint into CloudWatch for latency, error-rate, and instance-count metrics; enable SageMaker Model Monitor if you want drift detection; and put your own input/output validation or a safety layer in front for production. Tag the endpoint for cost allocation so the spend is attributable.
• Step 8 — Iterate on cost — Once it is serving real traffic, benchmark cost-per-million-tokens at your actual utilization and compare it against (a) a Bedrock per-token equivalent and (b) the same model on Inferentia inf2 if you started on GPU. This three-way number at your real traffic shape is what tells you whether to stay on JumpStart-GPU, move to inf2, or push spiky overflow to Bedrock. Then apply AWS credits so the instance-hours bill to credits, not your card.

VSelf-hosting on EC2 GPU or Inferentia with vLLM / TGI

When you need full control — a custom serving stack, the absolute lowest unit cost at high volume, or a model nothing else hosts — you run the server yourself on EC2. This is more work, and the work has a known shape.

The self-host pattern is: pick the compute, run a high-throughput inference server, put it behind scaling. On compute, you choose between EC2 GPU instances (the g- and p-families — the familiar CUDA path, broadest model support, no porting) and EC2 Inferentia/Trainium instances (inf2 / trn — AWS's custom silicon, lower cost-per-token at high utilization for supported models, but you serve through the Neuron SDK rather than CUDA and your model must have a clean Neuron compilation path). For the inference server, the two standard choices are vLLM — a high-throughput engine known for PagedAttention and continuous batching, with both CUDA and Neuron support — and Hugging Face Text Generation Inference (TGI), a production-grade server with streaming, continuous batching, and broad model coverage. Both turn raw weights into an efficient, batched, streaming endpoint; the choice between them is mostly ecosystem preference and specific model/feature support.

On scaling and operations, you have two common topologies. The simpler one runs the server on EC2 instances behind an Application Load Balancer with an EC2 Auto Scaling group sized to a utilization or queue-depth metric. The more flexible one runs it as pods on EKS (Kubernetes) with the Horizontal Pod Autoscaler and a GPU/Neuron device plugin, which gives you fine-grained packing, rolling deploys, and multi-model density at the cost of running Kubernetes. Either way you own the parts SageMaker would otherwise manage: the AMI/container build, driver and CUDA (or Neuron SDK) versions, health checks, autoscaling policy, cold-start/warm-pool behavior, and patching. That standing operational load is the real price of self-hosting — it is rarely the model that is hard, it is keeping the fleet healthy, full, and cheap over time.

The reason teams take this on anyway is unit cost at scale and total control. At steady high volume, a well-utilized self-hosted fleet — especially on Inferentia inf2 — generally delivers the lowest cost-per-million-tokens of any path, because you have stripped out every managed-service margin and can tune batching, quantization, and instance choice to your exact model and traffic. And you control everything: the serving engine, the model version, quantization and speculative-decoding tricks, the network boundary, and the deployment cadence. If you are running one or a few high-traffic models where every cent of unit cost matters, self-hosting is the path that lets you drive it to the floor.

VICost math, autoscaling, and cold-starts

The whole decision turns on one number: cost per million tokens at your real utilization. This section shows how to compute it for each path and how autoscaling and cold-starts move it — with representative figures you should replace with current rates and your own benchmarks.

For the self-managed paths (JumpStart and self-host), the unit cost is a division problem: cost per million tokens = (instance $/hour ÷ tokens served per hour) × 1,000,000. The instance hourly rate is fixed; the tokens-per-hour you achieve depends on the model, the instance, and — above all — how effectively you batch concurrent requests. This is why utilization is everything: the same GPU instance serving 10× the tokens per hour has roughly one-tenth the cost per token. A box at 80% utilization can be dramatically cheaper per token than the identical box at 15%, even though the hourly bill is the same. The lever you control is keeping the instance full (good batching + right-sized autoscaling), and choosing cheaper silicon (inf2) where it fits.

For Bedrock, the unit cost is simply the published per-token price for the model (input and output priced separately), with no idle cost — and reducible further with Batch (roughly half-price for non-real-time work) and prompt caching (cuts the cost of repeated context). Because you pay nothing in the troughs, Bedrock's effective cost per token over a spiky day can beat a self-managed box that sat idle for half of it — even when Bedrock's headline per-token price is higher than the self-managed box's best-case per-token cost. The comparison that matters is therefore not "list price vs list price" but "Bedrock's pay-per-use total vs the self-managed box's 24/7 total" across your actual daily traffic curve.

The honest way to choose, then, is to take your real traffic shape — requests per hour across a representative day, with realistic prompt and completion lengths — and compute the monthly cost three ways: Bedrock per-token (including idle troughs at $0), a JumpStart/GPU endpoint at its achievable utilization, and an inf2 endpoint at its achievable utilization. The cheapest depends almost entirely on how steady and high your volume is. The representative table below shows the shape of that answer; your numbers come from current AWS pricing and a benchmark of your specific model.

VIIWhen managed beats self-host (and when it does not)

The instinct to self-host "to save money" is often wrong, and sometimes exactly right. Here is the honest version of when a managed path (Bedrock or JumpStart) beats running your own server — and the narrower set of cases where self-hosting genuinely wins.

Managed wins more often than engineers expect, for one structural reason: the cost advantage of self-hosting is conditional on high utilization, but the operational cost of self-hosting is unconditional. You pay the ops tax — building, scaling, patching, on-call — whether the box is full or empty, and you only collect the unit-cost reward if you keep it full. For the many teams whose traffic is not yet steady-and-high, a managed path is both cheaper (no idle burn, no engineer-months) and faster. The time-to-value gap compounds the point: Bedrock is live in minutes and JumpStart in an afternoon, while a hardened self-hosted fleet with proper autoscaling and observability is a multi-week project that then needs ongoing care.

VIIIHow CloudRoute gets the model deployed — and the bill to $0

Deploying an open LLM on AWS is two problems: doing the deployment well (the right path, tuned autoscaling, optimized unit cost) and paying for it. A vetted AWS partner plus AWS credits solve both — and the credits take the cash cost to zero.

Whichever path fits, the compute it runs on is exactly what AWS credits are designed to cover. SageMaker endpoint instance-hours and self-hosted EC2 GPU/inf2 hours are standard AWS compute; Bedrock tokens are standard Bedrock usage. All of it draws from the same credit pools: AWS Activate (up to $100K), Bedrock / GenAI PoC funding ($10K–$50K to prove out a model or feature), and the Generative AI Accelerator (up to $1M for selected AI-first companies). For an inference workload — which, once live, runs continuously — that credit runway can cover a production open-LLM deployment for a long time.

But the deployment itself benefits from expertise that is easy to underestimate. Choosing the path (Bedrock vs JumpStart vs self-host), picking GPU versus Inferentia, getting a clean Neuron compilation path if you go inf2, tuning autoscaling and cold-start behavior, and routing traffic so each slice runs where it is cheapest — this is real engineering, and getting it wrong is the difference between a deployment that quietly drains budget and one that is fast, cheap, and stable. CloudRoute (cloudroutehq.com) addresses both halves: it routes you to a vetted AWS partner who actually does the deployment — stands up the JumpStart endpoint or the vLLM/TGI fleet or the Bedrock import, tunes the autoscaling and the FinOps, and benchmarks the unit cost across paths — and files the AWS credit applications through the ACE program so the bill is funded from day one.

The economics for you: the customer pays $0. AWS funds the credit pool because it wants production GenAI on AWS for the long term; the partner is paid by AWS through engagement-funding programs; CloudRoute is paid by the partner as a routing commission. You never see an invoice from CloudRoute. You get an open LLM deployed on the right path, autoscaling and unit cost tuned by people who do this daily, and credits that cover the GPU/accelerator hours or Bedrock tokens — a deployment that is funded and optimized rather than improvised and expensive. For the broader credit picture, see AWS credits for generative-AI startups and how AWS PoC / Bedrock POC funding works.