A reference router and a token-cost experiment on GKE

Context: Saturday’s talk

On Saturday, October 17, 2026, I am presenting at AWS Community Day Paraguay, the first edition of the event in the country, the talk “¿Y si tu factura dejara de contar tokens? Router, costo real y observabilidad de inferencia en AWS” (given in Spanish; roughly “What if your bill stopped counting tokens? Router, real cost and inference observability on AWS”). Lutflow is part of AWS Activate.

The talk starts from a practical question. A router can send routine tasks to an open model on your own GPU and keep Amazon Bedrock for the harder ones. What does each path really cost? We first explain the reference architecture published on AWS Builder Center. We then show how we measured load's effect on cost per token in GKE. These are two different experiments: our lab does not reproduce the router or validate its quality results.

Every number carries the same label as on the slides:

• CITED an external source says it, linked.
• MEASURED it comes from counters observed in our lab.
• MODELED it combines something measured with a stated assumption, such as a list rate.
• PROPOSED it is designed and has no data yet.

The reference architecture on AWSCITED

On September 24, 2026, Ben Baruch, from AWS, published on AWS Builder Center “Fixing the token economy: a cost-aware LLM router proven on Amazon SageMaker AI and Amazon Bedrock”. Here is my own summary; the original is worth reading.

The architecture exposes a single OpenAI-compatible endpoint that chooses between two destinations behind the scenes:

• Router. vLLM Semantic Router runs as an Envoy ext_proc filter on Amazon EKS. For each request it estimates the domain and a complexity score and compares it with a threshold.
• Local path. Routine requests go to Qwen3.8-27B-AWQ-INT4, a 27B-parameter model quantized to 4 bits, on an Amazon SageMaker AI real-time inference endpoint with a single GPU.
• Large path. Harder ones go to Claude Sonnet or Claude Opus on Amazon Bedrock, depending on difficulty.
• Gateway. Amazon Bedrock AgentCore Gateway fronts both backends with a single IAM role.
• Verifier and rescue. A lightweight verifier checks every answer from the local model (unexpected refusals, answers cut mid-way, invalid syntax, low-confidence classification) and, when something looks wrong, rescues it through Claude before it reaches the user.

What this architecture solves

The router chooses a destination for each request; SageMaker AI hosts the self-hosted model and Bedrock serves the more capable path. The gateway unifies access and permissions. If the verifier rescues a local answer, both model calls run and both costs count. The economics depend on the request mix, required quality and GPU load, not just the advertised token price.

Its numbers, with their caveats

• ~USD 2.93 per million generated tokens on the local path. The author computes it “at full GPU utilization”: with the GPU working at full capacity.
• 74.2% of requests were served entirely by the local model.
• 9.7% escalation to the rescue path, down from 25.8% before tuning.
• 37.5% quality parity: local answers that a judge model rated equal to or better than Claude’s. The author notes it is a small sample, to be read as a signal rather than a precise metric.
• The GPU type is not named. The text only says it is a single-GPU instance.

The article also projects monthly and yearly amounts for teams of different sizes. Those are the author’s projections, computed from his measured throughput and published list prices; we do not reproduce them here, and they are in the original.

The talk’s question: what if the GPU is not at full capacity?MODELED

That caveat, “at full utilization”, is where the talk starts. A real service does not spend the day with a full GPU: demand rises and falls, and allocated capacity is paid for while it waits. There are three ways to price the same tokens:

• Allocated capacity. You pay every hour the GPU is reserved, busy or idle. The denominator is the tokens of the whole window.
• Active time. You attribute cost only to the intervals with work. It helps to understand serving efficiency, but it is an attribution: the provider bills the whole hour.
• API. You pay input and output tokens, each at its rate. There is no waiting to pay for, and no control over the model or the hardware either.

A synthetic example, the same one as on the talk’s slide: in one hour, 20,000 tokens are generated on a USD 2.40/h GPU that works for 10 minutes and waits for 50.

The same 20,000 tokens cost six times more per token under the allocated account than under the active one. Neither is “the real one”: they answer different questions. What you cannot do is compare a cost computed at full capacity with an API price without saying which denominator was used.

What we measured: one NVIDIA L4 on GKEMEASURED

To see how much the cost per token moves with load, we set up a small lab and measured it with the Lutflow SDK. The lab the talk proposes for AWS (EC2 and EKS) is PROPOSED and has no data yet: these numbers are from Google Kubernetes Engine.

How the SDK knows a window can be defended

Measuring with counters looks simple: read vllm:generation_tokens_total at the start and the end of the window, and subtract. But that subtraction only holds if the process serving the model is the same from end to end. If vLLM restarts in the middle, the counter goes back to zero and the subtraction mixes two processes.

That is why the SDK reads two identity signals vLLM publishes on /metrics:

• process_start_time_seconds: when the process started. If it changes between two reads, there was a restart.
• The _created series of each counter, such as vllm:generation_tokens_created: when the counter was born. If it changes, the counter went back to zero even if its value keeps rising.

If either one moves inside the window, the SDK does not emit a number for that window. And if a value cannot be verified (a series the engine does not publish, a sensor that does not exist), the report marks it UNAVAILABLE: it never fills it with a zero or an estimate. In an earlier validation on this same setup we restarted vLLM in the middle of a loaded window: version 0.3.0 ended without a report, while an older version published a partial total as if it were valid.

The results

Reading. On the same GPU at the same hourly price, the modeled cost per output token falls about 10 times between 1 and 16 streams. The GPU did not change: what changed is how much work went through it in each paid hour.

Latency is the price. The client-side p95 latency MEASURED went from 37 s with 1 stream to 60 s with 16, with answers of up to 1024 tokens. With 1 stream there were only 5 answers in the window, so that p95 is not robust. And 16 streams is not a demonstrated ceiling for the L4: we did not observe a plateau, it is simply the last level we ran.

Quality not measured. More than half of the answers stopped at the maximum length. We counted generated tokens, not accepted tasks: answer quality was not evaluated in this lab.

Against Bedrock, at list priceMODELED

With the same mix of input and output tokens, we compared the L4’s cost per processed token with the list price of Qwen3-32B on Amazon Bedrock. The self-hosted L4 comes out 8.2 times more expensive with 1 stream and at 0.80 times the Bedrock price with 16. The crossover is between 8 and 16 streams.

The caveat travels with the number: these are different models (4B versus 32B) and quality was not compared. What it shows is the mechanism: the API price per token is fixed, while the self-hosted GPU’s depends on its load.

If the same L4 ran on AWSMODELED

For the talk we projected the same throughput onto an AWS g6.xlarge instance, which has the same GPU, at its On-Demand list price in us-east-1, USD 0.8048/h. C_out goes from USD 7.68 with 1 stream to USD 0.759 with 16 per million output tokens, and against Bedrock from 9.3× to 0.91×. It is a projection: it assumes g6.xlarge delivers the same throughput we measured on GKE, even though the host (CPU and network) is different. It is not measured on AWS.

Bridging to the reference case

We do not compare our numbers with those in the AWS Builder Center article. The model (27B quantized to 4 bits versus 4B in BF16), the GPU (unnamed versus an L4), the platform (SageMaker AI versus GKE) and the load are all different. Putting them side by side would compare two different experiments.

What we do share is a point of method. In our lab, the cost per token of the same GPU varied about 10 times between 1 and 16 concurrent streams. That matches the author’s own caveat: his USD 2.93 per million is computed with the GPU at full capacity. When planning a router like this, the useful question is not only “what does a local token cost?” but “how much load will that GPU carry over the day?”. The local path’s cost per token depends on that answer; the API’s does not.

How we instrumented the GKE experiment

Before measuring, you need an endpoint that can be measured. For the lab we used a Lutflow deploy CLI that is not public yet. It deploys vLLM on GKE with a least-privilege identity, waits for the server to be Ready, and then checks that it is measurable: that its /metrics publishes what it takes to defend a window (the process start time, the token counters with their type and their _created series, and the expected model). It then suggests a read-only measurement command and, when done, tears down the deployment.

From there, the flow is the one in the diagram:

1. vLLM publishes counters on /metrics: input and output tokens, running and waiting requests, KV cache usage and the identity signals.
2. The SDK builds windows: it reads /metrics at intervals, computes deltas and assigns a state to every metric.
3. The local report tier0.measure.v1 stays on your machine, with every value labeled measured, derived or unavailable.
4. The modeled cost is the list rate divided by the measured throughput.

The formula and the rate

C_out = rate (USD/h) ÷ 3600 ÷ output throughput (tok/s) × 10⁶

With 16 streams: 0.706832 ÷ 3600 ÷ 294.4 × 10⁶ ≈ USD 0.667 per million output tokens. The published numbers use the unrounded throughput. The rate is the Google Cloud Billing Catalog list price for us-central1, On-Demand, read on October 14, 2026, built up by SKU:

How to reproduce it

The SDK is on PyPI and runs on Python 3.9 to 3.15. Install it with pipx, or run it without installing via uvx:

pipx install 'lutflow>=0.3.2,<0.4'

# or, without installing anything:
uvx --from 'lutflow>=0.3.2,<0.4' lutflow --help

Check before measuring: lutflow check

From published release 0.3.1, lutflow check checks one engine exposition before measuring. Use a reachable endpoint and the served model name. It generates no load and does not replace a measurement window.

lutflow check --endpoint http://<host>:8000 \
  --model <served-model> --json check.json

Exits: 0 PASS, 1 demonstrated FAIL, 3 no verdict because a read failed, 2 usage error. The tier0.check.v1 JSON can retain host and path: review it before sharing. A profile FAIL does not by itself mean measure cannot detect restarts. Check rules, proxy access and limits.

To observe a vLLM server that is already running, without generating load:

lutflow measure --endpoint http://<host>:8000 \
  --model <served-model> \
  --duration 120 \
  --gpu-hourly-rate <usd-per-hour> --rate-basis instance \
  --gpus-in-basis 1 --gpu-count 1 \
  --json report.json

<host> is the address of the vLLM server (the engine pod, not a router in front of it) and <usd-per-hour> is a rate you verified yourself: the SDK ships no default. Without a rate, the report carries the throughput and leaves the costs empty instead of guessing them. If you need to induce test load, lutflow probe uses synthetic prompts, has caps that cannot be raised (16 streams, 600 s) and asks for confirmation before generating traffic: it consumes GPU that others may be using.

Full documentation is at lutflow.dev/docs (Spanish: lutflow.dev/docs/es).

Limits: what these numbers do not say

• The SDK does not measure physical GPU utilization. It reads inference-server counters, not chip sensors. The physical GPU and answer quality need other instruments.
• It is not a bill, and not a saving. It is the hourly list rate divided by measured throughput, with unit conversion. What a provider charges depends on the account, discounts, disk, network and idle time.
• The load is induced. A lab corpus at fixed concurrency does not describe real production traffic.
• It is one configuration. One model, one GPU, one server and one platform: GKE, not AWS. The AWS projection is modeled.
• Bedrock, with a caveat. The comparison uses different models (4B versus 32B), and quality was not compared.
• The deploy CLI does not verify the GCP project number. It takes it as declared: the deployment’s identity is anchored by the project, the location, the cluster and the identifiers of the objects it creates.