Lutflow SDK

Tier 0 measures a vLLM server you already run. It reads the server's Prometheus metrics over a time window and reports throughput, idle capacity and active efficiency. With a declared hourly rate, it also models allocated cost per processed token. No Lutflow account, hosted backend, Kubernetes agent or GPU sensor is required. This page describes lutflow 0.3.2; releases 0.2.0 and 0.2.1 were withdrawn (yanked) from PyPI.

Prerequisites

• Python 3.9+ for the CLI; the anthropic and all extras require Python 3.10+.
• A running vLLM endpoint with /metrics reachable from this machine. If you do not have one, see vLLM on GKE with one L4.
• For dollar figures: your real GPU hourly price, whether it covers an entire instance or one GPU, and the number of GPUs it covers and this deployment uses.

Check the counters before installing (change the URL for your server):

curl -fsS http://localhost:8000/metrics | grep -E 'vllm:(prompt|generation)_tokens_total'

If no counters appear, check the address and vLLM metrics configuration. The live measurement path does not support TGI or Ollama metrics yet.

Install

The quickest way to use the lutflow command is uv or pipx. Each keeps lutflow in its own isolated environment, so nothing you already have installed changes:

# uv: run it without installing anything permanently
uvx --from 'lutflow>=0.3.2,<0.4' lutflow --version

# uv: install the command for good
uv tool install 'lutflow>=0.3.2,<0.4'

# pipx: install the command for good
pipx install 'lutflow>=0.3.2,<0.4'

The version range makes the install fail instead of quietly picking an older release. With uvx, write every command as uvx --from 'lutflow>=0.3.2,<0.4' lutflow ...; after uv tool install or pipx install, the command is just lutflow.

To use lutflow from your own Python code, install it in a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install 'lutflow>=0.3.2,<0.4'
lutflow --version

In a uv project, uv add 'lutflow>=0.3.2,<0.4' does the same.

Check the endpoint before measuring

Measurable means one engine-process identity and attributable counters, with the typed families and creation markers required by the profile.

lutflow check is available from 0.3.1. It reads one vLLM /metrics exposition without generating inference traffic; it is not a measurement. Point it at the engine, not a router. Use the served model name when adding --model:

lutflow check --endpoint http://localhost:8000 \
  --model Qwen/Qwen2.5-3B-Instruct --json check.json

# Or inspect a saved exposition without contacting a server:
lutflow check --metrics-file saved.prom --json check.json

The vLLM v0.30.0 profile requires exactly one process_start_time_seconds, five present and typed metric families, and matching _created samples for engine counters in each labelset. With --model, descriptor samples must match that name and a live /v1/models must list it. An older vLLM can fail this profile while measure still works; a FAIL does not by itself prove restart detection is impossible.

If --json cannot be written, exit is 1 even after a PASS. An unreadable endpoint is exit 3 for check, but 2 for measure/probe.

check.json uses tier0.check.v1: verdict, reasons/findings, required and found families, model information, time, source and an exposition fingerprint. Unparsed families and model labels are null, not an empty list; models_served is null when not requested. The endpoint retains scheme, host, port and path, but strips credentials, query and fragment. Unlike a measurement report, it can reveal infrastructure addresses: review it before sharing.

A PASS establishes the identity signals available at that instant. It does not inspect GPU sensors, nodes, quotas or the pod specification, and it does not certify a later measurement window, latency, cost or answer quality. Use measure or confirmed probe over a window next.

Kubernetes access and request environment

Read-only pods/proxy GET access in the namespace is enough for check through a local kubectl proxy; its endpoint can include the proxy path. This does not authorize a deployment or require a port-forward:

lutflow check --endpoint \
  'http://127.0.0.1:8001/api/v1/namespaces/<NAMESPACE>/pods/<ENGINE_POD>:8000/proxy'

probe needs an in-cluster client or a port-forward with permission to generate load; the path-bearing proxy base used by check does not work for it. For measure through this proxy, use the complete URL ending in /proxy/metrics.

Requests refuse redirects and use a 5 s timeout. HTTPX uses trust_env: proxy variables (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, NO_PROXY) and CA variables (SSL_CERT_FILE, SSL_CERT_DIR) affect routing and TLS. The supported HTTPX versions do not load .netrc credentials through trust_env. See the versioned check contract.

Observe existing traffic

Replace 0.85 with the hourly price from your bill or provider price sheet:

lutflow measure --endpoint http://localhost:8000 \
  --duration 300 --interval 10 \
  --gpu-hourly-rate 0.85 --rate-basis instance \
  --gpus-in-basis 1 --gpu-count 1 \
  --json measurement.json

instance means the price covers the whole machine. Use per_gpu if it covers one GPU. Set --gpus-in-basis to the GPUs covered by an instance price and --gpu-count to the GPUs paid for by this deployment. Lutflow does not guess these numbers or silently substitute a price. Without a verified price, omit all four options: the report still comes out, with dollar fields unavailable. measure only reads metrics; it does not send inference requests.

The report separates allocated economics for the whole window, including idle time, from active efficiency during productive intervals. In this report, "useful tokens" are prompt plus generated tokens, not a measure of answer quality. Dollar figures are derived from the declared rate and observed work; they are not reconciled billing charges. With no useful work, cost per useful token is undefined. Power, energy, VRAM and GPU utilization require a separate sensor and report unavailable, never zero, in this Tier 0 path. The JSON file stays on your machine.

If the endpoint serves multiple models, pass --model MODEL_ID to select one; Lutflow does not combine their token counts into a misleading number.

A window must come from one engine process with continuous counters. From 0.3.0, if the engine restarts, a token counter disappears from some scrapes, or the engine's process identity changes inside the window, measure and probe end with exit code 1 and no report. Version 0.2.1 reported some of those windows with exit 0. Detecting a restart whose counters keep rising requires the engine to publish its process identity; see known limitations.

Generate bounded test traffic

Use probe only when you are authorized to generate requests against the server. Replace the model ID and price:

lutflow probe --endpoint http://localhost:8000 \
  --model Qwen/Qwen2.5-3B-Instruct \
  --concurrency 8 --duration 120 --interval 10 \
  --gpu-hourly-rate 0.85 --rate-basis instance \
  --gpus-in-basis 1 --gpu-count 1 \
  --json measurement.json

probe displays a load plan and requires you to type yes. It uses packaged synthetic prompts and labels the output as induced load. Its caps are built into the package and cannot be raised: 16 concurrent streams, 600 seconds and 5000 requests. It never runs implicitly as part of measure.

What's new in 0.3.2

• TPOT on vLLM v0.30.0: when vllm:time_per_output_token_seconds is absent, use vllm:inter_token_latency_seconds. It is the window mean per output-token gap, Δsum / Δcount, labeled derived, not a per-request mean. The older family retains priority when present; vllm:request_time_per_output_token_seconds is not used.
• TPOT continuity: compare sums and counts within the same family, including across gaps. A family switch keeps the window with TPOT unavailable, without a false reset. Resets within a family, token-counter gaps and process-identity changes still reject the window.
• p95: interpolation within a bucket adds latency_p95_bucket_bounded_<lower>_<upper>_ms to warnings[]. The value, its source and the tier0.measure.v1 schema do not change.

Known limitations

These are the known limitations still present in 0.3.2, from its versioned changelog. Plans for a later release do not change the behavior of the package installed here.

• Restart detection depends on the engine publishing its identity. If vLLM publishes neither process_start_time_seconds nor the _created series of its token counters, a restart whose counters keep rising can go unnoticed. The window is then not rejected: warnings[] carries engine_process_identity_unpublished_restart_with_rising_counters_not_detectable. Before measuring, check that /metrics contains those series.
• The p95 resolves only to a histogram bucket. It is interpolated inside the engine's bucket that contains it. With vLLM's default end-to-end buckets, a window whose requests all take 2.5-5 seconds reports 4875 ms whatever the real tail. Read it as "within that bucket".
• Wrapper prices need review. The built-in table remains dated, with separate input and output rates. Unknown models silently use gpt-4o-mini pricing. The price_per_1k_usd override applies one rate per thousand tokens to both input and output. lutflow measure uses your supplied rates, not this table.

Other SDK capabilities

lutflow measure --from-telemetry export.jsonl remains available for an existing GPU telemetry export. This is an offline analysis path with different data requirements from --endpoint.

The Python SDK also offers OpenAI, Anthropic and Gemini wrappers with in-process budget actions such as raise, warn and callback. Those actions affect your Python process, not a Kubernetes deployment or cluster. See the package README for API examples and the license.