vLLM on GKE with one L4

To measure with lutflow you need a vLLM server. If you do not have one, this recipe runs Qwen3-4B-Instruct-2507 on vLLM v0.30.0 on one NVIDIA L4 GPU in GKE, measures it from inside the cluster and shuts everything down at the end. The image is pinned by digest and the model by revision.

Before you start

• An existing zonal GKE Standard cluster and permission to create node pools in it. The recipe does not create the cluster.
• gcloud and kubectl configured for that cluster.
• NVIDIA L4 GPU quota in the cluster's region.
• Internet egress from the nodes: the pod pulls the image from Docker Hub, the model from Hugging Face and lutflow from PyPI.
• The list price of a g2-standard-4 in your region, checked on the day you use it, if you want dollar figures.

1. Variables

PROYECTO='<PROYECTO>'   # your GCP project ID
CLUSTER='<CLUSTER>'     # cluster name
ZONA='<ZONA>'           # cluster zone, for example us-central1-a
REGION='<REGION>'       # that zone's region, for example us-central1

Replace every value in <>. Do not post them in issues or screenshots.

Check which zones in the region offer L4:

gcloud compute accelerator-types list --project "$PROYECTO" \
  --filter="name=nvidia-l4 AND zone~$REGION"

Before creating anything, confirm the target context and check that vllm-l4 and gpu-l4 are unused. Do not apply this recipe over existing resources: the cleanup deletes that namespace and pool. Record existing VMs and regional L4 quota usage so you can compare them after cleanup.

kubectl config current-context
kubectl get namespace vllm-l4
gcloud container node-pools list --cluster "$CLUSTER" --zone "$ZONA" --project "$PROYECTO" \
  --filter="name=gpu-l4" --format="value(name)"
gcloud compute instances list --project "$PROYECTO"
gcloud compute regions describe "$REGION" --project "$PROYECTO" \
  --format="value(quotas)"

The namespace lookup should report NotFound; the pool lookup should return no name. If either resource exists, stop and choose an isolated test setup. A permissions or connection error is not proof that a resource is absent.

2. Create the GPU pool

Quota does not guarantee a free L4. On October 14 we could not obtain an On-Demand L4 in the us-central1 zones we tried. That is why the pool spans several zones, starts at 0 nodes, grows to at most 1 and uses --location-policy ANY: the autoscaler takes the first zone that frees an L4.

gcloud container node-pools create gpu-l4 --cluster "$CLUSTER" --zone "$ZONA" --project "$PROYECTO" \
  --node-locations "$REGION-a,$REGION-b,$REGION-c" --num-nodes 0 \
  --enable-autoscaling --total-min-nodes 0 --total-max-nodes 1 --location-policy ANY \
  --machine-type g2-standard-4 --accelerator type=nvidia-l4,count=1,gpu-driver-version=default \
  --image-type UBUNTU_CONTAINERD --disk-type pd-ssd --disk-size 100 \
  --node-taints nvidia.com/gpu=present:NoSchedule

If zone a, b or c was missing from the list above, set --node-locations to the zones that were listed.

3. Deploy vLLM

Save these two manifests. vllm.yaml creates the vllm-l4 namespace, the vllm Deployment and a ClusterIP Service:

apiVersion: v1
kind: Namespace
metadata:
  name: vllm-l4
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: vllm-l4
  labels: {app: vllm}
spec:
  replicas: 1
  strategy: {type: Recreate}
  selector:
    matchLabels: {app: vllm}
  template:
    metadata:
      labels: {app: vllm}
    spec:
      # Required: the "vllm" Service would inject VLLM_PORT=tcp://... and vLLM would read it as its port.
      enableServiceLinks: false
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-l4
      tolerations:
        - {key: nvidia.com/gpu, operator: Equal, value: present, effect: NoSchedule}
      containers:
        - name: vllm
          # vLLM v0.30.0 pinned by digest; the model, by revision.
          image: docker.io/vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
            - --model=Qwen/Qwen3-4B-Instruct-2507
            - --revision=cdbee75f17c01a7cc42f958dc650907174af0554
            - --tokenizer-revision=cdbee75f17c01a7cc42f958dc650907174af0554
            - --max-model-len=8192
            - --gpu-memory-utilization=0.85
            - --max-num-seqs=16
            - --host=0.0.0.0
            - --port=8000
          env:
            - {name: HF_HOME, value: /models/hf}
            - {name: VLLM_NO_USAGE_STATS, value: "1"}
            - {name: DO_NOT_TRACK, value: "1"}
          ports:
            - {name: http, containerPort: 8000}
          resources:
            requests: {cpu: "3", memory: 11Gi, nvidia.com/gpu: "1"}
            limits: {memory: 13Gi, nvidia.com/gpu: "1"}
          readinessProbe:
            httpGet: {path: /health, port: 8000}
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 90
          volumeMounts:
            - {name: modelos, mountPath: /models}
            - {name: shm, mountPath: /dev/shm}
            # torch.compile cache on the pod volume: a container restart does not recompile.
            - {name: modelos, mountPath: /root/.cache/vllm, subPath: vllm-cache}
      volumes:
        - name: modelos
          emptyDir: {sizeLimit: 30Gi}
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 2Gi}
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: vllm-l4
spec:
  type: ClusterIP
  selector: {app: vllm}
  ports:
    - {name: http, port: 8000, targetPort: 8000}

cliente.yaml is a pod to measure from inside the cluster:

apiVersion: v1
kind: Pod
metadata:
  name: cliente
  namespace: vllm-l4
  labels: {app: cliente}
spec:
  enableServiceLinks: false
  restartPolicy: Never
  nodeSelector:
    cloud.google.com/gke-accelerator: nvidia-l4
  tolerations:
    - {key: nvidia.com/gpu, operator: Equal, value: present, effect: NoSchedule}
  containers:
    - name: cliente
      # The same vLLM image, already on the node: it ships python3, pip and uv.
      image: docker.io/vllm/vllm-openai@sha256:8a69ffad015f138d7170c4ddc429e230a3bc1c1719f67e14324749df200a4b90
      command: ["sleep", "7200"]
      resources:
        requests: {cpu: 250m, memory: 512Mi}
        limits: {memory: 1536Mi}

Apply both and wait for vLLM to be ready:

kubectl apply -f vllm.yaml -f cliente.yaml
kubectl -n vllm-l4 get pods -o wide
kubectl -n vllm-l4 rollout status deploy/vllm --timeout=30m

The pods stay Pending until the autoscaler gets an L4. On October 14, once it had one, the node was ready in about 2 minutes, the image was pulled in about 5 and vLLM was ready about 4 minutes later: about 11 minutes in total. If rollout status times out because there is still no L4, run it again.

4. Measure from inside the cluster

Measure from the cliente pod, not from your machine. Through kubectl port-forward, each /metrics read took 0.7 to 1.5 seconds and the tunnel stalled under load; from inside the cluster, a read took 12 ms in that experiment. These timings describe the October 14 setup, not a latency guarantee.

kubectl -n vllm-l4 exec -it cliente -- bash

Inside the pod, first check that vLLM publishes its process identity:

curl -fsS http://vllm:8000/metrics | grep -E '^(process_start_time_seconds|vllm:(prompt|generation)_tokens_created)'

Three lines must appear: process_start_time_seconds, vllm:prompt_tokens_created and vllm:generation_tokens_created. With them, lutflow detects an engine restart inside the window and rejects the window. If none of them appears, a restart whose counters keep rising goes through without a rejection: the report only warns about it. With these manifests, all three appeared on October 14.

With the published SDK, run check from this pod first. It generates no load; a PASS evaluates that exposition, not a future window. See check exits and limits.

uvx --from 'lutflow>=0.3.2,<0.4' lutflow check --endpoint http://vllm:8000

Then check the lutflow version and observe the server:

uvx --from 'lutflow>=0.3.2,<0.4' lutflow --version
uvx --from 'lutflow>=0.3.2,<0.4' lutflow measure --endpoint http://vllm:8000 \
  --duration 120 --interval 5 \
  --json /tmp/medicion.json

measure only observes. If nothing sends requests to this vLLM, cost per useful token is undefined. To generate bounded synthetic load, use probe; it shows a load plan and asks you to type yes before it starts:

uvx --from 'lutflow>=0.3.2,<0.4' lutflow probe --endpoint http://vllm:8000 \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --concurrency 8 --duration 120 --interval 5 \
  --json /tmp/medicion.json

For dollar figures, add these four options to measure or probe, with the g2-standard-4 hourly price you verified. That price covers the whole VM with its single GPU:

--gpu-hourly-rate '<USD_PER_HOUR>' --rate-basis instance --gpus-in-basis 1 --gpu-count 1

Without them, the dollar fields are unavailable; the rest of the report is unchanged.

If uvx is not available in the pod, install lutflow in a virtual environment and use /tmp/l/bin/lutflow instead of uvx --from 'lutflow>=0.3.2,<0.4' lutflow:

python3 -m venv /tmp/l && /tmp/l/bin/pip install -q 'lutflow>=0.3.2,<0.4'
/tmp/l/bin/lutflow --version

To copy the JSON to your machine, from another terminal:

kubectl -n vllm-l4 cp cliente:/tmp/medicion.json ./medicion.json

With vLLM v0.30.0, 0.3.2 derives TPOT from vllm:inter_token_latency_seconds when the older family is absent: a mean per token gap, not per request. The p95 remains interpolated within its bucket; the report now adds a warning identifying its bounds without changing the value or schema. See known limitations.

5. Shut down and verify

kubectl delete namespace vllm-l4
gcloud container node-pools delete gpu-l4 --cluster "$CLUSTER" --zone "$ZONA" --project "$PROYECTO" --quiet

A command that finishes cleanly does not prove the VM is gone. Read the state:

gcloud compute instances list --project "$PROYECTO"
gcloud compute regions describe "$REGION" --project "$PROYECTO" \
  --format='value(quotas)' | tr ';' '\n' | grep "'NVIDIA_L4_GPUS'"

The first list must show no VM from this recipe's gpu-l4 pool. Compare quota usage with the baseline recorded before creating it. Zero is expected only if there were no other L4 allocations and none changed during the test; unrelated workloads can keep usage above zero. Verify that the namespace and node pool are also gone.

Optional: managed DCGM

On a compatible cluster, first review GKE's requirements and your existing monitoring packages. This command configures cluster-level collection; preserve other packages you need and avoid collecting concurrently with a self-deployed DCGM exporter:

gcloud container clusters update "$CLUSTER" --project "$PROYECTO" --location "$ZONA" \
  --enable-managed-prometheus --monitoring=SYSTEM,DCGM

In the October 16 experiment, values updated about every 30 s despite reads every 2 s. DCGM measures the whole GPU; deployment attribution requires an exclusive GPU. It excludes CPU, RAM and network. This collection is separate from the SDK, which does not yet integrate these sensors. Cloud Monitoring ingestion incurs charges.

Lessons from October 14

1. enableServiceLinks: false is required. A Service named vllm injects VLLM_PORT=tcp://... into the pod; vLLM reads it as its own port and does not start.
2. Measure from inside the cluster. In the October 14 experiment, port-forward stalled under load.
3. An L4 may be unavailable even with quota. A pool across several zones with --location-policy ANY and at most 1 node takes the first one that frees up.
4. Load changes modeled cost per token. The same GPU at the same hourly price gives very different cost per token depending on how many concurrent streams it serves. You can see it by repeating probe with --concurrency 1 and --concurrency 16.
5. Verify the teardown by reading the state. A command that finishes cleanly does not prove the VM is gone.
6. To stop vLLM, delete the namespace. On SIGTERM the HTTP server can keep waiting for open connections. Deleting the namespace makes Kubernetes send SIGKILL when the grace period ends.