Qwen3-8B on 1x H100 SXM (GCP), pipeline validation
The runs used to build and debug the benchmark pipeline: BF16 performance, an accuracy re-run, and a full FP8 campaign, plus a smoke test. The code ran from uncommitted working trees; the exact code bundles (minus private Terraform state) are in the release.
Exactly what ran
Reproductions must use this provider, machine, image and code. The agent refuses anything else.
- Provider
- GCP · spot
- Machine
- a3-highgpu-1g · us-central1-a
- Boot image
- common-cu129-ubuntu-2404-nvidia-580 (image family, resolved at launch)
- Runtime
- docker
- Commit
- uncommitted working tree (exact code bundles in the release)
- Config
- configs/qwen3-8b.yaml
- Engine
- vLLM 0.30.0 (sha256:8a69ffad015f…)
- Model · BF16
- Qwen/Qwen3-8B
- Model · FP8
- Qwen/Qwen3-8B
Hardware fingerprint
Not captured: these runs predate hardware fingerprinting. Recorded: GPU model, driver 580.173.02, CUDA 13.0.
Scenarios
Each scenario is measured at every listed number of simultaneous users, then the most users that still meet its latency target are searched for.
| Scenario | Prompt → answer | Prompts | Users measured | Latency target |
|---|---|---|---|---|
Short chat chat-128-128 | 128 → 128 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
RAG / long prompt, short answer prefill-1024-128 | 1,024 → 128 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Short prompt, long answer decode-128-1024 | 128 → 1,024 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Long prompt, long answer balanced-2048-2048 | 2,048 → 2,048 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Long context (8k) longctx-8192-512 | 8,192 → 512 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Cost: self-hosted vs API
Per 1M requests of each shape. Self-hosted = the cheapest GPU offer kept busy at the measured capacity (any precision). Today's GPU prices refresh every 6 h; API prices are fetched live. The experiment-day columns are frozen.
| Request shape | Cheapest self-hosted today | Today | Experiment day | Cheapest API now | Now | Experiment day | Saving now |
|---|---|---|---|---|---|---|---|
Short chat 128 in → 128 out | FP8 · vast (H100 SXM, marketplace median) | $5.16 | – | Fireworks | $51.20 | – | 90% |
RAG / long prompt, short answer 1,024 in → 128 out | FP8 · vast (H100 SXM, marketplace median) | $20.69 | – | Fireworks | $230 | – | 91% |
Short prompt, long answer 128 in → 1024 out | FP8 · vast (H100 SXM, marketplace median) | $45.24 | – | Fireworks | $230 | – | 80% |
Long prompt, long answer 2,048 in → 2048 out | FP8 · vast (H100 SXM, marketplace median) | $374 | – | Fireworks | $819 | – | 54% |
Long context (8k) 8,192 in → 512 out | FP8 · vast (H100 SXM, marketplace median) | $434 | – | Fireworks | $1,741 | – | 75% |
Measurements
How many users can one deployment serve?
Max concurrent users while p99 time to first token ≤ 2.00 s and median inter-token time ≤ 50 ms. Memory limit = requests that fit in the KV cache. Cost = GPU price per hour ÷ output tokens per hour at max users.
| Model | Hardware | Max users (SLO) | Memory limit | Inter-token, 1 user | p99 TTFT @ max | Peak tok/s | $ / 1M output tok | Avg accuracy |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B · BF16 | 1× H100-80GB | ≥ 512 | 1,472 | 6.8 ms | 1.89 s | 14,540 | $0.126 | 74.8% |
| Qwen3-8B · FP8 | 1× H100-80GB | ≥ 512 | 1,680 | 4.6 ms | 1.51 s | 16,542 | $0.110 | 74.6% |
Throughput vs. per-user speed
Each point is one concurrency level. Moving left adds users: total throughput rises while each user's token rate falls.
Accuracy
lm-evaluation-harness against the same vLLM server. Thinking mode off.
p99 time to first token
How long the slowest 1% of users wait for the first token.
Median inter-token time
Typical gap between streamed tokens (IDT).
Downloads
Everything captured, including per-repeat raw measurements, telemetry, logs and code. Release pipeline-2026-09-24.
| 20260924T122817Z_h100-80gbx1_qwen3-8b_bf16.tar.gz complete run directory 20260924T122817Z_h100-80gbx1_qwen3-8b_bf16: result.json, per-repeat vllm bench JSON, GPU telemetry, vLLM/lm-eval logs, accuracy results and per-question samples | 160 kB | sha256 534bc1198864… |
| 20260924T125045Z_h100-80gbx1_qwen3-8b_bf16.tar.gz complete run directory 20260924T125045Z_h100-80gbx1_qwen3-8b_bf16: result.json, per-repeat vllm bench JSON, GPU telemetry, vLLM/lm-eval logs, accuracy results and per-question samples | 2.4 MB | sha256 2793161c033b… |
| 20260924T222914Z_h100-80gbx1_qwen3-8b_bf16.tar.gz complete run directory 20260924T222914Z_h100-80gbx1_qwen3-8b_bf16: result.json, per-repeat vllm bench JSON, GPU telemetry, vLLM/lm-eval logs, accuracy results and per-question samples | 9.8 MB | sha256 0a930000fd27… |
| 20260924T225750Z_h100-80gbx1_qwen3-8b_fp8.tar.gz complete run directory 20260924T225750Z_h100-80gbx1_qwen3-8b_fp8: result.json, per-repeat vllm bench JSON, GPU telemetry, vLLM/lm-eval logs, accuracy results and per-question samples | 12.1 MB | sha256 03e10d23996f… |
| h100x1_qwen3-8b-20260924T124657Z.tgz exact code bundle that ran on the VM (MANIFEST.sha256 lists every file; 6 private terraform state/vars files removed) | 287 kB | sha256 fce805b4566f… |
| h100x1_qwen3-8b-20260924T222508Z.tgz exact code bundle that ran on the VM (MANIFEST.sha256 lists every file; 6 private terraform state/vars files removed) | 298 kB | sha256 c10d4d703e61… |
| h100x1_qwen3-8b-20260924T225310Z.tgz exact code bundle that ran on the VM (MANIFEST.sha256 lists every file; 6 private terraform state/vars files removed) | 314 kB | sha256 b60e76f0f6ff… |
| h100x1_qwen3-8b-20260924T225329Z.tgz exact code bundle that ran on the VM (MANIFEST.sha256 lists every file; 6 private terraform state/vars files removed) | 314 kB | sha256 d350083ca44d… |
| h100x1_qwen3-8b-20260924T225348Z.tgz exact code bundle that ran on the VM (MANIFEST.sha256 lists every file; 6 private terraform state/vars files removed) | 314 kB | sha256 f2298c008850… |
| smoke-20260924T122303Z.tgz exact code bundle that ran on the VM (MANIFEST.sha256 lists every file; 4 private terraform state/vars files removed) | 283 kB | sha256 f6695c49ef4c… |
| smoke-20260924T122401Z.tgz exact code bundle that ran on the VM (MANIFEST.sha256 lists every file; 5 private terraform state/vars files removed) | 283 kB | sha256 b0dc223182be… |
| gpubench-h100-1g-20260924T123340Z.log VM log | 40 kB | sha256 463f7f682266… |
| gpubench-h100-1g-20260924T224953Z.log VM log | 35 kB | sha256 426b60f6594e… |
| gpubench-h100-1g-20260925T065455Z.log VM log | 944 kB | sha256 6fff2f1627d2… |
| gpubench-h100-1g-bf16-crashed.log VM log | 957 kB | sha256 0130b897d9fa… |
Result files: 20260924T125045Z_h100-80gbx1_qwen3-8b_bf16 · 20260924T225750Z_h100-80gbx1_qwen3-8b_fp8