Qwen3-8B · BF16
1× H100-80GB · 2026-09-24
Configuration
- Model
- Qwen/Qwen3-8B @ b968826d9c46
- Precision
- BF16
- Max model length
- 16,384
- Thinking mode
- Off
- Machine
- gcp · a3-highgpu-1g (spot)
- Engine
- vLLM 0.30.0
- Driver / CUDA
- 580.173.02 / 13.0
- KV cache
- 377,024 tokens
- Parallelism
- TP1 PP1 DP1 EP1
- Price
- $6.57/h
- Git commit
- uncommitted-
- Run date
- 2026-09-24
vllm serve Qwen/Qwen3-8B --revision main --max-model-len 16384 --gpu-memory-utilization 0.9 --port 8000 --seed 42 --dtype bfloat16 --tensor-parallel-size 1 --pipeline-parallel-size 1 --reasoning-parser qwen3 --default-chat-template-kwargs {"enable_thinking": false}balanced-2048-2048
Max concurrent users: 40 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 92
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 61.3 ms | 62.2 ms | 7 ms | 7.1 ms | 7.2 ms | 7 ms | 14.4 s | 14.4 s | 142 | 142.9 | 0.33 | $12.82 | ✓ pass |
| 4 | 203.7 ms | 205 ms | 7.5 ms | 7.7 ms | 7.8 ms | 7.6 ms | 15.6 s | 15.6 s | 506 | 132.9 | 1.1 | $3.60 | ✓ pass |
| 16 | 787.3 ms | 788.6 ms | 9.5 ms | 10.2 ms | 10.5 ms | 9.8 ms | 20.2 s | 20.3 s | 1,625 | 105.7 | 3.22 | $1.12 | ✓ pass |
| 32 | 1.39 s | 1.59 s | 11.9 ms | 13.7 ms | 14.2 ms | 12.6 ms | 26.6 s | 26.8 s | 2,529 | 84.3 | 4.41 | $0.722 | ✓ pass |
| 40 | 1.60 s | 1.99 s | 13.2 ms | 15.3 ms | 15.8 ms | 14.2 ms | 30.0 s | 30.4 s | 2,815 | 75.5 | 4.79 | $0.648 | ✓ pass |
| 44 | 1.80 s | 2.21 s | 13.9 ms | 16 ms | 16.5 ms | 14.9 ms | 31.6 s | 32.0 s | 2,950 | 71.9 | 4.89 | $0.619 | ✕ fail |
| 48 | 1.91 s | 2.36 s | 14.6 ms | 16.8 ms | 17.4 ms | 15.6 ms | 33.3 s | 33.7 s | 3,067 | 68.6 | 5.08 | $0.595 | ✕ fail |
| 64 | 2.48 s | 3.09 s | 17.1 ms | 19.9 ms | 20.5 ms | 18.4 ms | 39.6 s | 40.2 s | 3,463 | 58.6 | 5.28 | $0.527 | ✕ fail |
| 128 | 12.4 s | 28.6 s | 25.5 ms | 26.9 ms | 202.8 ms | 43.4 ms | 85.8 s | 91.0 s | 3,917 | 39.2 | 5.69 | $0.466 | ✕ fail |
| 256 | 71.7 s | 106.4 s | 26.3 ms | 27.6 ms | 207.9 ms | 58.2 ms | 152.8 s | 169.1 s | 3,881 | 38 | 5.64 | $0.470 | ✕ fail |
| 512 | 201.8 s | 207.1 s | 26.3 ms | 27.7 ms | 207.7 ms | 58.1 ms | 296.7 s | 302.7 s | 3,901 | 38 | 5.66 | $0.468 | ✕ fail |
chat-128-128
Max concurrent users: ≥ 512 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 1,472
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 14.2 ms | 14.5 ms | 6.8 ms | 6.9 ms | 7 ms | 6.8 ms | 882.5 ms | 882.9 ms | 145 | 146.5 | 0.34 | $12.56 | ✓ pass |
| 4 | 24.2 ms | 24.7 ms | 6.9 ms | 7 ms | 7.3 ms | 6.9 ms | 901.2 ms | 903.8 ms | 548 | 144.5 | 1.3 | $3.33 | ✓ pass |
| 16 | 57.5 ms | 64.7 ms | 7.1 ms | 7.3 ms | 8.1 ms | 7.4 ms | 966.7 ms | 971.5 ms | 2,156 | 140.1 | 4.94 | $0.846 | ✓ pass |
| 32 | 80.7 ms | 84.3 ms | 7.4 ms | 7.7 ms | 12.4 ms | 7.8 ms | 1.04 s | 1.05 s | 4,030 | 135.2 | 8.2 | $0.453 | ✓ pass |
| 64 | 181.9 ms | 189 ms | 8.1 ms | 8.7 ms | 14.6 ms | 9.2 ms | 1.24 s | 1.24 s | 6,958 | 122.7 | 12.68 | $0.262 | ✓ pass |
| 128 | 329 ms | 358.9 ms | 9.3 ms | 11.1 ms | 20.6 ms | 11.6 ms | 1.60 s | 1.61 s | 11,194 | 107.1 | 17.77 | $0.163 | ✓ pass |
| 256 | 828.5 ms | 916.9 ms | 12.8 ms | 16.3 ms | 144.8 ms | 17.9 ms | 2.59 s | 2.60 s | 13,093 | 77.9 | 19.53 | $0.139 | ✓ pass |
| 512 | 1.63 s | 1.89 s | 21.6 ms | 44.8 ms | 194.2 ms | 32.4 ms | 4.71 s | 4.74 s | 14,540 | 46.3 | 21.42 | $0.126 | ✓ pass |
decode-128-1024
Max concurrent users: 396 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 327
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 14.1 ms | 14.6 ms | 6.9 ms | 6.9 ms | 7 ms | 6.9 ms | 7.04 s | 7.05 s | 146 | 145.8 | 0.34 | $12.53 | ✓ pass |
| 4 | 24.8 ms | 25.1 ms | 7 ms | 7.2 ms | 7.2 ms | 7 ms | 7.21 s | 7.21 s | 548 | 142.7 | 1.29 | $3.33 | ✓ pass |
| 16 | 49.2 ms | 49.5 ms | 7.5 ms | 7.9 ms | 8 ms | 7.5 ms | 7.76 s | 7.76 s | 2,117 | 132.8 | 4.55 | $0.862 | ✓ pass |
| 32 | 86 ms | 88.6 ms | 8.1 ms | 8.9 ms | 9.1 ms | 8.2 ms | 8.41 s | 8.41 s | 3,910 | 123.3 | 7.43 | $0.467 | ✓ pass |
| 64 | 249.2 ms | 251.1 ms | 9.5 ms | 11 ms | 11.4 ms | 9.6 ms | 9.99 s | 9.99 s | 6,597 | 105.3 | 10.74 | $0.277 | ✓ pass |
| 128 | 447 ms | 463.9 ms | 12 ms | 14.9 ms | 15.9 ms | 12.3 ms | 12.8 s | 12.8 s | 10,301 | 83.3 | 14.94 | $0.177 | ✓ pass |
| 256 | 848.8 ms | 929.8 ms | 18.1 ms | 23.6 ms | 25.3 ms | 18.7 ms | 19.5 s | 19.5 s | 13,567 | 55.3 | 19.67 | $0.135 | ✓ pass |
| 384 | 1.74 s | 1.75 s | 24.8 ms | 31.1 ms | 32.5 ms | 30 ms | 30.0 s | 31.4 s | 13,591 | 40.4 | 19.96 | $0.134 | ✓ pass |
| 392 | 1.76 s | 1.76 s | 25.7 ms | 31.7 ms | 49.1 ms | 31.7 ms | 31.7 s | 32.9 s | 13,319 | 39 | 19.58 | $0.137 | ✓ pass |
| 396 | 1.95 s | 1.97 s | 25.8 ms | 31.8 ms | 48.2 ms | 32 ms | 32.1 s | 33.5 s | 13,322 | 38.8 | 19.55 | $0.137 | ✓ pass |
| 400 | 1.96 s | 2.13 s | 26 ms | 31.8 ms | 49.9 ms | 32.6 ms | 32.6 s | 33.9 s | 13,323 | 38.4 | 19.57 | $0.137 | ✕ fail |
| 416 | 2.33 s | 2.36 s | 27.5 ms | 32.9 ms | 52.8 ms | 35.2 ms | 35.3 s | 36.8 s | 13,062 | 36.4 | 19.16 | $0.140 | ✕ fail |
| 448 | 2.93 s | 5.43 s | 28.8 ms | 33.9 ms | 58.9 ms | 38.7 ms | 38.6 s | 40.6 s | 13,230 | 34.7 | 19.31 | $0.138 | ✕ fail |
| 512 | 4.33 s | 12.5 s | 30.4 ms | 38 ms | 71.1 ms | 46.2 ms | 46.2 s | 47.9 s | 13,523 | 32.9 | 19.67 | $0.135 | ✕ fail |
longctx-8192-512
Max concurrent users: 7 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 43
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 253.9 ms | 254.6 ms | 7.3 ms | 7.4 ms | 7.5 ms | 7.3 ms | 3.98 s | 3.98 s | 129 | 137 | 0.29 | $14.16 | ✓ pass |
| 4 | 893.4 ms | 898.1 ms | 8.6 ms | 8.7 ms | 8.9 ms | 9.9 ms | 5.30 s | 5.47 s | 378 | 116.1 | 0.76 | $4.83 | ✓ pass |
| 7 | 1.35 s | 1.57 s | 10.2 ms | 10.5 ms | 10.9 ms | 12.6 ms | 6.96 s | 7.13 s | 502 | 98.2 | 0.95 | $3.64 | ✓ pass |
| 10 | 1.91 s | 2.24 s | 11.8 ms | 12.4 ms | 25.3 ms | 15.5 ms | 8.79 s | 9.23 s | 626 | 84.9 | 1.12 | $2.92 | ✕ fail |
| 16 | 2.92 s | 3.59 s | 13.8 ms | 14.7 ms | 223.5 ms | 19.8 ms | 12.2 s | 12.9 s | 784 | 72.7 | 1.3 | $2.33 | ✕ fail |
| 32 | 5.62 s | 7.09 s | 20.3 ms | 39.6 ms | 231.2 ms | 33 ms | 21.6 s | 23.1 s | 951 | 49.3 | 1.46 | $1.92 | ✕ fail |
| 64 | 23.5 s | 26.5 s | 25.3 ms | 233 ms | 238.7 ms | 57.7 ms | 45.1 s | 48.2 s | 982 | 39.6 | 1.48 | $1.86 | ✕ fail |
| 128 | 47.0 s | 53.0 s | 25.3 ms | 233 ms | 238.5 ms | 57.9 ms | 68.6 s | 74.6 s | 994 | 39.5 | 1.49 | $1.84 | ✕ fail |
| 256 | 112.0 s | 118.3 s | 25.3 ms | 233.5 ms | 238.6 ms | 59.4 ms | 133.7 s | 139.9 s | 994 | 39.5 | 1.49 | $1.84 | ✕ fail |
| 512 | 245.5 s | 246.9 s | 25.3 ms | 233.4 ms | 238.8 ms | 59.4 ms | 267.1 s | 270.1 s | 997 | 39.5 | 1.49 | $1.83 | ✕ fail |
prefill-1024-128
Max concurrent users: 80 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 327
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 34.3 ms | 35.1 ms | 6.9 ms | 7 ms | 7.1 ms | 6.9 ms | 909.2 ms | 909.8 ms | 141 | 145.2 | 0.32 | $12.94 | ✓ pass |
| 4 | 32.2 ms | 33 ms | 7.1 ms | 7.2 ms | 7.5 ms | 7.1 ms | 933.9 ms | 934.6 ms | 532 | 140.5 | 1.24 | $3.43 | ✓ pass |
| 16 | 236.1 ms | 261 ms | 7.9 ms | 8.1 ms | 19.7 ms | 10.2 ms | 1.36 s | 1.36 s | 1,757 | 126.6 | 3.55 | $1.04 | ✓ pass |
| 32 | 265.8 ms | 268.1 ms | 8.8 ms | 9.5 ms | 74.2 ms | 13.9 ms | 1.84 s | 1.84 s | 2,697 | 113.1 | 4.77 | $0.677 | ✓ pass |
| 64 | 1.27 s | 1.55 s | 10.9 ms | 24.4 ms | 193.7 ms | 22 ms | 3.37 s | 3.76 s | 2,847 | 91.6 | 4.31 | $0.641 | ✓ pass |
| 80 | 1.59 s | 1.95 s | 12 ms | 97.6 ms | 198.1 ms | 26 ms | 4.25 s | 4.63 s | 3,018 | 83.6 | 4.53 | $0.605 | ✓ pass |
| 84 | 1.65 s | 2.04 s | 12.5 ms | 123.6 ms | 197.5 ms | 27.1 ms | 4.41 s | 4.79 s | 3,029 | 80 | 4.56 | $0.603 | ✕ fail |
| 88 | 1.78 s | 2.17 s | 12.6 ms | 172.6 ms | 197.9 ms | 28 ms | 4.71 s | 5.11 s | 3,087 | 79.5 | 4.6 | $0.591 | ✕ fail |
| 96 | 1.84 s | 2.33 s | 13.1 ms | 191.6 ms | 199.1 ms | 29.9 ms | 5.01 s | 5.56 s | 3,141 | 76.5 | 4.62 | $0.581 | ✕ fail |
| 128 | 2.50 s | 3.08 s | 14.9 ms | 195.4 ms | 200 ms | 37.6 ms | 6.51 s | 7.10 s | 3,349 | 67.2 | 4.89 | $0.545 | ✕ fail |
| 256 | 5.11 s | 6.33 s | 23.7 ms | 205.3 ms | 209 ms | 68.7 ms | 13.3 s | 14.5 s | 3,618 | 42.2 | 5.24 | $0.504 | ✕ fail |
| 512 | 13.5 s | 16.2 s | 31.4 ms | 211.5 ms | 214.8 ms | 113 ms | 25.0 s | 27.7 s | 3,518 | 31.9 | 5.07 | $0.519 | ✕ fail |
Accuracy
| Task | Metric | Score | ± stderr |
|---|---|---|---|
| MMLU-Pro | exact_match | 63.2% | 0.4% |
| GPQA Diamond | exact_match | 49.5% | 3.6% |
| GSM8K | exact_match | 83.9% | 1.0% |
| MATH-500 | math_verify | 80.8% | 1.8% |
| IFEval | prompt_level_strict_acc | 83.4% | 1.6% |
| ARC Challenge | exact_match_letter | 88.0% | 1.0% |