Qwen3-8B · FP8
1× H100-80GB · 2026-09-24
Configuration
- Model
- Qwen/Qwen3-8B @ b968826d9c46
- Precision
- FP8
- Max model length
- 16,384
- Thinking mode
- Off
- Machine
- gcp · a3-highgpu-1g (spot)
- Engine
- vLLM 0.30.0
- Driver / CUDA
- 580.173.02 / 13.0
- KV cache
- 430,272 tokens
- Parallelism
- TP1 PP1 DP1 EP1
- Price
- $6.57/h
- Git commit
- uncommitted-
- Run date
- 2026-09-24
vllm serve Qwen/Qwen3-8B --revision main --max-model-len 16384 --gpu-memory-utilization 0.9 --port 8000 --seed 42 --quantization fp8 --tensor-parallel-size 1 --pipeline-parallel-size 1 --reasoning-parser qwen3 --default-chat-template-kwargs {"enable_thinking": false}balanced-2048-2048
Max concurrent users: 52 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 105
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 50.3 ms | 50.7 ms | 4.8 ms | 4.9 ms | 5 ms | 4.8 ms | 9.88 s | 9.90 s | 208 | 208.5 | 0.52 | $8.79 | ✓ pass |
| 4 | 161.4 ms | 162.5 ms | 5.3 ms | 5.5 ms | 5.6 ms | 5.4 ms | 11.1 s | 11.1 s | 716 | 188 | 1.65 | $2.55 | ✓ pass |
| 16 | 554.6 ms | 584.9 ms | 7.2 ms | 8 ms | 8.1 ms | 7.5 ms | 15.4 s | 15.5 s | 2,136 | 138.6 | 4.32 | $0.854 | ✓ pass |
| 32 | 996.6 ms | 1.17 s | 9.8 ms | 11.3 ms | 11.7 ms | 10.3 ms | 21.7 s | 21.9 s | 3,086 | 101.9 | 5.39 | $0.591 | ✓ pass |
| 48 | 1.43 s | 1.77 s | 12.3 ms | 14.6 ms | 15.3 ms | 13 ms | 27.6 s | 27.9 s | 3,667 | 81.6 | 5.84 | $0.498 | ✓ pass |
| 52 | 1.52 s | 1.93 s | 13.3 ms | 16 ms | 16.5 ms | 14.2 ms | 30.1 s | 30.5 s | 3,653 | 75 | 5.83 | $0.500 | ✓ pass |
| 56 | 1.66 s | 2.08 s | 13.5 ms | 16.2 ms | 16.9 ms | 14.5 ms | 30.8 s | 31.3 s | 3,855 | 73.9 | 5.93 | $0.473 | ✕ fail |
| 64 | 1.84 s | 2.36 s | 14.8 ms | 17.7 ms | 18.4 ms | 15.8 ms | 33.8 s | 34.3 s | 4,033 | 67.5 | 6.09 | $0.453 | ✕ fail |
| 128 | 4.23 s | 10.5 s | 24.6 ms | 27.3 ms | 153 ms | 34.8 ms | 68.2 s | 73.0 s | 4,428 | 40.6 | 6.74 | $0.412 | ✕ fail |
| 256 | 67.2 s | 71.1 s | 28.3 ms | 30 ms | 159 ms | 60.2 ms | 131.7 s | 135.6 s | 4,310 | 35.4 | 6.5 | $0.423 | ✕ fail |
| 512 | 196.0 s | 199.9 s | 28.3 ms | 29.9 ms | 159 ms | 60.2 ms | 259.9 s | 264.0 s | 4,344 | 35.4 | 6.53 | $0.420 | ✕ fail |
chat-128-128
Max concurrent users: ≥ 512 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 1,680
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 14.5 ms | 15 ms | 4.6 ms | 4.7 ms | 4.8 ms | 4.6 ms | 601.1 ms | 602.1 ms | 213 | 216.5 | 0.55 | $8.56 | ✓ pass |
| 4 | 24.6 ms | 25.7 ms | 4.7 ms | 4.8 ms | 5 ms | 4.7 ms | 622.6 ms | 624.9 ms | 797 | 212.1 | 2.09 | $2.29 | ✓ pass |
| 16 | 56.1 ms | 57 ms | 4.9 ms | 5.2 ms | 6.1 ms | 5.1 ms | 676.1 ms | 680.6 ms | 3,088 | 205.6 | 7.78 | $0.591 | ✓ pass |
| 32 | 85.1 ms | 91.7 ms | 5.3 ms | 5.7 ms | 10.1 ms | 5.8 ms | 779 ms | 785.1 ms | 5,413 | 188.5 | 11.33 | $0.337 | ✓ pass |
| 64 | 155.3 ms | 161.6 ms | 5.9 ms | 6.8 ms | 14.6 ms | 6.9 ms | 946.3 ms | 953.5 ms | 9,170 | 169 | 18.33 | $0.199 | ✓ pass |
| 128 | 330.6 ms | 333.9 ms | 7.5 ms | 9.4 ms | 19.1 ms | 9.3 ms | 1.31 s | 1.33 s | 13,377 | 132.5 | 23.99 | $0.136 | ✓ pass |
| 256 | 731.8 ms | 738 ms | 10.9 ms | 15.7 ms | 93.1 ms | 14.7 ms | 2.17 s | 2.18 s | 15,704 | 92 | 24.7 | $0.116 | ✓ pass |
| 512 | 1.34 s | 1.51 s | 19.7 ms | 41.1 ms | 143.3 ms | 27.8 ms | 4.14 s | 4.16 s | 16,542 | 50.8 | 25.55 | $0.110 | ✓ pass |
decode-128-1024
Max concurrent users: 472 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 373
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 14.4 ms | 15 ms | 4.7 ms | 4.7 ms | 4.9 ms | 4.7 ms | 4.79 s | 4.80 s | 214 | 214.5 | 0.54 | $8.52 | ✓ pass |
| 4 | 25.3 ms | 26 ms | 4.8 ms | 5 ms | 5.1 ms | 4.8 ms | 4.96 s | 4.96 s | 797 | 207.7 | 1.99 | $2.29 | ✓ pass |
| 16 | 57.6 ms | 57.7 ms | 5.3 ms | 5.6 ms | 5.8 ms | 5.3 ms | 5.43 s | 5.44 s | 3,027 | 190.5 | 6.9 | $0.603 | ✓ pass |
| 32 | 82 ms | 82.4 ms | 6 ms | 6.8 ms | 7 ms | 6.1 ms | 6.27 s | 6.27 s | 5,249 | 165.8 | 10.15 | $0.348 | ✓ pass |
| 64 | 201.9 ms | 212.8 ms | 7.3 ms | 8.7 ms | 9 ms | 7.4 ms | 7.65 s | 7.65 s | 8,621 | 137.7 | 14.88 | $0.212 | ✓ pass |
| 128 | 383.9 ms | 385.7 ms | 10.2 ms | 12.9 ms | 13.7 ms | 10.4 ms | 10.8 s | 10.8 s | 12,176 | 98.4 | 19.5 | $0.150 | ✓ pass |
| 256 | 694 ms | 738.3 ms | 16.1 ms | 21.4 ms | 23.4 ms | 16.6 ms | 17.3 s | 17.3 s | 15,264 | 62 | 22.47 | $0.120 | ✓ pass |
| 384 | 964.2 ms | 1.10 s | 24.8 ms | 33 ms | 35.3 ms | 26.1 ms | 26.8 s | 26.9 s | 14,803 | 40.2 | 21.57 | $0.123 | ✓ pass |
| 448 | 1.56 s | 1.58 s | 27.1 ms | 34.4 ms | 59.5 ms | 33 ms | 32.9 s | 34.1 s | 14,784 | 36.9 | 21.74 | $0.123 | ✓ pass |
| 464 | 1.72 s | 1.73 s | 27.4 ms | 34.6 ms | 61.2 ms | 34.1 ms | 34.2 s | 35.4 s | 15,006 | 36.5 | 22.11 | $0.122 | ✓ pass |
| 472 | 1.87 s | 1.89 s | 27.4 ms | 34.7 ms | 63 ms | 34.8 ms | 34.7 s | 36.0 s | 15,103 | 36.5 | 22.24 | $0.121 | ✓ pass |
| 476 | 1.88 s | 2.02 s | 27.6 ms | 34.8 ms | 63.1 ms | 35.2 ms | 35.2 s | 36.3 s | 15,131 | 36.2 | 22.29 | $0.121 | ✕ fail |
| 480 | 2.02 s | 2.03 s | 27.8 ms | 34.8 ms | 64.4 ms | 35.6 ms | 35.6 s | 36.8 s | 15,122 | 36 | 22.27 | $0.121 | ✕ fail |
| 512 | 2.48 s | 5.54 s | 28.3 ms | 36.2 ms | 69.4 ms | 38.2 ms | 38.4 s | 39.7 s | 15,389 | 35.3 | 22.62 | $0.119 | ✕ fail |
longctx-8192-512
Max concurrent users: 10 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 49
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 205.1 ms | 206.1 ms | 5.1 ms | 5.2 ms | 5.4 ms | 5.1 ms | 2.81 s | 2.81 s | 183 | 195.9 | 0.41 | $10.00 | ✓ pass |
| 4 | 684.8 ms | 691.4 ms | 6.4 ms | 6.5 ms | 6.7 ms | 7.4 ms | 3.97 s | 3.98 s | 505 | 155.6 | 1.01 | $3.61 | ✓ pass |
| 10 | 1.47 s | 1.71 s | 9.5 ms | 9.7 ms | 19.9 ms | 12.3 ms | 6.97 s | 7.30 s | 787 | 105.2 | 1.44 | $2.32 | ✓ pass |
| 13 | 1.83 s | 2.21 s | 10.2 ms | 10.7 ms | 167.3 ms | 14 ms | 8.18 s | 8.62 s | 903 | 97.7 | 1.56 | $2.02 | ✕ fail |
| 16 | 2.23 s | 2.74 s | 11.4 ms | 12.2 ms | 170.3 ms | 16.1 ms | 9.70 s | 10.2 s | 966 | 87.5 | 1.6 | $1.89 | ✕ fail |
| 32 | 4.33 s | 5.46 s | 18.2 ms | 28.4 ms | 180.7 ms | 27.9 ms | 17.9 s | 19.1 s | 1,124 | 54.8 | 1.75 | $1.62 | ✕ fail |
| 64 | 16.3 s | 24.1 s | 27.2 ms | 184.9 ms | 187.4 ms | 56.5 ms | 37.8 s | 45.4 s | 1,141 | 36.8 | 1.75 | $1.60 | ✕ fail |
| 128 | 41.2 s | 48.2 s | 27.2 ms | 185.6 ms | 188.1 ms | 60.1 ms | 62.8 s | 69.7 s | 1,154 | 36.7 | 1.76 | $1.58 | ✕ fail |
| 256 | 98.0 s | 103.7 s | 27.2 ms | 185.6 ms | 188.2 ms | 62.2 ms | 122.5 s | 125.4 s | 1,154 | 36.7 | 1.76 | $1.58 | ✕ fail |
| 512 | 215.8 s | 215.9 s | 27.2 ms | 185.6 ms | 188.1 ms | 63.4 ms | 237.3 s | 237.5 s | 1,157 | 36.7 | 1.76 | $1.58 | ✕ fail |
prefill-1024-128
Max concurrent users: 108 within SLO (p99 TTFT ≤ 2.00 s, median inter-token ≤ 50 ms) · memory limit 373
| Users | TTFT p95 | TTFT p99 | Inter-token median | Inter-token p95 | Inter-token p99 | TPOT p99 | E2E p95 | E2E p99 | Output tok/s | tok/s/user | Tokens/J | $/1M tok | SLO |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 29.3 ms | 31.1 ms | 4.7 ms | 4.8 ms | 4.9 ms | 4.7 ms | 625.8 ms | 626.9 ms | 205 | 212.9 | 0.52 | $8.90 | ✓ pass |
| 4 | 31.8 ms | 32.2 ms | 4.9 ms | 5.1 ms | 5.3 ms | 4.9 ms | 654 ms | 655.5 ms | 760 | 203.6 | 1.95 | $2.40 | ✓ pass |
| 16 | 206.6 ms | 219.5 ms | 5.6 ms | 5.9 ms | 10.3 ms | 7.5 ms | 1.01 s | 1.01 s | 2,393 | 178.1 | 5.22 | $0.763 | ✓ pass |
| 32 | 260.1 ms | 287.2 ms | 6.7 ms | 9.3 ms | 60.8 ms | 10.7 ms | 1.44 s | 1.46 s | 3,444 | 148.4 | 6.12 | $0.530 | ✓ pass |
| 64 | 978.7 ms | 1.17 s | 8.6 ms | 19.5 ms | 142 ms | 16.9 ms | 2.57 s | 2.81 s | 3,701 | 115.7 | 5.83 | $0.493 | ✓ pass |
| 96 | 1.40 s | 1.70 s | 11.1 ms | 135.9 ms | 144.3 ms | 23.3 ms | 3.85 s | 4.16 s | 4,025 | 90.3 | 6.18 | $0.453 | ✓ pass |
| 104 | 1.57 s | 1.85 s | 11.6 ms | 138.8 ms | 146.6 ms | 24.9 ms | 4.20 s | 4.48 s | 4,080 | 86.2 | 6.29 | $0.447 | ✓ pass |
| 108 | 1.59 s | 1.90 s | 11.8 ms | 139.4 ms | 146.5 ms | 25.5 ms | 4.30 s | 4.58 s | 4,129 | 85 | 6.36 | $0.442 | ✓ pass |
| 112 | 1.59 s | 2.01 s | 12 ms | 140.1 ms | 148.1 ms | 26.4 ms | 4.43 s | 4.85 s | 4,153 | 83.3 | 6.38 | $0.439 | ✕ fail |
| 128 | 1.85 s | 2.29 s | 12.9 ms | 142.7 ms | 148.6 ms | 29.3 ms | 5.09 s | 5.51 s | 4,264 | 77.3 | 6.42 | $0.428 | ✕ fail |
| 256 | 3.73 s | 4.68 s | 21.6 ms | 154.1 ms | 157 ms | 54.6 ms | 10.2 s | 11.1 s | 4,561 | 46.2 | 6.67 | $0.400 | ✕ fail |
| 512 | 7.72 s | 13.0 s | 35.1 ms | 162.8 ms | 167.1 ms | 105 ms | 19.0 s | 23.7 s | 4,418 | 28.5 | 6.43 | $0.413 | ✕ fail |
Accuracy
| Task | Metric | Score | ± stderr |
|---|---|---|---|
| MMLU-Pro | exact_match | 62.6% | 0.4% |
| GPQA Diamond | exact_match | 45.5% | 3.5% |
| GSM8K | exact_match | 86.4% | 0.9% |
| MATH-500 | math_verify | 82.2% | 1.7% |
| IFEval | prompt_level_strict_acc | 83.5% | 1.6% |
| ARC Challenge | exact_match_letter | 87.6% | 1.0% |