Benchmarks
LLM inference benchmarks
Latency, throughput, concurrent-user capacity and accuracy of open Hugging Face models, served with vLLM v0.30.0 on H100-80GB.
2 runs1 modelUpdated 2026-09-24
Pipeline validation data. These Qwen3-8B runs were used to build the benchmark pipeline and are not a published experiment. The first published experiment (Qwen3.8-27B on H100) is coming soon.
128 in → 128 out tokens per request
Max concurrent users within SLO
≥ 512users
TieQwen3-8B · FP8Qwen3-8B · BF16
All met the SLO at 512 users, the most tested. Qwen3-8B · FP8 has more headroom there: p99 TTFT 1.51 s vs 1.89 s.
Peak output throughput
16,542tok/s
Qwen3-8B · FP8
1.14× the next best
Cost per 1M output tokens
$0.110USD
Qwen3-8B · FP8
GPU price ($6.57/h) ÷ tokens generated while serving ≥ 512 users within SLO. 1.14× cheaper than the next best.
How many users can one deployment serve?
Max concurrent users while p99 time to first token ≤ 2.00 s and median inter-token time ≤ 50 ms. Memory limit = requests that fit in the KV cache. Cost = GPU price per hour ÷ output tokens per hour at max users.
| Model | Hardware | Max users (SLO) | Memory limit | Inter-token, 1 user | p99 TTFT @ max | Peak tok/s | $ / 1M output tok | Avg accuracy |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B · BF16 | 1× H100-80GB | ≥ 512 | 1,472 | 6.8 ms | 1.89 s | 14,540 | $0.126 | 74.8% |
| Qwen3-8B · FP8 | 1× H100-80GB | ≥ 512 | 1,680 | 4.6 ms | 1.51 s | 16,542 | $0.110 | 74.6% |
Throughput vs. per-user speed
Each point is one concurrency level. Moving left adds users: total throughput rises while each user's token rate falls.
Accuracy
lm-evaluation-harness against the same vLLM server. Thinking mode off.
p99 time to first token
How long the slowest 1% of users wait for the first token.
Median inter-token time
Typical gap between streamed tokens (IDT).