Every result comes from a pinned vLLM release on a rented GPU (the Docker image where the host has Docker, otherwise the same version installed from pip), started fresh for each benchmark campaign. Configs, code, raw outputs and prices are public and versioned, so any number on this site can be reproduced.
Serving
- One vLLM server per model × precision (BF16, FP8). Model revisions are pinned to a commit.
- Qwen3 runs with thinking mode off (server default chat-template kwarg).
- Parallelism (tensor / pipeline / data / expert) is recorded with every run.
Performance
- Load generator:
vllm bench serve, closed loop at a fixed number of concurrent users (1 → 512), random prompts with fixed input/output lengths,--ignore-eosso every request does the same work. - Each point: warmup, then 3 repeats; we report the median of each metric across repeats.
- Latency is reported at the 95th and 99th percentiles only: time to first token (TTFT), time per output token (TPOT), inter-token latency and end-to-end latency.
- Median inter-token time (IDT): the typical gap between streamed tokens, and its inverse, tokens/s per user.
- GPU power and memory are sampled every 500 ms with nvidia-smi → tokens per joule.
- Cost per 1M output tokens on this page uses the hourly price the run was measured at; the cost pages reprice every run with current list prices (below).
How many users can one deployment serve?
Max concurrent users (SLO) is the highest concurrency at which both hold: p99 TTFT ≤ 2 s and median inter-token time ≤ 50 ms (≥ 20 tokens/s per user, faster than people read). After the coarse sweep we bisect between the last passing and first failing level to find the exact number.
Memory limit is vLLM's KV-cache capacity divided by the tokens one request needs (input + output): how many requests fit on the GPU at once, whatever the latency.
Accuracy
lm-evaluation-harness (chat completions) against the same vLLM server: MMLU-Pro, GPQA Diamond (zero-shot CoT), GSM8K, MATH-500, IFEval and ARC-Challenge. Greedy decoding. Running accuracy per precision shows what FP8 costs in quality, not just what it gains in speed.
Self-hosting vs API cost
- Throughput at capacity. For each request shape we take the measured requests/s and tokens/s at the SLO capacity above. That is the most a deployment can serve while every user still gets a fast first token and ≥ 20 tokens/s.
- Repricing. Performance depends on the GPU, not on who rents it, so each benchmark is priced with every provider renting that GPU type (list prices with source and date in
prices/gpu_hourly.yaml). Cost per request = hourly price ÷ requests per hour at capacity. - APIs. The same model's list prices per provider (OpenRouter's public endpoint data plus hand-checked pricing pages): input tokens × input price + output tokens × output price.
- Break-even is the daily volume at which an always-on GPU (730 h/month) costs the same as the cheapest API. Below it, pay per token. Above it, self-host.
- Calculator sizing buys enough machines for your peak load (requests/day × peak factor), using the closest measured request shape and scaling capacity by your answer length (decode-bound serving scales with output tokens).
- Not included: engineering and on-call time, storage, egress, and idle time beyond your peak factor. FP8 accuracy is measured and shown so its savings aren't free-lunch claims.