What does it really cost to serve an open model?
Measured on rented GPUs with vLLM: how many users one GPU serves within a latency target, turned into dollars with every provider's list price and compared with pay-per-token APIs for the same model.
Serving performance of Qwen3.8-27B with vLLM 0.30.0 on one H100 SXM: BF16 (base checkpoint) and FP8 (Qwen's official FP8 checkpoint), six request shapes from short chat to 100k-token tool-calling agent contexts, latency percentiles and the most concurrent users served within the latency target, plus two FP8 serving options on the 100k agent workloads (MTP speculative decoding, and prefix caching on 10-turn agent sessions). Performance only, plus a correctness suite for the 100k tool-calling scenario (long-context recall + a drift baseline); standard accuracy tests come later.
- Tool calling, 100k context100,000 → 512 tokens · first token ≤ 30.0 s
- Agent sessions, 100k (10 turns)100,000 → 512 tokens · first token ≤ 30.0 s
- Short chat128 → 128 tokens · first token ≤ 2.00 s
- RAG / long prompt, short answer1,024 → 128 tokens · first token ≤ 2.00 s
- Short prompt, long answer128 → 1,024 tokens · first token ≤ 2.00 s
- Long prompt, long answer2,048 → 2,048 tokens · first token ≤ 2.00 s
- Long context (8k)8,192 → 512 tokens · first token ≤ 2.00 s
Self-hosting figures are rented-GPU list prices at the most concurrent users that still get a first token within 2 s (p99) and at least 20 tokens/s each. They exclude engineering and operations time, storage and egress. APIs are list prices for the same model; some serve quantized weights. All inputs and the method are open: see Data and Methodology.