Qwen3.8-27B on 1x H100 SXM 80GB (GCP a3-highgpu-1g Spot)
Serving performance of Qwen3.8-27B with vLLM 0.30.0 on one H100 SXM: BF16 (base checkpoint) and FP8 (Qwen's official FP8 checkpoint), six request shapes from short chat to 100k-token tool-calling agent contexts, latency percentiles and the most concurrent users served within the latency target, plus two FP8 serving options on the 100k agent workloads (MTP speculative decoding, and prefix caching on 10-turn agent sessions). Performance only, plus a correctness suite for the 100k tool-calling scenario (long-context recall + a drift baseline); standard accuracy tests come later.
Exactly what ran
Reproductions must use this provider, machine, image and code. The agent refuses anything else.
- Provider
- GCP · spot
- Machine
- a3-highgpu-1g · us-central1-a
- Boot image
- common-cu129-ubuntu-2404-nvidia-580-v20260909
- Runtime
- docker
- Commit
- 5c24413982a6
- Engine
- vLLM 0.30.0 (pinned image)
- Model · BF16
- Qwen/Qwen3.8-27B
- Model · FP8
- Qwen/Qwen3.8-27B-FP8
- Model · FP8 + MTP
- Qwen/Qwen3.8-27B-FP8
- Model · FP8 + prefix cache
- Qwen/Qwen3.8-27B-FP8
- Model · FP8 + MTP + prefix cache
- Qwen/Qwen3.8-27B-FP8
Hardware fingerprint
Captured automatically when the experiment runs: GPU identity (model, PCI device id, power limit, VBIOS), driver/CUDA, CPU/RAM, measured memory bandwidth and matmul throughput, download speed and temperatures. It becomes the reference every reproduction is checked against.
Scenarios
Each scenario is measured at every listed number of simultaneous users, then the most users that still meet its latency target are searched for.
| Scenario | Prompt → answer | Prompts | Users measured | Latency target |
|---|---|---|---|---|
Tool calling, 100k context+ quality suite toolcall-100k-512 | 100,000 → 512 tokens | real agent traces (CC-BY-4.0) | 1, 2, 4, 8 × 3 | first token ≤ 30.0 s (p99), ≥ 20 tok/s per user |
Agent sessions, 100k (10 turns) agent-sessions-100k | 100,000 → 512 tokens | real agent traces (CC-BY-4.0) | 1, 2, 3, 4, 6, 8, 10, 12 × 2 | first token ≤ 30.0 s (p99), ≥ 20 tok/s per user |
Short chat chat-128-128 | 128 → 128 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
RAG / long prompt, short answer prefill-1024-128 | 1,024 → 128 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Short prompt, long answer decode-128-1024 | 128 → 1,024 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Long prompt, long answer balanced-2048-2048 | 2,048 → 2,048 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Long context (8k) longctx-8192-512 | 8,192 → 512 tokens | random tokens | 1, 4, 16, 32, 64, 128, 256, 512 × 3 | first token ≤ 2.00 s (p99), ≥ 20 tok/s per user |
Run order: 1. all other scenarios → 2. all other scenarios → 3. all other scenarios → 4. all other scenarios
toolcall-100k.jsonl: 200 real tool-calling agent contexts of exactly 100,000 tokens (Qwen3.8 tokenizer and chat template). Built from SWE-agent sessions (the agent fixes GitHub issues through shell and editor commands): commands became run_command tool calls, their outputs tool results; sessions from the same repository were chained and the middle trimmed to the exact length. sha256 c35ce2d6c4eb9c5b… · rebuild: gpubench prompts make-toolcall --model Qwen/Qwen3.8-27B --tokens 100000 --count 200 --shards 4 --seed 42
toolcall-100k-quality.jsonl: Correctness suite from the same agent contexts, rendered with thinking off. 100 recall questions ("What was the first line of the output of the command X?") about tool outputs 20-80% deep in the 100k context, each answer present exactly once, scored by exact match; and 100 drift prompts at the agent's next decision point (greedy, top-20 logprobs), the baseline an optimized build is compared with (gpubench quality compare). sha256 2cf0a857ef5f9f37… · rebuild: gpubench quality make --model Qwen/Qwen3.8-27B --tokens 100000 --count 100 --shards 4 --seed 42
agent-sessions-100k.jsonl: 24 multi-turn agent sessions from the same 100k-token contexts, 10 turns each. Turn j is the conversation up to one of its last 10 tool results, so each turn resends the previous turn's prompt plus the agent's next call and its tool output (median 684 new tokens) and sessions grow from ~93k to exactly 100k tokens. Every turn is an exact text prefix of the next, which is what a prefix cache can reuse. sha256 ed29e1657c610246… · rebuild: gpubench prompts make-sessions --model Qwen/Qwen3.8-27B --tokens 100000 --count 24 --turns 10 --shards 4 --seed 42
Measurements
Not measured yet. Results, per-scenario latency percentiles, users served and the cost comparison appear here as soon as the runs are merged into silybench-data.
Downloads
Everything captured, including per-repeat raw measurements, telemetry, logs and code. Release exp-2026-10-qwen3.8-27b-h100.
| toolcall-100k.jsonl prompt dataset | 74.2 MB | sha256 c35ce2d6c4eb… |
| toolcall-100k-quality.jsonl prompt dataset | 74.4 MB | sha256 2cf0a857ef5f… |
| agent-sessions-100k.jsonl prompt dataset | 85.3 MB | sha256 ed29e1657c61… |
Reproduce this experiment
The agent checks out the exact code, refuses other providers, verifies the hardware fingerprint before measuring, and shows the cost (from this experiment's measured durations) before creating anything.
Clone https://github.com/tgarg01/silybench-bench and follow its AGENTS.md to reproduce experiment 2026-10-qwen3.8-27b-h100. Show me the cost before creating anything.