Every experiment, exactly reproducible
An experiment is one model on one machine at one provider, run from a tagged commit with a recorded hardware fingerprint. All raw data is downloadable, and anyone can re-run it with one prompt to Claude.
Serving performance of Qwen3.8-27B with vLLM 0.30.0 on one H100 SXM: BF16 (base checkpoint) and FP8 (Qwen's official FP8 checkpoint), six request shapes from short chat to 100k-token tool-calling agent contexts, latency percentiles and the most concurrent users served within the latency target, plus two FP8 serving options on the 100k agent workloads (MTP speculative decoding, and prefix caching on 10-turn agent sessions). Performance only, plus a correctness suite for the 100k tool-calling scenario (long-context recall + a drift baseline); standard accuracy tests come later.
The runs used to build and debug the benchmark pipeline: BF16 performance, an accuracy re-run, and a full FP8 campaign, plus a smoke test. The code ran from uncommitted working trees; the exact code bundles (minus private Terraform state) are in the release.