silybench
Cost
Self-host vs APICalculatorManaged hosting
Benchmarks
ExperimentsPerformanceRuns
Open data
DataReproduceMethodology
Theme
Experiments

Every experiment, exactly reproducible

An experiment is one model on one machine at one provider, run from a tagged commit with a recorded hardware fingerprint. All raw data is downloadable, and anyone can re-run it with one prompt to Claude.

planned2026-10-qwen3.8-27b-h100
Qwen3.8-27B on 1x H100 SXM 80GB (GCP a3-highgpu-1g Spot)

Serving performance of Qwen3.8-27B with vLLM 0.30.0 on one H100 SXM: BF16 (base checkpoint) and FP8 (Qwen's official FP8 checkpoint), six request shapes from short chat to 100k-token tool-calling agent contexts, latency percentiles and the most concurrent users served within the latency target, plus two FP8 serving options on the 100k agent workloads (MTP speculative decoding, and prefix caching on 10-turn agent sessions). Performance only, plus a correctness suite for the 100k tool-calling scenario (long-context recall + a drift baseline); standard accuracy tests come later.

GCP a3-highgpu-1g7 scenariosBF16 + FP8not measured yet
pipeline2026-09-24-qwen3-8b-pipeline
Qwen3-8B on 1x H100 SXM (GCP), pipeline validation

The runs used to build and debug the benchmark pipeline: BF16 performance, an accuracy re-run, and a full FP8 campaign, plus a smoke test. The code ran from uncommitted working trees; the exact code bundles (minus private Terraform state) are in the release.

GCP a3-highgpu-1g5 scenariosBF16 + FP83 runs
silybench · measured with vLLM and lm-evaluation-harness on rented GPUs · open data · reproduce