Reproduce
Run the benchmark on your own GPU
Every number here can be re-measured on any rented GPU (RunPod, Vast, Lambda, Hyperbolic, a cloud VM or your own box). An AI agent can do the whole thing from one prompt.
1 · Rent a GPU and start Claude Code on it
Any Linux box with an NVIDIA GPU, SSH, and ~100 GB of disk. Docker is optional: RunPod/Vast pods work as-is.
Paste this into Claude:
Clone https://github.com/tgarg01/silybench-bench and follow its AGENTS.md to benchmark this GPU. Ask me for the provider, hourly price and campaign, and show me the cost estimate before starting.
- Claude runs setup and pre-flight checks (GPU, disk, limits, tokens) and picks Docker or a pip-installed vLLM.
- It asks for your provider and hourly price, then shows the estimated hours and dollars and waits for your OK.
- The run is detached and resumable, so a dropped SSH session or crash loses at most one measurement.
- When it finishes, it opens a pull request with the results on the public dataset and reminds you to shut the box down.
2 · Or by hand
The same steps without an agent.
git clone https://github.com/tgarg01/silybench-bench && cd silybench-bench ./setup.sh configs/qwen3-8b-quick.yaml uv run gpubench plan configs/qwen3-8b-quick.yaml --price-per-hour 2.69 uv run gpubench run configs/qwen3-8b-quick.yaml --provider runpod --price-per-hour 2.69 --resume --detach uv run gpubench status gh auth login && uv run gpubench submit results/*/
| Campaign | What | ~1× H100 |
|---|---|---|
| smoke | Pipeline check, not published | 15 min |
| qwen3-8b-quick | BF16 + FP8, 3 request shapes, no accuracy | 1.5 h |
| qwen3-8b | Full: 5 shapes × 8 load levels × 3 repeats, capacity search, 6 accuracy tasks | 12-16 h |
| qwen3-14b | Full, Qwen3-14B | 20+ h |
3 · Re-run a specific published result
REPRODUCE.md lists, for every run, the bench commit, vLLM image digest, model revision, serving flags and the exact command. Raw per-repeat measurements and GPU telemetry are in each run's raw.tar.gz.