Hardware › H100 PCIe local LLM benchmarks: 19 measured configs
H100 PCIe local LLM benchmarks: 19 measured configs
19 measured local-LLM configs on NVIDIA H100 PCIe (80GB); fastest 14B+ model Qwen3.6-35B-A3B at 152.6 tok/s. Official specs, every number linked to its source.
Snapshot 2026-10-15 · 394 configs on AI boxes and 423 on GPU cards · 123 models. Every number is generated from the tracker snapshot and linked to where it was measured; nothing here is typed by hand.
Quick answers
- What is the fastest model on H100 PCIe?
- Qwen3.6-35B-A3B at 152.6 tok/s single-stream (UD-Q4_K_M, llama.cpp/cuda), source last checked 2026-10-14. Models under 14B parameters are left out of this answer. source
- How many configs does TokenMark track for H100 PCIe?
- 19 single-stream configs across 6 models, each linked to the README or model card it came from.
Measured single-stream configs on NVIDIA H100 PCIe (80GB)
| Model | Quant | Backend | Decode | Context | Trust | Source |
|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | UD-Q4_K_M | llama.cpp/cuda | 152.6 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.6-35B-A3B | UD-Q4_K_XL | llama.cpp/cuda | 152.1 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.6-35B-A3B | UD-Q5_K_M | llama.cpp/cuda | 151.3 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.5-35B-A3B | Q4_K_M | llama.cpp/cuda | 151.3 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.6-35B-A3B | UD-Q6_K | llama.cpp/cuda | 146.1 tok/s | 8192 | Verified (ran it) | source |
| gemma-4-26B-A4B-it | UD-Q8_K_XL | llama.cpp/cuda | 138 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.8-27B | INT4 | vllm (speculative decode, mtp-4) | 103.7 tok/s | 65536 | Verified (ran it) | source |
| Qwen3.8-27B | INT4 | vllm (speculative decode, mtp-2) | 101.4 tok/s | 65536 | Verified (ran it) | source |
| Qwen3.8-27B | FP8 | vllm (speculative decode, mtp-4) | 95.9 tok/s | 65536 | Verified (ran it) | source |
| Qwen3.8-27B | FP8 | vllm (speculative decode, mtp-2) | 91.1 tok/s | 65536 | Verified (ran it) | source |
| Qwen3.8-27B | INT4 | vllm | 60 tok/s (prefill 4645.6) | 65536 | Verified (ran it) | source |
| Qwen3.8-27B | UD-Q4_K_M | llama.cpp/cuda | 56.7 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.8-27B | UD-Q4_K_XL | llama.cpp/cuda | 54.4 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.6-27B | Q4_K_M | llama.cpp/cuda | 52.7 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.5-27B | Q4_K_M | llama.cpp/cuda | 52.3 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.8-27B | FP8 | vllm | 51.7 tok/s (prefill 7316.3) | 65536 | Verified (ran it) | source |
| Qwen3.8-27B | Q8_0 | llama.cpp/cuda | 46 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.6-27B | Q8_0 | llama.cpp/cuda | 44.8 tok/s | 8192 | Verified (ran it) | source |
| Qwen3.8-27B | UD-Q6_K_XL | llama.cpp/cuda | 44.4 tok/s | 8192 | Verified (ran it) | source |
More: all configs · hardware · news · contributors · methodology · llms.txt · llms-full.txt · API (OpenAPI) · JSON snapshot. Built by Altronis, private on-prem AI, Singapore.