TokenMark own benchmark run: vLLM on NVIDIA H100 PCIe, 2026-10-14 Rented from Lambda Cloud, 1 GPU. Measured by TokenMark, not a vendor or community figure. nvidia-smi (name, driver, power limit, max SM clock, PCIe gen, PCIe width, memory): NVIDIA H100 PCIe, 570.148.08, 350.00 W, 1755 MHz, 5, 16, 81559 MiB vLLM 0.31.0 (CUDA 12.9 wheel, torch 2.13.0+cu129), model Qwen/Qwen3.8-27B-FP8, max model length 65536, prefix caching off. vLLM's pick for this card, from the server log: weight kernel CutlassFp8BlockScaledMMKernel. Client: vbench.py (TokenMark), streaming /v1/completions on the raw prompt (no chat template), 256 generated tokens per request with ignore_eos, temperature 0.7, top_p 0.8, top_k 20. DeepGEMM off (VLLM_USE_DEEP_GEMM=0): the box's nvcc was 12.8, DeepGEMM's JIT needs 12.9+; vLLM picked another FP8 kernel. Not measured with DeepGEMM on. - 1 user: the first human turn of the first 10 ShareGPT V3 conversations, the same 10 prompts ppb uses, so the row lines up with the ppb llama.cpp rows. tok/s = generated tokens / wall time per request, time to first token included; the row is the median of the 10 requests. - N users: N requests in flight at once from a pool of max(16, 2N) further ShareGPT prompts. total tok/s = all generated tokens / time from the first request sent to the last token received. per-user tok/s is each request's own tok/s, median. decode tok/s = tokens after the first / time after the first token, median. - GPU W: nvidia-smi power.draw every 0.5 s, mean over the level. tok/J = total tok/s / W. - MTP accept len: 1 + accepted draft tokens / drafts, from vLLM's /metrics counters over the level. - long prompts: one request at a time; prefill tok/s = prompt tokens / time to first token. A target is approximate: each prompt is cut from the ShareGPT pool by a characters-per-token ratio, the same text on every card, so "prompt tokens" is what was sent (up to 1.6x the target). vLLM refuses a prompt + output over the max model length. Site rows: per config, every level below is a row (1 user single-stream, N users as a total for the card). Site tables show the 1-user row and the highest total (marked below) until a user count is picked in the Users filter. A level with any failed or short request is never a row. Each level ran once, so neighbouring levels can trade places (a total can dip from one user count to the next): read the trend across levels, not one step. == plain: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only (venv install; bench 2026-10-14T06:16:07Z to 2026-10-14T06:24:42Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 51.7 51.70 52.70 113 138 271.4 0.19 - 10/10 <- 1-user row 2 100.0 50.28 51.89 189 235 276.9 0.36 - 16/16 4 173.0 45.25 46.48 185 1161 331.2 0.52 - 16/16 8 341.5 42.74 44.80 191 418 338.3 1.01 - 16/16 16 558.3 35.27 41.18 492 1778 343.1 1.63 - 32/32 32 748.2 23.41 33.58 2355 4387 344.5 2.17 - 64/64 64 1365.0 22.12 27.61 773 5182 345.4 3.95 - 128/128 128 1698.2 13.30 18.08 4831 7022 347.1 4.89 - 256/256 256 1727.7 7.77 10.46 7856 27071 346.9 4.98 - 512/512 512 1835.0 4.33 10.31 35581 55639 346.9 5.29 - 1024/1024 <- highest total long prompts, 1 user: target tokens prompt tokens TTFT p50 ms prefill tok/s decode tok/s ok/n 8000 10185 1221 7316.3 51.85 3/3 32000 41750 7814 5034.2 49.35 3/3 60000 - - - - 0/3 refused: the prompt came to 65281 tokens, + 256 output is over the max model length 65536 == mtp4: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":4} (venv install; bench 2026-10-14T06:27:45Z to 2026-10-14T06:36:41Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 99.5 95.91 99.82 133 135 268.8 0.37 2.91 10/10 <- 1-user row 2 167.5 97.16 102.88 155 327 252.4 0.66 3.05 16/16 4 194.2 69.28 76.72 159 3276 229.7 0.84 3.06 16/16 8 378.6 51.05 71.42 244 3271 259.0 1.46 3.02 16/16 16 410.6 26.85 39.75 1238 5280 245.9 1.67 3.05 32/32 32 910.2 32.13 40.91 1525 2500 341.1 2.67 2.93 64/64 64 1149.7 21.99 35.38 2957 11058 343.9 3.34 3.00 128/128 128 1025.6 8.80 26.90 19389 25750 344.9 2.97 3.01 256/256 256 1266.8 5.69 33.67 36702 48989 344.3 3.68 3.01 512/512 <- highest total 512 1224.9 2.50 32.02 93150 99663 344.3 3.56 3.01 1024/1024 == mtp2: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2} (venv install; bench 2026-10-14T06:39:53Z to 2026-10-14T06:48:00Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 93.3 91.14 94.85 127 130 265.9 0.35 2.32 10/10 <- 1-user row 2 179.4 90.60 97.81 150 665 273.4 0.66 2.46 16/16 4 253.5 75.39 86.12 154 3408 266.7 0.95 2.47 16/16 8 384.8 51.98 71.06 235 3382 248.1 1.55 2.37 16/16 16 381.5 25.94 48.67 1765 9590 244.1 1.56 2.41 32/32 32 671.8 22.72 28.86 1508 6851 260.4 2.58 2.38 64/64 64 1508.9 27.50 33.16 764 5363 341.4 4.42 2.38 128/128 <- highest total 128 1191.3 10.21 20.21 13565 17605 345.0 3.45 2.39 256/256 256 1371.7 6.37 23.68 27798 42848 344.9 3.98 2.38 512/512 512 1415.4 3.01 22.73 70615 83041 345.4 4.10 2.38 1024/1024 summary.json: every level above with all its fields, plus the server's /version and /v1/models answers. requests.jsonl: one line per request (prompt and output tokens, TTFT, tok/s), every level and config.