TokenMark own benchmark run: vLLM on NVIDIA H100 80GB HBM3, 2026-10-15 Rented from Lambda Cloud, 1 GPU. Measured by TokenMark, not a vendor or community figure. nvidia-smi (name, driver, power limit, max SM clock, PCIe gen, PCIe width, memory): NVIDIA H100 80GB HBM3, 570.148.08, 700.00 W, 1980 MHz, 5, 16, 81559 MiB vLLM 0.31.0 (CUDA 12.9 wheel, torch 2.13.0+cu129), model RedHatAI/Qwen3.8-27B-INT4, max model length 65536, prefix caching off. vLLM's pick for this card, from the server log: weight kernel MacheteLinearKernel. Client: vbench.py (TokenMark), streaming /v1/completions on the raw prompt (no chat template), 256 generated tokens per request with ignore_eos, temperature 0.7, top_p 0.8, top_k 20. - 1 user: the first human turn of the first 10 ShareGPT V3 conversations, the same 10 prompts ppb uses, so the row lines up with the ppb llama.cpp rows. tok/s = generated tokens / wall time per request, time to first token included; the row is the median of the 10 requests. - N users: N requests in flight at once from a pool of max(16, 2N) further ShareGPT prompts. total tok/s = all generated tokens / time from the first request sent to the last token received. per-user tok/s is each request's own tok/s, median. decode tok/s = tokens after the first / time after the first token, median. - GPU W: nvidia-smi power.draw every 0.5 s, mean over the level. tok/J = total tok/s / W. - MTP accept len: 1 + accepted draft tokens / drafts, from vLLM's /metrics counters over the level. - long prompts: one request at a time; prefill tok/s = prompt tokens / time to first token. A target is approximate: each prompt is cut from the ShareGPT pool by a characters-per-token ratio, the same text on every card, so "prompt tokens" is what was sent (up to 1.6x the target). vLLM refuses a prompt + output over the max model length. Site rows: per config, every level below is a row (1 user single-stream, N users as a total for the card). Site tables show the 1-user row and the highest total (marked below) until a user count is picked in the Users filter. A level with any failed or short request is never a row. Each level ran once, so neighbouring levels can trade places (a total can dip from one user count to the next): read the trend across levels, not one step. == plain: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only (venv install; bench 2026-10-15T01:49:45Z to 2026-10-15T01:54:48Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 85.3 85.26 86.00 38 57 397.5 0.21 - 10/10 <- 1-user row 2 165.6 83.50 85.25 81 131 407.8 0.41 - 16/16 4 322.8 82.34 83.84 71 282 421.4 0.77 - 16/16 8 607.0 76.00 79.85 76 285 443.9 1.37 - 16/16 16 967.7 61.38 73.93 322 1231 506.7 1.91 - 32/32 32 1499.9 47.17 58.86 830 1559 550.8 2.72 - 64/64 64 2324.1 37.20 44.94 399 2489 622.1 3.74 - 128/128 128 2780.7 21.79 28.15 2788 3216 671.3 4.14 - 256/256 256 3420.7 13.65 15.76 1584 7024 679.5 5.03 - 512/512 <- highest total 512 2823.4 5.82 11.19 25305 28018 661.6 4.27 - 1024/1024 long prompts, 1 user: target tokens prompt tokens TTFT p50 ms prefill tok/s decode tok/s ok/n 8000 10185 796 11225.3 84.27 3/3 32000 41750 4246 9621.1 79.83 3/3 == mtp4: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":4} (venv install; bench 2026-10-15T01:56:50Z to 2026-10-15T02:02:11Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 164.3 161.78 166.19 53 73 414.0 0.40 3.05 10/10 <- 1-user row 2 272.9 144.00 155.67 69 1490 405.3 0.67 2.98 16/16 4 363.9 124.55 140.08 70 1521 343.1 1.06 3.18 16/16 8 706.9 95.08 126.50 271 1557 422.9 1.67 3.00 16/16 16 882.4 61.01 97.41 599 2905 486.3 1.81 3.05 32/32 32 1581.2 56.52 66.35 363 1618 651.7 2.43 3.06 64/64 64 1870.5 35.61 40.77 1121 3368 663.5 2.82 2.98 128/128 128 1861.8 16.32 36.84 9052 11768 668.7 2.78 2.98 256/256 256 2011.4 8.60 38.54 23173 27148 664.6 3.03 3.01 512/512 <- highest total 512 1975.7 4.02 36.91 57529 59964 664.3 2.97 2.98 1024/1024 == mtp2: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2} (venv install; bench 2026-10-15T02:04:03Z to 2026-10-15T02:08:36Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 147.6 145.78 149.40 41 58 399.9 0.37 2.31 10/10 <- 1-user row 2 258.9 138.28 149.07 66 254 388.2 0.67 2.40 16/16 4 451.7 125.70 144.23 78 1633 384.0 1.18 2.43 16/16 8 479.9 62.63 116.07 1567 2805 304.6 1.58 2.37 16/16 16 1070.1 71.17 93.82 430 1769 507.8 2.11 2.41 32/32 32 1401.0 50.17 67.41 462 4239 529.2 2.65 2.36 64/64 64 2534.2 43.31 52.16 273 2611 676.7 3.75 2.38 128/128 <- highest total 128 2379.7 21.81 30.33 2605 9937 678.7 3.51 2.37 256/256 256 2507.6 11.52 30.62 15471 20498 670.0 3.74 2.38 512/512 512 2476.7 5.24 28.78 40202 45249 668.0 3.71 2.38 1024/1024 summary.json: every level above with all its fields, plus the server's /version and /v1/models answers. requests.jsonl: one line per request (prompt and output tokens, TTFT, tok/s), every level and config.