TokenMark own benchmark run: vLLM on NVIDIA H100 PCIe, 2026-10-14 Rented from Lambda Cloud, 1 GPU. Measured by TokenMark, not a vendor or community figure. nvidia-smi (name, driver, power limit, max SM clock, PCIe gen, PCIe width, memory): NVIDIA H100 PCIe, 570.148.08, 350.00 W, 1755 MHz, 5, 16, 81559 MiB vLLM 0.31.0 (CUDA 12.9 wheel, torch 2.13.0+cu129), model RedHatAI/Qwen3.8-27B-INT4, max model length 65536, prefix caching off. vLLM's pick for this card, from the server log: weight kernel MacheteLinearKernel. Client: vbench.py (TokenMark), streaming /v1/completions on the raw prompt (no chat template), 256 generated tokens per request with ignore_eos, temperature 0.7, top_p 0.8, top_k 20. - 1 user: the first human turn of the first 10 ShareGPT V3 conversations, the same 10 prompts ppb uses, so the row lines up with the ppb llama.cpp rows. tok/s = generated tokens / wall time per request, time to first token included; the row is the median of the 10 requests. - N users: N requests in flight at once from a pool of max(16, 2N) further ShareGPT prompts. total tok/s = all generated tokens / time from the first request sent to the last token received. per-user tok/s is each request's own tok/s, median. decode tok/s = tokens after the first / time after the first token, median. - GPU W: nvidia-smi power.draw every 0.5 s, mean over the level. tok/J = total tok/s / W. - MTP accept len: 1 + accepted draft tokens / drafts, from vLLM's /metrics counters over the level. - long prompts: one request at a time; prefill tok/s = prompt tokens / time to first token. A target is approximate: each prompt is cut from the ShareGPT pool by a characters-per-token ratio, the same text on every card, so "prompt tokens" is what was sent (up to 1.6x the target). vLLM refuses a prompt + output over the max model length. Site rows: per config, every level below is a row (1 user single-stream, N users as a total for the card). Site tables show the 1-user row and the highest total (marked below) until a user count is picked in the Users filter. A level with any failed or short request is never a row. Each level ran once, so neighbouring levels can trade places (a total can dip from one user count to the next): read the trend across levels, not one step. == plain: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only (venv install; bench 2026-10-14T07:46:52Z to 2026-10-14T07:55:22Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 60.0 60.00 61.35 111 113 280.4 0.21 - 10/10 <- 1-user row 2 114.7 58.19 60.39 184 298 295.5 0.39 - 16/16 4 218.8 56.27 58.29 177 615 293.0 0.75 - 16/16 8 401.3 50.30 54.47 198 647 291.6 1.38 - 16/16 16 585.8 37.61 49.30 716 2905 284.7 2.06 - 32/32 32 903.6 28.49 39.98 1922 3582 283.7 3.19 - 64/64 64 1335.4 21.62 28.02 987 5592 282.8 4.72 - 128/128 128 1523.0 11.95 17.35 6892 7994 283.2 5.38 - 256/256 256 1877.7 7.54 9.07 3338 15243 282.4 6.65 - 512/512 <- highest total 512 1611.0 3.25 6.59 43901 49940 277.9 5.80 - 1024/1024 long prompts, 1 user: target tokens prompt tokens TTFT p50 ms prefill tok/s decode tok/s ok/n 8000 10185 2029 4645.6 59.30 3/3 32000 41750 9005 4579.9 56.73 3/3 60000 - - - - 0/3 refused: the prompt came to 65281 tokens, + 256 output is over the max model length 65536 == mtp4: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":4} (venv install; bench 2026-10-14T07:58:04Z to 2026-10-14T08:07:04Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 106.7 103.73 108.84 131 132 287.2 0.37 2.91 10/10 <- 1-user row 2 179.2 104.62 110.97 149 462 277.9 0.65 3.05 16/16 4 220.1 81.62 94.58 153 3164 227.4 0.97 3.08 16/16 8 423.1 58.86 88.09 434 3272 266.5 1.59 3.00 16/16 16 535.6 37.01 56.31 624 5282 255.9 2.09 2.96 32/32 32 939.4 32.61 40.86 1376 3310 288.6 3.25 3.03 64/64 64 994.4 18.52 24.83 2486 5582 265.7 3.74 2.97 128/128 128 1104.7 9.69 21.89 15502 20090 279.0 3.96 2.99 256/256 256 1189.7 5.16 23.10 38382 46174 274.4 4.34 3.01 512/512 <- highest total 512 1174.7 2.42 22.31 95033 100471 273.8 4.29 3.00 1024/1024 == mtp2: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2} (venv install; bench 2026-10-14T08:09:46Z to 2026-10-14T08:17:24Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 103.3 101.39 106.19 122 125 277.2 0.37 2.33 10/10 <- 1-user row 2 191.6 95.46 100.55 142 205 294.7 0.65 2.36 16/16 4 286.9 85.56 93.03 144 3187 261.1 1.10 2.43 16/16 8 442.7 59.80 84.22 202 3204 252.4 1.75 2.39 16/16 16 469.2 33.62 61.43 670 8500 229.2 2.05 2.40 32/32 32 741.3 24.54 32.24 1224 7612 249.9 2.97 2.40 64/64 64 1513.2 25.79 32.21 486 4632 296.3 5.11 2.40 128/128 <- highest total 128 1348.5 12.34 18.13 5485 17621 276.5 4.88 2.38 256/256 256 1472.1 6.59 18.30 25715 35952 273.8 5.38 2.38 512/512 512 1430.9 3.04 16.85 69253 77965 271.6 5.27 2.38 1024/1024 summary.json: every level above with all its fields, plus the server's /version and /v1/models answers. requests.jsonl: one line per request (prompt and output tokens, TTFT, tok/s), every level and config.