TokenMark own benchmark run: vLLM on NVIDIA H100 80GB HBM3, 2026-10-15 Rented from Lambda Cloud, 1 GPU. Measured by TokenMark, not a vendor or community figure. nvidia-smi (name, driver, power limit, max SM clock, PCIe gen, PCIe width, memory): NVIDIA H100 80GB HBM3, 580.105.08, 700.00 W, 1980 MHz, 5, 16, 81559 MiB vLLM 0.31.0 (CUDA 12.9 wheel, torch 2.13.0+cu129), model Qwen/Qwen3.8-27B-FP8, max model length 65536, prefix caching off. vLLM's pick for this card, from the server log: weight kernel CutlassFp8BlockScaledMMKernel. Client: vbench.py (TokenMark), streaming /v1/completions on the raw prompt (no chat template), 256 generated tokens per request with ignore_eos, temperature 0.7, top_p 0.8, top_k 20. DeepGEMM off (VLLM_USE_DEEP_GEMM=0): the box's nvcc was 12.8, DeepGEMM's JIT needs 12.9+; vLLM picked another FP8 kernel. Not measured with DeepGEMM on. - 1 user: the first human turn of the first 10 ShareGPT V3 conversations, the same 10 prompts ppb uses, so the row lines up with the ppb llama.cpp rows. tok/s = generated tokens / wall time per request, time to first token included; the row is the median of the 10 requests. - N users: N requests in flight at once from a pool of max(16, 2N) further ShareGPT prompts. total tok/s = all generated tokens / time from the first request sent to the last token received. per-user tok/s is each request's own tok/s, median. decode tok/s = tokens after the first / time after the first token, median. - GPU W: nvidia-smi power.draw every 0.5 s, mean over the level. tok/J = total tok/s / W. - MTP accept len: 1 + accepted draft tokens / drafts, from vLLM's /metrics counters over the level. - long prompts: one request at a time; prefill tok/s = prompt tokens / time to first token. A target is approximate: each prompt is cut from the ShareGPT pool by a characters-per-token ratio, the same text on every card, so "prompt tokens" is what was sent (up to 1.6x the target). vLLM refuses a prompt + output over the max model length. Site rows: per config, every level below is a row (1 user single-stream, N users as a total for the card). Site tables show the 1-user row and the highest total (marked below) until a user count is picked in the Users filter. A level with any failed or short request is never a row. Each level ran once, so neighbouring levels can trade places (a total can dip from one user count to the next): read the trend across levels, not one step. == plain: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only (venv install; bench 2026-10-15T01:25:01Z to 2026-10-15T01:30:47Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 74.3 74.28 74.81 38 43 416.0 0.18 - 10/10 <- 1-user row 2 143.9 72.37 73.62 77 118 419.1 0.34 - 16/16 4 233.9 61.07 61.90 72 807 488.6 0.48 - 16/16 8 469.1 58.67 60.53 75 231 499.4 0.94 - 16/16 16 682.0 44.55 56.12 255 2654 550.6 1.24 - 32/32 32 1081.5 33.81 46.84 1483 2803 579.1 1.87 - 64/64 64 1927.5 30.69 38.69 878 3320 590.5 3.26 - 128/128 128 2652.9 21.02 25.89 2448 3148 622.7 4.26 - 256/256 256 2822.4 12.84 17.34 4805 17452 658.5 4.29 - 512/512 <- highest total 512 2776.1 6.76 15.73 22032 35280 657.9 4.22 - 1024/1024 long prompts, 1 user: target tokens prompt tokens TTFT p50 ms prefill tok/s decode tok/s ok/n 8000 10185 735 12149.0 73.55 3/3 32000 41750 4892 8040.2 70.06 3/3 == mtp4: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":4} (venv install; bench 2026-10-15T01:33:09Z to 2026-10-15T01:39:13Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 145.3 141.12 144.27 54 58 419.1 0.35 2.91 10/10 <- 1-user row 2 255.8 137.89 150.24 70 1512 405.9 0.63 3.01 16/16 4 305.1 94.50 110.90 76 1514 373.4 0.82 3.01 16/16 8 595.4 80.93 109.38 744 1568 434.4 1.37 3.10 16/16 16 702.4 44.40 73.75 1528 2917 483.0 1.45 3.06 32/32 32 1356.3 45.94 59.89 901 1318 635.1 2.14 2.92 64/64 64 1777.4 34.27 52.92 2200 6699 651.3 2.73 2.95 128/128 128 1463.1 12.81 37.98 13234 18837 662.3 2.21 3.01 256/256 256 1783.5 7.94 46.12 26246 34097 658.3 2.71 3.00 512/512 512 1794.7 3.68 44.51 63092 69125 658.1 2.73 2.99 1024/1024 <- highest total == mtp2: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.92 --max-num-seqs 512 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2} (venv install; bench 2026-10-15T01:41:25Z to 2026-10-15T01:46:57Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 135.3 139.08 142.09 52 55 408.5 0.33 2.33 10/10 <- 1-user row 2 236.8 132.38 139.45 67 1531 403.0 0.59 2.43 16/16 4 388.0 108.90 117.71 70 1567 421.6 0.92 2.48 16/16 8 632.0 82.39 113.46 186 1591 407.5 1.55 2.43 16/16 16 635.0 41.74 75.11 899 5126 428.0 1.48 2.43 32/32 32 1296.5 44.48 53.88 869 3988 493.4 2.63 2.39 64/64 64 2224.0 40.43 47.84 437 3410 647.7 3.43 2.39 128/128 <- highest total 128 1726.5 15.50 27.45 8681 12940 656.3 2.63 2.39 256/256 256 2017.4 9.02 34.54 19476 30415 658.0 3.07 2.39 512/512 512 1998.4 4.21 32.31 51844 57581 661.6 3.02 2.37 1024/1024 summary.json: every level above with all its fields, plus the server's /version and /v1/models answers. requests.jsonl: one line per request (prompt and output tokens, TTFT, tok/s), every level and config.