TokenMark own benchmark run: vLLM on NVIDIA A100-SXM4-40GB, 2026-10-14 Rented from Lambda Cloud, 1 GPU. Measured by TokenMark, not a vendor or community figure. nvidia-smi (name, driver, power limit, max SM clock, PCIe gen, PCIe width, memory): NVIDIA A100-SXM4-40GB, 570.148.08, 400.00 W, 1410 MHz, 4, 16, 40960 MiB vLLM 0.31.0 (CUDA 12.9 wheel, torch 2.13.0+cu129), model Qwen/Qwen3.8-27B-FP8, max model length per config (plain 65536, mtp4 32768, mtp2 32768), prefix caching off. vLLM's pick for this card, from the server log: weight kernel MarlinFP8ScaledMMLinearKernel. Client: vbench.py (TokenMark), streaming /v1/completions on the raw prompt (no chat template), 256 generated tokens per request with ignore_eos, temperature 0.7, top_p 0.8, top_k 20. - 1 user: the first human turn of the first 10 ShareGPT V3 conversations, the same 10 prompts ppb uses, so the row lines up with the ppb llama.cpp rows. tok/s = generated tokens / wall time per request, time to first token included; the row is the median of the 10 requests. - N users: N requests in flight at once from a pool of max(16, 2N) further ShareGPT prompts. total tok/s = all generated tokens / time from the first request sent to the last token received. per-user tok/s is each request's own tok/s, median. decode tok/s = tokens after the first / time after the first token, median. - GPU W: nvidia-smi power.draw every 0.5 s, mean over the level. tok/J = total tok/s / W. - MTP accept len: 1 + accepted draft tokens / drafts, from vLLM's /metrics counters over the level. - long prompts: one request at a time; prefill tok/s = prompt tokens / time to first token. A target is approximate: each prompt is cut from the ShareGPT pool by a characters-per-token ratio, the same text on every card, so "prompt tokens" is what was sent (up to 1.6x the target). vLLM refuses a prompt + output over the max model length. Site rows: per config, every level below is a row (1 user single-stream, N users as a total for the card). Site tables show the 1-user row and the highest total (marked below) until a user count is picked in the Users filter. A level with any failed or short request is never a row. Each level ran once, so neighbouring levels can trade places (a total can dip from one user count to the next): read the trend across levels, not one step. What the job cut or could not run on this card (from its log): - plain: profiling KV cache fits about 130 sequences, --max-num-seqs 128, levels 1,2,4,8,16,32,64,128 - plain: Mamba cache fits 127 sequences, --max-num-seqs 64, levels 1,2,4,8,16,32,64 - mtp4: KV cache too small for --max-model-len 65536, retrying at 32768 - mtp4: kept the cut an earlier config made: up to 64 users - mtp2: kept the cut an earlier config made: max model length 32768, up to 64 users == plain: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.95 --max-num-seqs 64 --no-enable-prefix-caching --language-model-only (venv install; bench 2026-10-14T08:03:24Z to 2026-10-14T08:09:44Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 41.1 41.15 41.63 95 131 277.6 0.15 - 10/10 <- 1-user row 2 78.1 39.73 40.78 210 439 284.4 0.28 - 16/16 4 149.0 38.61 39.66 205 948 289.0 0.52 - 16/16 8 276.5 34.78 37.88 207 964 302.5 0.91 - 16/16 16 386.1 24.80 33.38 1060 4994 340.4 1.13 - 32/32 32 481.4 18.64 22.05 1167 6191 353.3 1.36 - 64/64 64 601.3 11.94 25.85 11595 17881 354.9 1.69 - 128/128 <- highest total long prompts, 1 user: target tokens prompt tokens TTFT p50 ms prefill tok/s decode tok/s ok/n 8000 10185 3157 2830.4 40.87 3/3 32000 41750 16106 2536.2 38.37 3/3 60000 - - - - 0/3 refused: the prompt came to 65281 tokens, + 256 output is over the max model length 65536 == mtp4: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 32768 --gpu-memory-utilization 0.95 --max-num-seqs 64 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":4} (venv install; bench 2026-10-14T08:18:16Z to 2026-10-14T08:23:13Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 79.4 81.22 84.32 127 162 278.6 0.28 2.90 10/10 <- 1-user row 2 134.1 70.52 78.18 149 2587 271.9 0.49 2.93 16/16 4 206.0 62.55 66.78 153 1040 267.1 0.77 3.07 16/16 8 307.0 45.23 70.75 1911 3223 328.3 0.94 3.14 16/16 <- highest total 16 274.5 18.16 66.75 9655 13370 338.6 0.81 3.11 32/32 32 287.5 9.66 67.09 22323 25571 344.7 0.83 3.04 64/64 64 306.0 5.13 68.31 45927 52859 341.8 0.90 3.00 128/128 == mtp2: VLLM_USE_DEEP_GEMM=0 vllm serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen3.8-27b --max-model-len 32768 --gpu-memory-utilization 0.95 --max-num-seqs 64 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2} (venv install; bench 2026-10-14T08:27:46Z to 2026-10-14T08:31:53Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 74.8 76.49 79.16 125 129 272.7 0.27 2.31 10/10 <- 1-user row 2 137.0 70.28 73.32 146 212 278.5 0.49 2.35 16/16 4 218.0 58.55 69.92 219 2781 264.5 0.82 2.42 16/16 8 304.4 41.35 55.14 129 4362 248.9 1.22 2.46 16/16 16 347.9 23.42 47.80 4456 8222 366.2 0.95 2.40 32/32 32 373.6 13.00 50.41 13335 19254 362.9 1.03 2.40 64/64 64 426.7 7.34 55.20 29633 35659 359.5 1.19 2.37 128/128 <- highest total summary.json: every level above with all its fields, plus the server's /version and /v1/models answers. requests.jsonl: one line per request (prompt and output tokens, TTFT, tok/s), every level and config.