TokenMark own benchmark run: vLLM on NVIDIA A100-SXM4-40GB, 2026-10-14 Rented from Lambda Cloud, 1 GPU. Measured by TokenMark, not a vendor or community figure. nvidia-smi (name, driver, power limit, max SM clock, PCIe gen, PCIe width, memory): NVIDIA A100-SXM4-40GB, 570.148.08, 400.00 W, 1410 MHz, 4, 16, 40960 MiB vLLM 0.31.0 (CUDA 12.9 wheel, torch 2.13.0+cu129), model RedHatAI/Qwen3.8-27B-INT4, max model length 65536, prefix caching off. vLLM's pick for this card, from the server log: weight kernel MarlinLinearKernel. Client: vbench.py (TokenMark), streaming /v1/completions on the raw prompt (no chat template), 256 generated tokens per request with ignore_eos, temperature 0.7, top_p 0.8, top_k 20. - 1 user: the first human turn of the first 10 ShareGPT V3 conversations, the same 10 prompts ppb uses, so the row lines up with the ppb llama.cpp rows. tok/s = generated tokens / wall time per request, time to first token included; the row is the median of the 10 requests. - N users: N requests in flight at once from a pool of max(16, 2N) further ShareGPT prompts. total tok/s = all generated tokens / time from the first request sent to the last token received. per-user tok/s is each request's own tok/s, median. decode tok/s = tokens after the first / time after the first token, median. - GPU W: nvidia-smi power.draw every 0.5 s, mean over the level. tok/J = total tok/s / W. - MTP accept len: 1 + accepted draft tokens / drafts, from vLLM's /metrics counters over the level. - long prompts: one request at a time; prefill tok/s = prompt tokens / time to first token. A target is approximate: each prompt is cut from the ShareGPT pool by a characters-per-token ratio, the same text on every card, so "prompt tokens" is what was sent (up to 1.6x the target). vLLM refuses a prompt + output over the max model length. Site rows: per config, every level below is a row (1 user single-stream, N users as a total for the card). Site tables show the 1-user row and the highest total (marked below) until a user count is picked in the Users filter. A level with any failed or short request is never a row. Each level ran once, so neighbouring levels can trade places (a total can dip from one user count to the next): read the trend across levels, not one step. What the job cut or could not run on this card (from its log): - plain: profiling KV cache fits about 265 sequences, --max-num-seqs 256, levels 1,2,4,8,16,32,64,128,256 - mtp4: kept the cut an earlier config made: up to 256 users - mtp2: kept the cut an earlier config made: up to 256 users == plain: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.95 --max-num-seqs 256 --no-enable-prefix-caching --language-model-only (venv install; bench 2026-10-14T08:01:16Z to 2026-10-14T08:09:20Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 62.2 62.27 63.16 74 111 335.1 0.19 - 10/10 <- 1-user row 2 116.4 59.48 61.40 160 381 337.3 0.34 - 16/16 4 218.2 57.19 58.95 157 837 353.6 0.62 - 16/16 8 396.3 49.96 55.66 163 854 357.3 1.11 - 16/16 16 509.5 33.12 47.42 938 4457 392.3 1.30 - 32/32 32 727.7 23.12 27.01 1037 5358 397.4 1.83 - 64/64 64 981.8 16.14 20.18 1042 8523 388.7 2.53 - 128/128 <- highest total 128 836.7 8.94 11.71 9007 26080 396.5 2.11 - 256/256 256 955.9 4.39 13.33 39469 54360 396.5 2.41 - 512/512 long prompts, 1 user: target tokens prompt tokens TTFT p50 ms prefill tok/s decode tok/s ok/n 8000 10185 2843 3142.5 61.88 3/3 32000 41750 15272 2674.9 59.11 3/3 60000 - - - - 0/3 refused: the prompt came to 65281 tokens, + 256 output is over the max model length 65536 == mtp4: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.95 --max-num-seqs 256 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":4} (venv install; bench 2026-10-14T08:14:13Z to 2026-10-14T08:23:46Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 70.9 72.19 74.44 120 162 262.4 0.27 2.86 10/10 <- 1-user row 2 127.9 65.34 75.45 180 951 270.4 0.47 3.01 16/16 4 180.4 56.16 66.45 185 4320 251.2 0.72 3.09 16/16 8 340.0 47.37 56.27 249 2738 299.2 1.14 3.04 16/16 16 333.4 21.95 25.05 1194 1279 302.2 1.10 3.03 32/32 32 494.3 17.92 33.41 6361 11390 379.2 1.30 3.03 64/64 64 562.7 9.99 35.89 18342 23321 380.7 1.48 3.06 128/128 128 546.4 4.60 33.67 47554 53566 381.1 1.43 3.02 256/256 256 580.4 2.45 34.94 96839 108150 376.1 1.54 3.00 512/512 <- highest total == mtp2: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 65536 --gpu-memory-utilization 0.95 --max-num-seqs 256 --no-enable-prefix-caching --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2} (venv install; bench 2026-10-14T08:28:58Z to 2026-10-14T08:36:55Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 68.9 69.02 70.88 109 147 252.0 0.27 2.31 10/10 <- 1-user row 2 131.9 66.45 70.47 161 202 262.2 0.50 2.40 16/16 4 208.6 57.97 62.96 162 3021 263.2 0.79 2.38 16/16 8 342.0 45.66 61.41 865 2757 278.4 1.23 2.36 16/16 16 382.2 24.88 31.50 2186 3486 286.8 1.33 2.42 32/32 32 490.4 17.71 23.80 2025 8123 332.5 1.48 2.40 64/64 64 715.0 13.48 29.55 12020 15983 378.1 1.89 2.39 128/128 128 695.3 6.04 26.41 32542 40396 386.4 1.80 2.38 256/256 256 754.5 3.24 28.97 69268 80907 381.4 1.98 2.38 512/512 <- highest total summary.json: every level above with all its fields, plus the server's /version and /v1/models answers. requests.jsonl: one line per request (prompt and output tokens, TTFT, tok/s), every level and config.