TokenMark own benchmark run: vLLM on NVIDIA A10, 2026-10-14 Rented from Lambda Cloud, 1 GPU. Measured by TokenMark, not a vendor or community figure. nvidia-smi (name, driver, power limit, max SM clock, PCIe gen, PCIe width, memory): NVIDIA A10, 570.148.08, 150.00 W, 1695 MHz, 4, 16, 23028 MiB vLLM 0.31.0 (CUDA 12.9 wheel, torch 2.13.0+cu129), model RedHatAI/Qwen3.8-27B-INT4, max model length 32768, prefix caching off. vLLM's pick for this card, from the server log: weight kernel MarlinLinearKernel. Client: vbench.py (TokenMark), streaming /v1/completions on the raw prompt (no chat template), 256 generated tokens per request with ignore_eos, temperature 0.7, top_p 0.8, top_k 20. - 1 user: the first human turn of the first 10 ShareGPT V3 conversations, the same 10 prompts ppb uses, so the row lines up with the ppb llama.cpp rows. tok/s = generated tokens / wall time per request, time to first token included; the row is the median of the 10 requests. - N users: N requests in flight at once from a pool of max(16, 2N) further ShareGPT prompts. total tok/s = all generated tokens / time from the first request sent to the last token received. per-user tok/s is each request's own tok/s, median. decode tok/s = tokens after the first / time after the first token, median. - GPU W: nvidia-smi power.draw every 0.5 s, mean over the level. tok/J = total tok/s / W. - MTP accept len: 1 + accepted draft tokens / drafts, from vLLM's /metrics counters over the level. - long prompts: one request at a time; prefill tok/s = prompt tokens / time to first token. A target is approximate: each prompt is cut from the ShareGPT pool by a characters-per-token ratio, the same text on every card, so "prompt tokens" is what was sent (up to 1.6x the target). vLLM refuses a prompt + output over the max model length. Site rows: per config, every level below is a row (1 user single-stream, N users as a total for the card). Site tables show the 1-user row and the highest total (marked below) until a user count is picked in the Users filter. A level with any failed or short request is never a row. Each level ran once, so neighbouring levels can trade places (a total can dip from one user count to the next): read the trend across levels, not one step. What the job cut or could not run on this card (from its log): - plain: profiling KV cache fits about 50 sequences, --max-num-seqs 32, levels 1,2,4,8,16,32 - plain: Mamba cache fits 26 sequences, --max-num-seqs 16, levels 1,2,4,8,16 - mtp4: the server never came up, out of GPU memory (asked for 2.37 GiB with 1.72 GiB free of 22.07 GiB), so no rows - mtp2: the server never came up, out of GPU memory (asked for 2.37 GiB with 1.72 GiB free of 22.07 GiB), so no rows == plain: VLLM_USE_DEEP_GEMM=0 vllm serve RedHatAI/Qwen3.8-27B-INT4 --served-model-name qwen3.8-27b --max-model-len 32768 --gpu-memory-utilization 0.95 --max-num-seqs 16 --no-enable-prefix-caching --language-model-only (venv install; bench 2026-10-14T08:01:37Z to 2026-10-14T08:08:12Z) users total tok/s per-user tok/s decode tok/s TTFT p50 ms TTFT p90 ms GPU W tok/J MTP accept len ok/n 1 28.3 28.41 28.55 69 297 149.7 0.19 - 10/10 <- 1-user row 2 51.1 26.37 26.71 155 987 149.9 0.34 - 16/16 4 95.3 25.34 25.71 185 2268 149.9 0.64 - 16/16 8 124.2 24.84 25.63 262 10377 149.8 0.83 - 16/16 <- highest total 16 108.8 8.23 25.20 19136 32704 149.6 0.73 - 32/32 long prompts, 1 user: target tokens prompt tokens TTFT p50 ms prefill tok/s decode tok/s ok/n 8000 10185 8039 1111.3 27.80 3/3 32000 - - - - 0/3 refused: the prompt came to 32513 tokens, + 256 output is over the max model length 32768 summary.json: every level above with all its fields, plus the server's /version and /v1/models answers. requests.jsonl: one line per request (prompt and output tokens, TTFT, tok/s), every level and config.