[Question] Blackwell (B200) performance

[haydn-jones]

Anyone have B200 Tok/s numbers they can share? I swear I had this working well at some point, but now I'm getting numbers far lower than I would expect.

For olmocr I'm seeing something like

Avg prompt throughput: 3431.0 tokens/s
Avg generation throughput: 1062.0 tokens/s

but with a text only model like Qwen3-30B-A3B-Instruct-2507 I'm getting closer to 45,000 Tok/s on prefill and 5k Tok/s on generation.

[jakep-allenai]

On H100's I'm seeing 3400 tok/sec, so I'd expect B200's to be higher. Let me run a quick test. I know that vllm 0.10 has some improvements on the B200 side

[jakep-allenai]

Yeah, Im getting similar, around 1500 tok/sec on B200, it should be much higher... I can look at this once we finish this 0.3 release which is coming imminently.

[jakep-allenai]

Just push 0.3.2 with vllm 0.10 and also some fixes to help limit CPU overloading on the pdf-page-renderer can you try again?

[haydn-jones]

It started out much better, but then fell off:

Avg prompt throughput: 171.1 tokens/s,   Avg generation throughput: 0.1 tokens/s,     Running: 1 reqs 
Avg prompt throughput: 0.0 tokens/s,     Avg generation throughput: 0.0 tokens/s,     Running: 1 reqs 
Avg prompt throughput: 5060.8 tokens/s,  Avg generation throughput: 10.5 tokens/s,    Running: 31 reqs
Avg prompt throughput: 24200.0 tokens/s, Avg generation throughput: 298.1 tokens/s,   Running: 177 reqs
Avg prompt throughput: 23075.2 tokens/s, Avg generation throughput: 691.9 tokens/s,   Running: 318 reqs
Avg prompt throughput: 22634.4 tokens/s, Avg generation throughput: 1067.0 tokens/s,  Running: 450 reqs
Avg prompt throughput: 18859.6 tokens/s, Avg generation throughput: 1183.5 tokens/s,  Running: 562 reqs
Avg prompt throughput: 15862.1 tokens/s, Avg generation throughput: 1209.3 tokens/s,  Running: 652 reqs
Avg prompt throughput: 33460.2 tokens/s, Avg generation throughput: 3215.2 tokens/s,  Running: 833 reqs
Avg prompt throughput: 10789.1 tokens/s, Avg generation throughput: 13891.9 tokens/s, Running: 841 reqs
Avg prompt throughput: 0.0 tokens/s,     Avg generation throughput: 21185.6 tokens/s, Running: 758 reqs
Avg prompt throughput: 168.3 tokens/s,   Avg generation throughput: 8342.0 tokens/s,  Running: 711 reqs
Avg prompt throughput: 836.3 tokens/s,   Avg generation throughput: 972.5 tokens/s,   Running: 705 reqs
Avg prompt throughput: 1186.7 tokens/s,  Avg generation throughput: 1319.6 tokens/s,  Running: 697 reqs
Avg prompt throughput: 509.9 tokens/s,   Avg generation throughput: 550.5 tokens/s,   Running: 696 reqs
Avg prompt throughput: 1162.1 tokens/s,  Avg generation throughput: 1133.7 tokens/s,  Running: 694 reqs
Avg prompt throughput: 1853.0 tokens/s,  Avg generation throughput: 1126.4 tokens/s,  Running: 698 reqs
Avg prompt throughput: 1459.1 tokens/s,  Avg generation throughput: 691.0 tokens/s,   Running: 701 reqs
Avg prompt throughput: 2409.3 tokens/s,  Avg generation throughput: 1176.7 tokens/s,  Running: 711 reqs
Avg prompt throughput: 1758.1 tokens/s,  Avg generation throughput: 1122.6 tokens/s,  Running: 713 reqs
Avg prompt throughput: 3439.7 tokens/s,  Avg generation throughput: 1280.6 tokens/s,  Running: 718 reqs
Avg prompt throughput: 5001.1 tokens/s,  Avg generation throughput: 1261.6 tokens/s,  Running: 733 reqs

A large backlog also built up:

Worker ID | finished | started
----------+----------+--------
0         | 330      | 501
1         | 242      | 411
2         | 58       | 586
3         | 0        | 470
4         | 0        | 472
5         | 0        | 570
6         | 0        | 565
7         | 0        | 514
8         | 0        | 582
9         | 0        | 470
10        | 0        | 423
11        | 0        | 538
12        | 0        | 579

Are those ~5.5k started requests actually in flight? Perhaps vLLMs scheduler is being overloaded?

[jakep-allenai]

Try one more time with v.0.3.3 🥲

Also, at least on my side, I've seen that flash infer might need an upgrade, investigating that now.

[haydn-jones]

Will do. I've yet to have vLLM (0.10 or nightly) work with flash-infer past 0.2.8, though you may have better luck.

[jakep-allenai]

Yeah, it's not quite working for me either yet, but I have fixed a bug with lowering OMP_MAX_THREADS which should help with the server overloading aspect.

[haydn-jones]

Looking better, nice work:

Avg prompt throughput: 17673.5 tokens/s, Avg generation throughput: 1673.8 tokens/s, Running: 772 reqs, Waiting: 9 reqs, GPU KV cache usage: 62.8%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 16800.9 tokens/s, Avg generation throughput: 1786.5 tokens/s, Running: 854 reqs, Waiting: 2 reqs, GPU KV cache usage: 67.7%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 8306.7 tokens/s, Avg generation throughput: 1920.4 tokens/s, Running: 891 reqs, Waiting: 0 reqs, GPU KV cache usage: 70.3%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 8785.7 tokens/s, Avg generation throughput: 1803.4 tokens/s, Running: 922 reqs, Waiting: 0 reqs, GPU KV cache usage: 72.3%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 13639.7 tokens/s, Avg generation throughput: 2073.9 tokens/s, Running: 978 reqs, Waiting: 0 reqs, GPU KV cache usage: 75.7%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 53251.2 tokens/s, Avg generation throughput: 34985.5 tokens/s, Running: 1021 reqs, Waiting: 133 reqs, GPU KV cache usage: 82.8%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 12825.1 tokens/s, Avg generation throughput: 10136.0 tokens/s, Running: 1024 reqs, Waiting: 58 reqs, GPU KV cache usage: 84.6%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 9734.3 tokens/s, Avg generation throughput: 12324.9 tokens/s, Running: 975 reqs, Waiting: 0 reqs, GPU KV cache usage: 82.7%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 1028.5 tokens/s, Avg generation throughput: 10640.7 tokens/s, Running: 885 reqs, Waiting: 0 reqs, GPU KV cache usage: 77.9%, Prefix cache hit rate: 4.8%
Avg prompt throughput: 5550.0 tokens/s, Avg generation throughput: 1486.7 tokens/s, Running: 906 reqs, Waiting: 0 reqs, GPU KV cache usage: 79.4%, Prefix cache hit rate: 4.7%
Avg prompt throughput: 6691.3 tokens/s, Avg generation throughput: 2000.8 tokens/s, Running: 922 reqs, Waiting: 0 reqs, GPU KV cache usage: 80.2%, Prefix cache hit rate: 4.7%
Avg prompt throughput: 11772.8 tokens/s, Avg generation throughput: 1780.1 tokens/s, Running: 970 reqs, Waiting: 2 reqs, GPU KV cache usage: 82.9%, Prefix cache hit rate: 4.7%
Avg prompt throughput: 46553.3 tokens/s, Avg generation throughput: 20231.1 tokens/s, Running: 1022 reqs, Waiting: 62 reqs, GPU KV cache usage: 86.0%, Prefix cache hit rate: 4.7%
Avg prompt throughput: 10844.0 tokens/s, Avg generation throughput: 9076.0 tokens/s, Running: 1003 reqs, Waiting: 0 reqs, GPU KV cache usage: 85.3%, Prefix cache hit rate: 4.7%

[jakep-allenai]

Phew!

[jakep-allenai]

Just a random question on this, are you using the native docker image directly, or do you have a custom vllm/flash infer setup?

[haydn-jones]

I'm not using docker right now, vllm and flashinfer through pypi. I've been having a ton of strange issues with vLLM on blackwell (like the flashinfer trt llm decode kernel just hanging, gpt-oss not working, etc.). I've had some luck compiling flashinfer from scratch. I'm going to compare flashinfer with the flash-attn 3 backend (the one under the hopper directory in the flash attn repo, not the pypi package) and then just pick whatever works best and process my documents.

[jakep-allenai]

Yeah, also seeing all sorts of random issues... I did a pip install vllm==v0.10.1.1 within the v.0.3.3 olmocr official container and that seems to be going okay? But I also had issues where it seemed okay but was generating garbage.

Still investigating.

Plan is to use B200's for some GPRO training against synthetic olmocr-bench like data, but so I also need vllm to work on there nicely.

[Backdrop9019]

Hi, is there any updated guidance for inference on B200?

Training on B200 works fine for me, but when I run the benchmark, the model fails to produce valid text tokens. Using the same weights on an L40s yields correct outputs.

Could you share a known-good environment/config for B200 (e.g., driver/CUDA, vLLM/flash-infer versions)? Any tips would be much appreciated!