[vLLM] Quality Regression with FlashInfer vs Triton Attention

[jwfromm]

Hello,

While running GPT-OSS 120B in vLLM, we noticed that FlashInfer introduces a quality regression. Specifically we achieve the expected quality scores (92% on AIME) when setting:

VLLM_USE_FLASHINFER_MOE_MXFP4_BF16=1 \
VLLM_USE_TRTLLM_ATTENTION=0 \
VLLM_USE_TRTLLM_CONTEXT_ATTENTION=0 \
VLLM_USE_TRTLLM_DECODE_ATTENTION=0 \
VLLM_ATTENTION_BACKEND=TRITON_ATTN_VLLM_V1 \
VLLM_USE_V1=1 \

Switching to FlashInfer based attention (ie):

VLLM_USE_FLASHINFER_MOE_MXFP4_BF16=1 \
VLLM_USE_TRTLLM_ATTENTION=1 \
VLLM_USE_TRTLLM_CONTEXT_ATTENTION=1 \
VLLM_USE_TRTLLM_DECODE_ATTENTION=1 \
VLLM_ATTENTION_BACKEND=CUTLASS_MLA \
VLLM_USE_V1=1 \

Consistently reduces the score to around 88% in high reasoning mode. We're not sure exactly where this issue is coming from, but it is preventing us from using the great kernels in this project. Does anyone have any ideas what the source might be?

@yeqcharlotte Added a simple repro for this issue here: yeqcharlotte/vllm#3

[yeqcharlotte]

discussed with @pavanimajety on slack, v3.0.1 passed this minimum repro. looking to understand which bug fix helped this issue and if they have good accuracy test coverage today.

[yzh119]

v3.0.1 pas this minimum repro

Hi @yeqcharlotte do you mean v0.3.1 can fix the regression? Can you also check v0.3.0?

I think #1611 fixes some mxfp4 quantization bugs and might be related.

[yeqcharlotte]

Hi @yeqcharlotte do you mean v0.3.1 can fix the regression? Can you also check v0.3.0?

I think #1611 fixes some mxfp4 quantization bugs and might be related.

It literally means yeqcharlotte/vllm#3 passed, which is a good indicator for our issue. I double checked v0.3.0 also passed on this example. https://gist.github.com/yeqcharlotte/741f7dda48a8c8dd650d015026f3a3e8

[yeqcharlotte]

@yzh119 gpt-oss only doesn't do fp4 for attention. could that be something else that's related?

  "quantization_config": {
    "modules_to_not_convert": [
      "model.layers.*.self_attn",
      "model.layers.*.mlp.router",
      "model.embed_tokens",
      "lm_head"
    ],
    "quant_method": "mxfp4"
  },

[pavanimajety]

The difference between the two tests only was in the Prefill and decode attention kernels with FLASHINFER_MOE_MXFP4_BF16. We may have to narrow down attention related changes.

[jwfromm]

While we're still evaluating, it appears that patching in #1584 resolves the issue.

[jwfromm]

While we're still evaluating, it appears that patching in #1584 resolves the issue.

Further testing revealed this was probably not the complete issue. We see better accuracy with Flashinfer v3.1.0 than 2.9.0 + this change, though it seemed to help a bit in some cases.