[Feature]: support for fp8 marlin with MoE

[ehartford]

🚀 The feature, motivation and pitch

I wanna run Qwen3-235B-A22B on Ampere (A100) in fp8.

I quantized it to w8a16 using llm-compressor

https://huggingface.co/cognitivecomputations/Qwen3-235B-A22B-FP8-W8A16

but when I run it, I get the error

ERROR 05-02 03:16:53 [multiproc_executor.py:435] AssertionError: float16 is required for MoE compressed models. Set dtype=torch.float16

Please support FP8 with MoE in Marlin kernel

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[zhangxiaoxing]

I guess the following error is also related? It also happened on running FP8 MoE modal with Ampere cards. Qwen3 on FP8 is nice to have, and llama4 on FP8 could be even more interesting due to its potentially larger context length.

(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] triton.compiler.errors.CompilationError: at 1:0:
(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] def _per_token_group_quant_fp8(
(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] ^                                                                                                                

(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")
(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470]                                                                                                                  

[sparkprime]

Is there any workaround for running 235b on a100? A different quantization maybe?

[ehartford]

AWQ should work theoretically

I haven't seen anyone quant this yet

Personally I'm working on it too.

But that also is failing for me.

vllm-project/llm-compressor#1406

[ehartford]

@mgoin Would a $1,000 bounty help prioritize MoE support for Marlin?

[ehartford]

Is there any workaround for running 235b on a100? A different quantization maybe?

235b runs in bf16 on 8x A100.
Not much room for kv cache and context though.
fp8 marlin would be 🤌
I'm sure nvidia would be saddened by the capabilities unlocked in their previous generation hardware.

[mgoin]

fp8 moe marlin already exists! we also support nvfp4 marlin nowadays #19990

[ehartford]

Oh awesome! I'll try it today!

[ehartford]

fp8 moe marlin already exists! we also support nvfp4 marlin nowadays #19990

Hmmmm nvfp4, interesting! Now I'm curious to train in nvfp4. (maybe staged, a large dataset in nvfp4, a medium dataset in fp8, then a small dataset in bf16)