[Bugfix] Fix quark fp8 format loading on AMD GPUs

[fxmarty-amd]

#10765 was merged into vllm to support Quark quantized models. Unfortunately, tests are somewhat limited and the issue fixed in this PR was not catched by the tests. Namely, if you compare

vllm/vllm/model_executor/layers/quantization/fp8.py

Lines 303 to 317 in 7a8987d

	if current_platform.is_rocm():
	weight, weight_scale, input_scale = \
	normalize_e4m3fn_to_e4m3fnuz(
	weight=weight,
	weight_scale=weight_scale,
	input_scale=layer.input_scale)
	if input_scale is not None:
	layer.input_scale = Parameter(input_scale,
	requires_grad=False)
	weight_scale, weight = requantize_with_max_scale(
	weight=weight,
	weight_scale=weight_scale,
	logical_widths=layer.logical_widths,
	)

and

vllm/vllm/model_executor/layers/quantization/quark/schemes/quark_w8a8_fp8.py

Lines 34 to 51 in 7a8987d

	if self.qscheme == "per_tensor":
	max_w_scale, weight = requantize_with_max_scale(
	weight=layer.weight,
	weight_scale=layer.weight_scale,
	logical_widths=layer.logical_widths,
	)
	if current_platform.is_rocm():
	weight, max_w_scale, input_scale = normalize_e4m3fn_to_e4m3fnuz(
	weight=weight,
	weight_scale=max_w_scale,
	input_scale=layer.input_scale)
	if input_scale is not None:
	layer.input_scale = Parameter(input_scale,
	requires_grad=False)
	layer.weight = Parameter(weight.t(), requires_grad=False)
	layer.weight_scale = Parameter(max_w_scale, requires_grad=False)

We see that the call order of normalize_e4m3fn_to_e4m3fnuz and requantize_with_max_scale is swapped for quark. This eventually loads in weights being different in case a checkpoint uses "quant_method": "quark" and is loaded in vllm.

We might want to add more tests or extend https://github.com/vllm-project/vllm/blob/main/tests/quantization/test_quark.py in this PR or in a later PR.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[BowenBao]

@fxmarty-amd let's update this PR with fix we discussed internally.

[kewang-xlnx]

ok

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mgoin]

LGTM!

[fxmarty-amd]

@mgoin is there anything I should do about the red CI? Some failing tests have been due to timeout to hf.co on a few commits

[mgoin]

Not much to do there, we are landing PRs to try to reduce flakiness with HF. Let me merge with main

[mgoin]

@fxmarty-amd There is an error related to your parameters in the quantization test

[2025-02-10T23:36:19Z] FAILED quantization/test_quark.py::test_quark_fp8_generate - RuntimeError: Creating a Parameter from an instance of type PerTensorScaleParameter requires that detach() returns an instance of the same type, but return type Tensor was found instead. To use the type as a Parameter, please correct the detach() semantics defined by its __torch_dispatch__() implementation.
[2025-02-10T23:36:19Z] FAILED quantization/test_quark.py::test_quark_fp8_parity - RuntimeError: Creating a Parameter from an instance of type PerTensorScaleParameter requires that detach() returns an instance of the same type, but return type Tensor was found instead. To use the type as a Parameter, please correct the detach() semantics defined by its __torch_dispatch__() implementation.

[kewang-xlnx]

@mgoin @robertgshaw2-redhat Could you please take a look at this error https://buildkite.com/vllm/ci/builds/19546#0196adcf-4658-4117-9f60-9faa0fb1fc90, and give some advice to fix it, thanks.

[DarkLight1337]

It is also failing on main, you can ignore it. I can force merge once I am back on PC