[NPU Roadmap] NPU support for Liger-Kernel

[zheliuyu]

Thanks very much to Liger-Kernel for accepting our first native support pr.
This roadmap details the future plans for NPU native support. Welcome to join in the discussion.

NPU Native Support

• [RFC] Native Ascend NPU Support for Liger Kernel #954
• Add Ascend NPU device support. #955
• [RFC][WIP] Code Organization for New Backend Support #965

Unit Test Coverage Improvement: Functionality & Precision

• chunked_loss
• convergence
• transformers
  • test_auto_model.py PASSED: 100%
  • test_cross_entropy.py PASSED: 6%
  • test_dyt.py PASSED: 96%
  • test_embedding.py PASSED: 0%
  • test_flex_attention.py PASSED: 0%
  • test_fused_add_rms_norm.py PASSED: 0%
  • test_fused_linear_cross_entropy.py PASSED: 86%
  • test_fused_linear_jsd.py PASSED: 66%
  • test_fused_neighborhood_attention.py PASSED: 96%
  • test_geglu.py PASSED: 100%
  • test_group_norm.py PASSED: 0%
  • test_grpo_loss.py PASSED: 85%
  • test_jsd.py PASSED: 49%
  • test_kl_div.py PASSED: 49%
  • test_layer_norm.py PASSED: 0%
  • test_mm_int8int2.py PASSED: 0%
  • test_monkey_patch.py PASSED: 100%
  • test_multi_token_attention.py PASSED: 97%
  • test_poly_norm.py PASSED: 23%
  • test_qwen2vl_mrope.py PASSED: 0%
  • test_rms_norm.py PASSED: 0%
  • test_rope.py PASSED: 77%
  • test_softmax.py PASSED: 100%
  • test_sparsemax.py PASSED: 100%
  • test_swiglu.py PASSED: 100%
  • test_tiled_mlp.py PASSED: 75%
  • test_transformers.py PASSED: 100%
  • test_tvd.py PASSED: 18%
• triton

Performance Enhancement For High-Frequency APIs

Unit tests can be used to APIs functionality and precision. However, as third-party devices may not fully align in their usage patterns of Triton, a performance optimization process is required.

We selected three training software with varying dependencies on Liger-Kernel APIs. We will subsequently measure the actual performance of these APIs and may initiate pull requests to provide native support.

Software	Utils	APIs	Status
VeOmni	liger_kernel.transformers import	LigerFusedLinearCrossEntropyLoss(reduction="mean")	DOING, @zheliuyu
	liger_kernel.transformers.rms_norm import	LigerRMSNorm	DOING, @ji-huazhong
	liger_kernel.transformers.rope import	liger_rotary_pos_emb
	liger_kernel.transformers.swiglu import	LigerSwiGLUMLP	DOING, @jiaqiw09
	liger_kernel.transformers.layer_norm import	LigerLayerNorm
	liger_kernel.transformers.qwen2vl_mrope import	liger_multimodal_rotary_pos_emb
	liger_kernel.ops.swiglu import	LigerSiLUMulFunction
LLaMA-Factory	liger_kernel.transformers import	apply_liger_kernel_to_gemma
	liger_kernel.transformers import	apply_liger_kernel_to_gemma2
	liger_kernel.transformers import	apply_liger_kernel_to_gemma3
	liger_kernel.transformers import	apply_liger_kernel_to_gemma3_text
	liger_kernel.transformers import	apply_liger_kernel_to_glm4
	liger_kernel.transformers import	apply_liger_kernel_to_glm4v
	liger_kernel.transformers import	apply_liger_kernel_to_granite
	liger_kernel.transformers import	apply_liger_kernel_to_llama
	liger_kernel.transformers import	apply_liger_kernel_to_llava
	liger_kernel.transformers import	apply_liger_kernel_to_mistral
	liger_kernel.transformers import	apply_liger_kernel_to_mixtral
	liger_kernel.transformers import	apply_liger_kernel_to_mllama
	liger_kernel.transformers import	apply_liger_kernel_to_olmo2
	liger_kernel.transformers import	apply_liger_kernel_to_paligemma
	liger_kernel.transformers import	apply_liger_kernel_to_phi3
	liger_kernel.transformers import	apply_liger_kernel_to_qwen2
	liger_kernel.transformers import	apply_liger_kernel_to_qwen2_vl
	liger_kernel.transformers import	apply_liger_kernel_to_qwen2_5_vl
	liger_kernel.transformers import	apply_liger_kernel_to_qwen3
	liger_kernel.transformers import	apply_liger_kernel_to_qwen3_moe
verl	liger_kernel.transformers.monkey_patch import	_apply_liger_kernel_to_instance(model=actor_module)

[Tcc0403]

Is there any repo with npu ci where contributors can test around?

[Ginray]

Is there any repo with npu ci where contributors can test around?

We’ve built NPU CI for LlamaFactory, swift and verl, and will verify liger kernel’s compatibility with these libraries in follow-up work. Additionally, we hope to set up NPU CI for the liger kernel library, and can provide NPU machines to support this effort. We’d like to connect with you to move this forward.

[momochen]

@Ginray hey there, we are excited to see community supports to NPU, aligned with our long term goal to support heterogenous accelerators. Let's setup a formal meeting sometime this week on making NPU CI available. @Tcc0403 let's find a mutually convenient time.

[zheliuyu]

Some new players have joined the game. ✨

• VeOmini
  • LigerRMSNorm @ji-huazhong
  • LigerSwiGLUMLP @jiaqiw09

[Tcc0403]

let's find a mutually convenient time.

@momochen Any time is fine. I can work around it.