[feat/multimodal] support for Qwen2-VL

[hiyouga]

🐛 Describe the bug

Example:

modeling_qwen2_vl.apply_rotary_pos_emb = liger_rotary_pos_emb
modeling_qwen2_vl.Qwen2RMSNorm = LigerRMSNorm
modeling_qwen2_vl.CrossEntropyLoss = LigerCrossEntropyLoss
modeling_qwen2_vl.Qwen2VLForConditionalGeneration.forward = qwen2_vl_lce_forward
modeling_qwen2_vl.Qwen2MLP = LigerSwiGLUMLP

The qwen2_vl_lce_forward needs to be implemented to adapt to the image inputs

Reproduce

Na

Versions

Environment Report:

Python version: 3.10.13
PyTorch version: 2.2.0+cu118
CUDA version: 11.8
Triton: Not installed
Transformers version: 4.45.0.dev0

[tyler-romero]

I'm interested but probably wont be able to get to it until next week. Would also be cool to patch parts of the ViT, like the LayerNorms, with LigerKernel as well: https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L263

Although improving the ViT may not move the needle much compared to the LLM.

[tyler-romero]

#take @ByronHsu

[tyler-romero]

@ByronHsu @lancerts this can be closed now!