Fix GLM-4 PP Missing Layer When using with PP. #21531
Conversation
Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
The pull request addresses an issue with pipeline parallelism by handling PPMissingLayer when iterating through model layers. However, the changes introduce a potential RuntimeError on pipeline parallel ranks that do not contain any MoE layers. The review includes suggestions to improve the logic and prevent the error from being raised unnecessarily.
| if example_moe is None: | ||
| raise RuntimeError("No Glm4MoE layer found in model.layers.") |
There was a problem hiding this comment.
This RuntimeError will be raised on pipeline parallel ranks that do not have any MoE layers. This will cause the model initialization to fail. Ensure that this error is only raised when there are supposed to be MoE layers, but none are found.
| example_moe = None | ||
| for layer in self.model.layers: | ||
| if isinstance(layer, PPMissingLayer): | ||
| continue |
There was a problem hiding this comment.
Consider skipping the loop entirely if self.model.layers is empty to avoid unnecessary iterations and potential errors. This can happen in pipeline parallel settings where some ranks might not have any layers.
| example_moe = None | |
| for layer in self.model.layers: | |
| if isinstance(layer, PPMissingLayer): | |
| continue | |
| if self.model.layers: | |
| for layer in self.model.layers: | |
| if isinstance(layer, PPMissingLayer): | |
| continue |
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]> Signed-off-by: shuw <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]> Signed-off-by: x22x22 <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
Cherry-pick: vllm-project@2ce90e5 Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]> Signed-off-by: Jinzhen Lin <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]> Signed-off-by: Paul Pak <[email protected]>
Cherry-pick: vllm-project@2ce90e5 Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]> Signed-off-by: Boyuan Feng <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]> Signed-off-by: Diego-Castan <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
Signed-off-by: zRzRzRzRzRzRzR <[email protected]>
No description provided.