[Fix] Adjust use_aclgraph logic #2156
Conversation
Signed-off-by: Yizhou Liu <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:
If CI fails, you can run linting and testing checks locally according Contributing and Testing. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2156 +/- ##
==========================================
- Coverage 76.67% 76.67% -0.01%
==========================================
Files 107 107
Lines 11968 11972 +4
==========================================
+ Hits 9177 9180 +3
- Misses 2791 2792 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| @@ -33,7 +34,15 @@ def unquantized_fused_moe_init_func(self, *args, **kwargs): | |||
| original_unquantized_fused_moe_init_func(self, *args, **kwargs) | |||
| vllm_config = get_current_vllm_config() | |||
| self.max_num_batched_tokens = vllm_config.scheduler_config.max_num_batched_tokens | |||
| self.use_aclgraph = vllm_config.compilation_config.level == CompilationLevel.PIECEWISE and not vllm_config.model_config.enforce_eager | |||
|
|
|||
| ascend_config = get_ascend_config() | |||
There was a problem hiding this comment.
I suggest referring to #1841 and adding a new class called AscendUnquantizedFusedMoEMethod, incorporating the ACLGraph switch into this class's __init__ method, while also placing the new forward_oot method within this class.
There was a problem hiding this comment.
@ApsarasX it's on the todo list. we'll first combine common_fused_moe and fused_moe into one. Then refactor to Custom OP register way.
There was a problem hiding this comment.
@ApsarasX it's on the todo list. we'll first combine common_fused_moe and fused_moe into one. Then refactor to Custom OP register way.
OK
|
let's merge this first to fix the known bug. For the next step, we should refactor the fused moe totally. |
What this PR does / why we need it?
Updates the FusedMoE method to determine whether to use ACL Graph based on the
torchair_graph_configThis is equivalent to #2154 on v0.9.1-dev.
Does this PR introduce any user-facing change?
None.
How was this patch tested?
None needed.