[Hardware][PPC64LE] Enable V1 for ppc64le and ARM #20554

Akashcodes732 · 2025-07-07T08:36:50Z

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

FIX #20622

Purpose

This PR enables V1 Engine for IBM POWER, as V0 Engine would be deprecated (#20437, #20412).

As chunked prefill is not supported on ppc64le, this PR turns off chunked prefill in case of ppc64le.

Also, the CPU binding logic is not optimized for POWER (The issue and its impact are described in detail in #20089. #20387 ), this PR adds that logic in case of V1 as well.

Test Plan

Local testing

Test Result

(Optional) Documentation Update

Signed-off-by: Akash Kaothalkar <[email protected]>

gemini-code-assist

Summary of Changes

Hello @Akashcodes732, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on extending VLLM's V1 engine support to the IBM POWER (ppc64le) architecture. It addresses platform-specific requirements by conditionally disabling chunked prefill and introducing tailored CPU binding optimizations to ensure efficient and compatible operation on POWER systems, in anticipation of the V0 engine's deprecation.

Highlights

V1 Engine Enablement for PPC64LE: This pull request primarily enables the V1 engine for IBM POWER (ppc64le) architecture, aligning with the ongoing deprecation of the V0 engine.
Conditional Chunked Prefill Disablement: It introduces logic to automatically disable chunked prefill when the V1 engine is used on POWER CPUs, as this feature is not supported on ppc64le.
Optimized CPU Binding for PPC64LE: A specialized CPU binding strategy is implemented for ppc64le systems. This new logic selects an optimal subset of threads per core to prevent oversubscription and enhance performance, robustly handling different SMT modes.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request enables V1 Engine support for the ppc64le (IBM POWER) architecture by disabling chunked prefill and adding optimized CPU binding logic. Consider refactoring the CPU binding logic in vllm/v1/worker/cpu_worker.py to reduce code duplication and improve maintainability.

gemini-code-assist · 2025-07-07T08:38:19Z

vllm/v1/worker/cpu_worker.py

+    def get_cpus_id_binding_based_on_numa_nodes_ppc64le(self) -> str:
+        """
+        Power (ppc64le) specific: Selects a subset of threads per core for 
+        each NUMA node.This is robust to SMT mode (SMT-8, SMT-4, etc) 
+        because the OS only exposes available threads.This maximizes 
+        performance by avoiding oversubscription of logical CPUs on Power.
+        """
+
+        def select_threads_per_power_core(node_cpu_ids):
+            return [cpu for cpu in node_cpu_ids if cpu % 8 < 4]
+
+        rank_to_cpus = self.local_omp_cpuid
+        world_size = self.vllm_config.parallel_config.world_size
+        libnuma_found = util.find_spec("numa") is not None
+        psutil_found = util.find_spec("psutil") is not None
+        if libnuma_found and psutil_found:
+            import psutil
+            from numa import info
+            cpus_allow_list = psutil.Process().cpu_affinity()
+            numa_size = info.get_num_configured_nodes()
+
+            node_to_cpus = []
+            for i in range(numa_size):
+                node_intersect = set(
+                    info.node_to_cpus(i)).intersection(cpus_allow_list)
+                if bool(node_intersect):
+                    node_to_cpus.append(sorted(list(node_intersect)))
+
+            if world_size > len(node_to_cpus):
+                logger.error(
+                    "Auto thread-binding failed due to "
+                    "world size: %d is larger than "
+                    "allowed NUMA nodes number: %d."
+                    "Please try to bind threads manually.", world_size,
+                    len(node_to_cpus))
+            else:
+                node_cpus_this_rank = node_to_cpus[self.rank]
+                node_cpus_this_rank = select_threads_per_power_core(
+                    node_cpus_this_rank)
+                cpu_count_per_numa = len(node_cpus_this_rank)
+                num_of_reserved_cpu = min(envs.VLLM_CPU_NUM_OF_RESERVED_CPU,
+                                          cpu_count_per_numa // 2)
+                end = cpu_count_per_numa - num_of_reserved_cpu
+                rank_to_cpus_list = node_cpus_this_rank[:end]
+                rank_to_cpus = ','.join(str(x) for x in rank_to_cpus_list)
+                logger.info("ppc64le thread-binding list: %s", rank_to_cpus)
+        else:
+            logger.warning(
+                "Auto thread-binding is not supported due to "
+                "the lack of package numa and psutil,"
+                "fallback to no thread-binding. To get better performance,"
+                "please try to manually bind threads.")
+        return rank_to_cpus


The function get_cpus_id_binding_based_on_numa_nodes_ppc64le largely duplicates the logic in get_cpus_id_binding_based_on_numa_nodes. Refactor the common logic into a single function and conditionally apply the architecture-specific logic to reduce code duplication and improve maintainability.

vllm/v1/worker/cpu_worker.py

github-actions · 2025-07-07T08:39:55Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: Akash Kaothalkar <[email protected]>

Akashcodes732 · 2025-07-08T08:01:56Z

Hi @WoosukKwon , @njhill , @comaniac , @alexm-redhat , @robertgshaw2-redhat , @ywang96 , can you please review this PR ?

vllm/engine/arg_utils.py

vllm/platforms/cpu.py

Co-authored-by: Nikhil Gupta <[email protected]> Signed-off-by: Akash Kaothalkar <[email protected]>

Signed-off-by: Akash Kaothalkar <[email protected]>

Akashcodes732 · 2025-07-09T02:45:27Z

Hi @DarkLight1337 , @bigPYJ1151 can someone please review this ?

DarkLight1337

LGTM, thanks for fixing!

Signed-off-by: Akash Kaothalkar <[email protected]> Co-authored-by: Akash Kaothalkar <[email protected]> Co-authored-by: Nikhil Gupta <[email protected]>

Signed-off-by: Akash Kaothalkar <[email protected]> Co-authored-by: Akash Kaothalkar <[email protected]> Co-authored-by: Nikhil Gupta <[email protected]> Signed-off-by: Patrick von Platen <[email protected]>

Signed-off-by: Akash Kaothalkar <[email protected]> Co-authored-by: Akash Kaothalkar <[email protected]> Co-authored-by: Nikhil Gupta <[email protected]>

22quinn · 2025-07-28T04:02:33Z

vllm/engine/arg_utils.py

+            # Disable chunked prefill for POWER (ppc64le)/ARM CPUs in V1
+            if current_platform.is_cpu(
+            ) and current_platform.get_cpu_architecture() in (
+                    CpuArchEnum.POWERPC, CpuArchEnum.ARM):
+                logger.info(
+                    "Chunked prefill is not supported for ARM and POWER CPUs; "
+                    "disabling it for V1 backend.")
+                self.enable_chunked_prefill = False


@Akashcodes732 Curious how the V1 scheduler supports no chunked prefill for POWER (ppc64le)/ARM CPUs? The chunking logic in scheduler cannot be turned off right?

Signed-off-by: Akash Kaothalkar <[email protected]> Co-authored-by: Akash Kaothalkar <[email protected]> Co-authored-by: Nikhil Gupta <[email protected]> Signed-off-by: avigny <[email protected]>

Signed-off-by: Akash Kaothalkar <[email protected]> Co-authored-by: Akash Kaothalkar <[email protected]> Co-authored-by: Nikhil Gupta <[email protected]>

Akash Kaothalkar added 2 commits July 7, 2025 03:04

feat: Disable Chunked prefill in v1 for ppc64le

5330156

Signed-off-by: Akash Kaothalkar <[email protected]>

chore: add comment

3bad7b4

Signed-off-by: Akash Kaothalkar <[email protected]>

Akashcodes732 requested review from WoosukKwon, robertgshaw2-redhat, njhill, ywang96, comaniac and alexm-redhat as code owners July 7, 2025 08:36

gemini-code-assist bot reviewed Jul 7, 2025

View reviewed changes

mergify bot added the v1 label Jul 7, 2025

gemini-code-assist bot reviewed Jul 7, 2025

View reviewed changes

Akash Kaothalkar added 2 commits July 7, 2025 04:02

fix: pre-commit issues

2b57899

Signed-off-by: Akash Kaothalkar <[email protected]>

chore: make v1 default for powerpc

e1c9bdd

Signed-off-by: Akash Kaothalkar <[email protected]>

DarkLight1337 mentioned this pull request Jul 8, 2025

[Bug]: Torch SDPA path broken on AArch64 due to default chunked_prefill in vLLM Engine V1 #20622

Closed

1 task

nikhil-arm reviewed Jul 8, 2025

View reviewed changes

vllm/engine/arg_utils.py Outdated Show resolved Hide resolved

nikhil-arm reviewed Jul 8, 2025

View reviewed changes

vllm/platforms/cpu.py Outdated Show resolved Hide resolved

Akashcodes732 and others added 2 commits July 8, 2025 06:43

ARM: Update vllm/platforms/cpu.py

d63183c

Co-authored-by: Nikhil Gupta <[email protected]> Signed-off-by: Akash Kaothalkar <[email protected]>

ARM: Update vllm/engine/arg_utils.py

de11286

Co-authored-by: Nikhil Gupta <[email protected]> Signed-off-by: Akash Kaothalkar <[email protected]>

Akashcodes732 force-pushed the feat/v1_enablement_ppc64le branch from 55e0687 to de11286 Compare July 8, 2025 11:47

chore: pre-commit and comments

5eff167

Signed-off-by: Akash Kaothalkar <[email protected]>

DarkLight1337 changed the title ~~[Hardware][PPC64LE] Enable V1 for ppc64le~~ [Hardware][PPC64LE] Enable V1 for ppc64le and ARM Jul 9, 2025

DarkLight1337 approved these changes Jul 9, 2025

View reviewed changes

vllm-bot merged commit 6db31e7 into vllm-project:main Jul 9, 2025
14 checks passed

22quinn reviewed Jul 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

[Hardware][PPC64LE] Enable V1 for ppc64le and ARM #20554

[Hardware][PPC64LE] Enable V1 for ppc64le and ARM #20554

Uh oh!

Akashcodes732 commented Jul 7, 2025 •

edited by github-actions bot

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jul 7, 2025

Uh oh!

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Akashcodes732 commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

Akashcodes732 commented Jul 9, 2025

Uh oh!

DarkLight1337 left a comment

Uh oh!

Uh oh!

22quinn Jul 28, 2025

Uh oh!

Uh oh!

Uh oh!

[Hardware][PPC64LE] Enable V1 for ppc64le and ARM #20554

[Hardware][PPC64LE] Enable V1 for ppc64le and ARM #20554

Uh oh!

Conversation

Akashcodes732 commented Jul 7, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jul 7, 2025

Uh oh!

Akashcodes732 commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

Akashcodes732 commented Jul 9, 2025

Uh oh!

DarkLight1337 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

22quinn Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Akashcodes732 commented Jul 7, 2025 •

edited by github-actions bot

Loading