Skip to content

CB: Hetero pipeline parallel support #2227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 23, 2025
Merged

CB: Hetero pipeline parallel support #2227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0f3d812
CB: Hetero pipeline parallel support
WeldonWangwang May 19, 2025
7448463
Merge branch 'master' into wangwang/support_hetero_cb
Wovchena Jun 18, 2025
0549a0a
Update src/cpp/src/continuous_batching/pipeline_impl.cpp
WeldonWangwang Jun 19, 2025
578d78f
Add a judgment that execution_devices is empty
WeldonWangwang Jun 19, 2025
64619c2
Remove the judgment to empty execution_devices
WeldonWangwang Jun 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions src/cpp/src/continuous_batching/cache_manager.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,18 @@ class CacheManager {
// extract information about inference device
ov::CompiledModel compiled_model = request.get_compiled_model();
std::vector<std::string> execution_devices = compiled_model.get_property(ov::execution_devices);
OPENVINO_ASSERT(execution_devices.size() == 1, "Contituous batching: execution device is expected to be CPU or GPU, but got ", execution_devices.size(), " devices");
const bool all_gpu_device =
std::all_of(execution_devices.begin(), execution_devices.end(), [&](const std::string& device) {
return device.find("GPU") != std::string::npos;
});
OPENVINO_ASSERT(all_gpu_device || execution_devices.size() == 1,
Copy link
Preview

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above, all_gpu_device will be true for an empty vector. Ensure execution_devices is not empty before using element 0 or combine this into the assertion.

Copilot uses AI. Check for mistakes.

"Continuous batching: execution device is expected to be single CPU / single GPU / multi GPUs");
m_device = execution_devices[0];

// set block_size depending on device
const size_t cpu_block_size = 32, gpu_block_size = 16;
const bool is_gpu = m_device.find("GPU") != std::string::npos;
m_block_size = is_gpu ? gpu_block_size : cpu_block_size;
m_block_size = all_gpu_device ? gpu_block_size : cpu_block_size;

if (is_gpu) {
if (all_gpu_device) {
m_context = m_request.get_compiled_model().get_context();
}
// extract information about KV cache precisions and shapes
Expand Down
7 changes: 6 additions & 1 deletion src/cpp/src/continuous_batching/pipeline_impl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,12 @@ void ContinuousBatchingPipeline::ContinuousBatchingImpl::initialize_pipeline(

ov::CompiledModel compiled_model = utils::singleton_core().compile_model(model, device, *filtered_properties);
std::vector<std::string> execution_devices = compiled_model.get_property(ov::execution_devices);
OPENVINO_ASSERT(execution_devices.size() == 1, "Contituous batching: execution device is expected to be CPU or GPU, but got ", execution_devices.size(), " devices");
const bool all_gpu_device =
std::all_of(execution_devices.begin(), execution_devices.end(), [&](const std::string& device) {
return device.find("GPU") != std::string::npos;
});
OPENVINO_ASSERT(all_gpu_device || execution_devices.size() == 1,
Copy link
Preview

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assertion allows empty execution_devices (since all_of on an empty vector is true). Add a check to ensure execution_devices is non-empty before accessing index 0.

Copilot uses AI. Check for mistakes.

"Continuous batching: execution device is expected to be single CPU / single GPU / multi GPUs");
const std::string execution_device = execution_devices[0];

ov::genai::utils::print_compiled_model_properties(compiled_model, "LLM with Paged Attention");
Expand Down