Add Benchmarking and Fine-Tuning Support for ZenFlow

[Antlera]

Description:

This PR introduces scripts for benchmarking and fine-tuning with ZenFlow:

• zf_benchmark.py: Benchmark script for evaluating offloading performance (adapted from offload_states.py by @tohtana ).
• output_table.py: Parses and summarizes benchmark logs.
• run_benchmark.sh: Automates benchmark runs with configurable parameters.
• finetune_llama.py: Fine-tuning script for Llama-2 with DeepSpeed + ZenFlow.
• finetune_llama.sh: Launch script for fine-tuning with environment setup.
• zf_config.json: Example DeepSpeed config with ZenFlow optimizations.

Note: This PR is complimentary to PR #7391 on the main repo, and should be merged with (or after) merging PR #7391.

[Antlera]

Hi, @tohtana @tjruwase. Could you help review this PR when you have time? This is the example PR for PR #7391. Thanks!

[Antlera]

@sfc-gh-truwase Thanks for the great suggestions — I’ve applied them all!

[delock]

Hi @Antlera I have a question. I saw ZenFlow has CPU running parameter update. Does DeepSpeed argument --bind_cores_to_rank helping with CPU optimizer efficiency? If true then maybe this switch could be add to launch script.

Here is a link to this switch. This is a switch for CPU backend, but for CPU offload this switch should help as well.
https://github.com/deepspeedai/DeepSpeed/blob/master/docs/_tutorials/accelerator-setup-guide.md#how-to-launch-deepspeed-on-intel-architecture-cpu

[Antlera]

Hi @delock. Thank you for bringing this up—this is a great observation! I just tested --bind_cores_to_rank on an Intel Xeon Gold 6330 (112 threads) with 2 × L4 GPUs and did see a tangible benefit—about 60–80 ms per step in the CPU-optimizer phase when I bound cores as 0-53,54-107. The key, however, is leaving enough cores free for background services; if too few are reserved(e.g. only 1 left for background application), the gains quickly flatten or even reverse.

ZenFlow currently sidesteps most contention by evenly sharding os.cpu_count() cores across ranks, but a smarter policy that explicitly reserves background cores—and perhaps factors in NUMA locality—could push things further. I can open an issue to track this as a potential enhancement, and would welcome any best-practice core-binding patterns you’ve found effective.

I’ve shared more detailed discussion and logs in the email thread that Tunji forwarded—please let me know if you didn’t receive them.

[delock]

Hi @delock. Thank you for bringing this up—this is a great observation! I just tested --bind_cores_to_rank on an Intel Xeon Gold 6330 (112 threads) with 2 × L4 GPUs and did see a tangible benefit—about 60–80 ms per step in the CPU-optimizer phase when I bound cores as 0-53,54-107. The key, however, is leaving enough cores free for background services; if too few are reserved(e.g. only 1 left for background application), the gains quickly flatten or even reverse.

ZenFlow currently sidesteps most contention by evenly sharding os.cpu_count() cores across ranks, but a smarter policy that explicitly reserves background cores—and perhaps factors in NUMA locality—could push things further. I can open an issue to track this as a potential enhancement, and would welcome any best-practice core-binding patterns you’ve found effective.

I’ve shared more detailed discussion and logs in the email thread that Tunji forwarded—please let me know if you didn’t receive them.

Yes, please open an issue, we could discuss this in detail. Yes, leaving one might not be enough if background service needs more cores. We can discuss how to make core binding tuning more easy to use.