[GPUNetIO] [ERR] kernel_progress: block 0 error CQE! poll_status -5 wqe 511 index 0

[foraxe]

1. Error msg when i tried to run GPUNetIO backend

Block Size (B) Batch Size B/W (GB/Sec) Avg Lat. (us) Avg Prep (us) P99 Prep (us) Avg Post (us) P99 Post (us) Avg Tx (us) P99 Tx (us) got completion with err: syndrome=0x5, vendor_err_synd=0xf9, hw_err_synd=0, hw_synd_type=0, wqe_counter=65281 wqe_qpn=ed020008 kernel_progress: block 0 error CQE! poll_status -5 wqe 511 index 0

2. Error occurs when running below commands:

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/gdrcopy/src:/opt/mellanox/doca NIXL_PLUGIN_DIR=/usr/local/nixl/lib/x86_64-linux-gnu/plugins CUDA_MODULE_LOADING=EAGER ./nixlbench --etcd-endpoints http://x.x.x.x:2379/ --backend=GPUNETIO --initiator_seg_type=VRAM --target_seg_type=DRAM --runtime_type=ETCD --gpunetio_device_list=1 --device_list=mlx5_0 --start_batch_size=512 --max_batch_size=512 --total_buffer_size=34359738368

3. Env & Settings

The running environment is built based on codes from main branch (25/10/27).
I am running on 8xH*** card server, with 4 ConnectX-7 net cards.

Anyone have ideas on solving the Error when running the GPUNetIO backend? Or ways to tackle it?

Notes: In former codes from pull/760 GPUNETIO backend returns 'Error 22 doca_gpu_dev_rdma_wait_all xfer' after printing the nixlbench settings.

[e-ago]

Is the mlx5_0 in IB mode? If yes then you need to use an extra parameter --gpunetio_oob_list=ethernet interface

[foraxe]

Thank you, I saw your pr pull/733.

The mlx5_0 is currently in eth mode, also mlx5_bond_0 (another dev environment). And i tried to use the gpunetio_oob_list=eth0 to run, the errors are still the same: [ERR] kernel_progress: block 0 error CQE! poll_status -5 wqe 511 index 0. For the mlx5_0 environment:

`$ ibstat mlx5_0

CA 'mlx5_0'
CA type: MT4126
Number of ports: 1
Firmware version: 32.39.3920
Hardware version: 1
Node GUID: xxxx
System image GUID: xxxx
Port 1:
State: Active
Physical state: LinkUp
Rate: 200
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: xxxxxxx
Link layer: Ethernet`

I can append the full error logs for both 2 nodes. The machines use BF3 net cards.
The tests runs on pods. The images are built by the benchmark/nixlbench/contrib/build.sh

[e-ago]

Thanks. Then if mlx5_0 is not IB, you don't need the extra gpunetio_oob_list parameter.
I assume you configured properly your machines for GPUNetIO: https://docs.nvidia.com/doca/sdk/doca+gpunetio/index.html

Can you try without using docker images? just building and running on bare metal.
I can share the meson/ninja command lines if you want.

If yes, you may also want to enable the extra debug info setting the macro DOCA_GPUNETIO_VERBS_ENABLE_DEBUG to 1 in the /opt/mellanox/doca/include/doca_gpunetio_verbs_def.h header file?

[foraxe]

Hi, I had configured the machine according to https://docs.nvidia.com/doca/sdk/doca+gpunetio/index.html

I am tring to add new DOCA on host machine.

Meanwhile, should I rebuild the nixl after setting DOCA_GPUNETIO_VERBS_ENABLE_DEBUG?
I tried to set it inside docker image, and i see no debug info from doca (even after rebuilding nixl). And I run below command to rebuild:

cd /workspace/newnixl1027/nixl
    mkdir -p build
meson setup --reconfigure build/ --prefix=/usr/local/nixl --buildtype=debug && \
    cd build
ninja && ninja install

cd /workspace/newnixl1027/nixl/benchmark/nixlbench
mkdir -p build && \
meson setup --reconfigure build/ -Dnixl_path=/usr/local/nixl/ -Dprefix=/usr/local/nixlbench --buildtype=debug && \
    cd build && ninja && ninja install

The nixlbench cmd for two nodes is:

LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/opt/gdrcopy/src:/opt/mellanox/doca NIXL_PLUGIN_DIR=/usr/local/nixl/lib/x86_64-linux-gnu/plugins CUDA_MODULE_LOADING=EAGER ./nixlbench --etcd-endpoints http://x.x.x.x:2379 --backend=GPUNETIO --initiator_seg_type=VRAM --target_seg_type=DRAM --runtime_type=ETCD --gpunetio_device_list=0 --device_list=mlx5_0 --start_batch_size=1 --max_batch_size=1 --total_buffer_size=131072  --max_block_size 16384

[foraxe]

Meanwhile, I just noticed that "Force reinstall of RDMA packages from DOCA repository" in the benchmark/nixlbench/contrib/build.sh . Does the RDMA packages might be an issue?

My current image is built from version: https://github.com/ai-dynamo/nixl/blob/3978a64fe7d765fb002b478a8805a38e4b54afa4/benchmark/nixlbench/contrib/Dockerfile

[foraxe]

Hi @e-ago ,
I’ve made some progress on the setup:

1. I modified the code so that the GPUNetIO backend can run in the mlx5_bond_0 environment. However, when running across two nodes, it still reports the Error CQE issue.

2. We’re not able to modify the host machines directly due to stability and configuration constraints.

3. The GPUNetIO samples themselves can run successfully between the two nodes, so the environment setup should be correct.

Do you happen to have a known-good test environment (e.g., AWS or other servers) where we could try this out?
Alternatively, we could share our Docker images with you for reproduction if that helps.

Thanks for the continued guidance.