[Bug]: Port binding failure when using pp > 1 after commit 7c7714d856eee6fa94aade729b67f00584f72a4c

[dengminhao]

Your current environment

The output of `python collect_env.py`

PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux 7 (Core) (x86_64)
GCC version: (conda-forge gcc 11.4.0-13) 11.4.0
Clang version: Could not collect
CMake version: version 2.8.12.2
Libc version: glibc-2.17

Python version: 3.11.9 | packaged by conda-forge | (main, Apr 19 2024, 18:36:13) [GCC 12.3.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.95.1.el7.x86_64-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce RTX 3090
GPU 1: NVIDIA GeForce RTX 3090
GPU 2: NVIDIA GeForce RTX 3090
GPU 3: NVIDIA GeForce RTX 3090
GPU 4: NVIDIA GeForce RTX 3090
GPU 5: NVIDIA GeForce RTX 3090

Nvidia driver version: 535.104.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    2
Core(s) per socket:    32
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 106
Model name:            Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz
Stepping:              6
CPU MHz:               800.000
CPU max MHz:           2301.0000
CPU min MHz:           800.0000
BogoMIPS:              4600.00
Virtualization:        VT-x
L1d cache:             48K
L1i cache:             32K
L2 cache:              1280K
L3 cache:              55296K
NUMA node0 CPU(s):     0-31,64-95
NUMA node1 CPU(s):     32-63,96-127
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single ssbd mba rsb_ctxsw ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear pconfig spec_ctrl intel_stibp flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] flashinfer==0.1.6+cu121torch2.4
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.20
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.0.dev0
[pip3] triton==3.0.0
[pip3] zmq==0.0.0
[conda] flashinfer                0.1.6+cu121torch2.4          pypi_0    pypi
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.1.3.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.1.105                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.1.105                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.0.2.54                pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.2.106               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.4.5.107               pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.1.0.106               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.20.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.6.20                  pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.1.105                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] torchvision               0.19.0                   pypi_0    pypi
[conda] transformers              4.45.0.dev0              pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
[conda] zmq                       0.0.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.1.post2@0faab90eb006c677add65cd4c2d0f740a63e064d
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NODE	NODE	SYS	SYS	SYS	0-31,64-95	0		N/A
GPU1	NODE	 X 	PXB	SYS	SYS	SYS	0-31,64-95	0		N/A
GPU2	NODE	PXB	 X 	SYS	SYS	SYS	0-31,64-95	0		N/A
GPU3	SYS	SYS	SYS	 X 	NODE	NODE	32-63,96-127	1		N/A
GPU4	SYS	SYS	SYS	NODE	 X 	PXB	32-63,96-127	1		N/A
GPU5	SYS	SYS	SYS	NODE	PXB	 X 	32-63,96-127	1		N/A

Model Input Dumps

No response

🐛 Describe the bug

After a binary search, I found that after commit 7c7714d, the main port binding will fail when pp> 1. But if we only set tp>1, the binding will success.

For example:
vllm serve /home/ai/ai/model/Qwen2.5-3B-Instruct/ --served-model-name qwen2.5-3B -pp 2 --trust-remote-code --max-model-len 4096 --enforce-eager --port 18004 --gpu-memory-utilization 1 --preemption-mode swap
will fail with ERROR:
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 18004): address already in use

But
vllm serve /home/ai/ai/model/Qwen2.5-3B-Instruct/ --served-model-name qwen2.5-3B -tp 2 --trust-remote-code --max-model-len 4096 --enforce-eager --port 18004 --gpu-memory-utilization 1 --preemption-mode swap
will run successfully with:
INFO: Uvicorn running on http://0.0.0.0:18004 (Press CTRL+C to quit)

If we checkout to commit 9d104b5 in main branch, we can launch successfully with pp>1

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[SolitaryThinker]

cc @youkaichao @kevin314

[youkaichao]

looks strange. @robertgshaw2-neuralmagic do you have any ideas?

@dengminhao do you try --disable-frontend-multiprocessing ?

[dengminhao]

Nothing changed after --disable-frontend-multiprocessing.
I added code in uvicorn so I am sure there is only one process try to bind the main port.
Also I copied the code in util which trys to find any process binding this port, and still I cannot find any process holds the main port after binding failure

[dengminhao]

I aslo tried reuse_port=True when calling loop.create_server in uvicorn/server.py. It won't help.

[dengminhao]

Oh, another finding.
Adding "--disable-frontend-multiprocessing" will make tp=2 fails to bind also.
Remove "--disable-frontend-multiprocessing", tp=2 can run successfully.

[dengminhao]

After this try, I have an idea and then find a quick fix
Hope it is helpful to you. It seems still a multi-process resource issue.

diff --git a/vllm/entrypoints/openai/api_server.py b/vllm/entrypoints/openai/api_server.py
index 1b9eb30..8512b9a 100644
--- a/vllm/entrypoints/openai/api_server.py
+++ b/vllm/entrypoints/openai/api_server.py
@@ -530,6 +530,7 @@ async def run_server(args, **uvicorn_kwargs) -> None:
         raise KeyboardInterrupt("terminated")
 
     signal.signal(signal.SIGTERM, signal_handler)
+    temp_socket.close()
 
     async with build_async_engine_client(args) as engine_client:
         # If None, creation of the client failed and we exit.
@@ -541,7 +542,7 @@ async def run_server(args, **uvicorn_kwargs) -> None:
         model_config = await engine_client.get_model_config()
         init_app_state(engine_client, model_config, app.state, args)
 
-        temp_socket.close()
 
         shutdown_task = await serve_http(
             app,

[youkaichao]

we use the socket to hold the port, so that engine does not take this port. your change will essentially disable the functionality.

It seems still a multi-process resource issue

did you try to set export VLLM_WORKER_MULTIPROC_METHOD=spawn or use ray backend?

[dengminhao]

I know what you expect, so I said it was a quick fix. Roll back my change, and then:
export VLLM_WORKER_MULTIPROC_METHOD=spawn
works.
add --distributed-executor-backend=ray
also works.

[dengminhao]

I upgrade to 0.6.2, the bug still exists.
And of couse I can use VLLM_WORKER_MULTIPROC_METHOD=spawn or distributed-executor-backend=ray, but I believe it is still a bug.

[youkaichao]

is it because of port resource management after fork

[dengminhao]

I guess in my case ,the temp socket resource duplicated when fork. But I don't know what's the difference between tp and pp in this case.

[HelloCard]

same issue, use 0.6.2.
python3 -m vllm.entrypoints.openai.api_server --model /mnt/e/Code/models/Mistral-Nemo-Instruct-2407-W8A8 --max-model-len 8192 --gpu-memory-utilization 0.88 --swap_space=0 --pipeline-parallel-size 2 --dtype=half --max-num-seqs=1
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use
set --port 8001 then:
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8001): address already in use
export VLLM_WORKER_MULTIPROC_METHOD=spawn it worked, thanks.

[youkaichao]

@HelloCard can you test if #8537 solve this issue? you can follow https://docs.vllm.ai/en/latest/getting_started/installation.html#install-the-latest-code to install the latest wheel.