多机多卡分布式部署DeeSeek-671B满血模型，使用IB网络通信时异常。
环境：
四节点海光KA-100，镜像vllm-072；
kubernetes启动四个Pod，组成Ray集群，然后启动vllm推理服务。

Ray集群状态：

Pod节点间IB通信：

vLLM服务启动报错：

(RayWorkerWrapper pid=567, ip=10.244.2.219) inference-deepseek-rjmvpa-head-0-model-854647bdfb-259lc:567:567 [1] NCCL INFO Channel 03/0 : 2[a000] -> 3[a000] [send] via NET/IB/0/GDRDMA comm 0x5611e5b88f50 nRanks 04
(RayWorkerWrapper pid=567, ip=10.244.2.219)
(RayWorkerWrapper pid=567, ip=10.244.2.219) inference-deepseek-rjmvpa-head-0-model-854647bdfb-259lc:567:8751 [1] /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/misc/ibvwrap.cc:150 NCCL WARN Call to ibv_reg_mr_iova2 failed with error Invalid argument
(RayWorkerWrapper pid=567, ip=10.244.2.219) inference-deepseek-rjmvpa-head-0-model-854647bdfb-259lc:567:8751 [1] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/transport/net.cc:85ERROR 03-11 19:30:41 worker_base.py:609] Error executing method 'init_device'. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] Traceback (most recent call last):
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 601, in execute_method
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] return run_method(target, method, args, kwargs)
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/utils.py", line 2313, in run_method
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] return func(*args, **kwargs)
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 166, in init_device
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] init_worker_distributed_environment(self.vllm_config, self.rank,
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 516, in init_worker_distributed_environment
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1103, in ensure_model_parallel_initialized
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] initialize_model_parallel(tensor_model_parallel_size,
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1064, in initialize_model_parallel
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] _PP = init_model_parallel_group(group_ranks,
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 876, in init_model_parallel_group
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] return GroupCoordinator(
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 218, in __init__
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] self.pynccl_comm = PyNcclCommunicator(
(RayWorkerWrapper pid=560, ip=10.244.2.219) *** SIGSEGV received at time=1741692641 on cpu 71 ***
(RayWorkerWrapper pid=560, ip=10.244.2.219) PC: @ 0x7f5dd2a7db14 (unknown) ncclProxyService()
(RayWorkerWrapper pid=560, ip=10.244.2.219) @ 0x7f7a123ef520 (unknown) (unknown)
(RayWorkerWrapper pid=560, ip=10.244.2.219) [2025-03-11 19:30:41,471 E 560 8753] logging.cc:460: *** SIGSEGV received at time=1741692641 on cpu 71 ***
(RayWorkerWrapper pid=560, ip=10.244.2.219) [2025-03-11 19:30:41,471 E 560 8753] logging.cc:460: PC: @ 0x7f5dd2a7db14 (unknown) ncclProxyService()
(RayWorkerWrapper pid=560, ip=10.244.2.219) [2025-03-11 19:30:41,472 E 560 8753] logging.cc:460: @ 0x7f7a123ef520 (unknown) (unknown)
(RayWorkerWrapper pid=560, ip=10.244.2.219) Fatal Python error: Segmentation fault
(RayWorkerWrapper pid=560, ip=10.244.2.219)
(RayWorkerWrapper pid=560, ip=10.244.2.219)
(RayWorkerWrapper pid=560, ip=10.244.2.219) Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, markupsafe._speedups, PIL._imaging, msgspec._core, PIL._imagingft, sentencepiece._sentencepiece, zmq.backend.cython._zmq, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, pyarrow.lib, pyarrow._json, hiredis.hiredis (total: 48)
(RayWorkerWrapper pid=567, ip=10.244.2.219)
(RayWorkerWrapper pid=567, ip=10.244.2.219)
(RayWorkerWrapper pid=574, ip=10.244.2.219)
(RayWorkerWrapper pid=574, ip=10.244.2.219)
c485bf-8fhgg:7:7 [0] /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/init.cc:1299 NCCL WARN -hcugon- rank 0 localRanks 1 nRanks 4 invalid rank num
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 00/04 : 0 1 2 3
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 01/04 : 0 1 2 3
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 02/04 : 0 1 2 3
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 03/04 : 0 1 2 3
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Trees [0] 2/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 2/-1/-1->0->-1 [3] -1/-1/-1->0->1 comm 0x55ca738d7b10 nRanks 04 busId 7000
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO -hcugon- create sdma group queue rank:0 localRanks:1 nRanks:4 sdma copy is not enabled
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO P2P Chunksize set to 131072
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 00/0 : 3[7000] -> 0[7000] [receive] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 01/0 : 3[7000] -> 0[7000] [receive] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 02/0 : 3[7000] -> 0[7000] [receive] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 03/0 : 3[7000] -> 0[7000] [receive] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 00/0 : 0[7000] -> 1[7000] [send] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 01/0 : 0[7000] -> 1[7000] [send] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 02/0 : 0[7000] -> 1[7000] [send] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO Channel 03/0 : 0[7000] -> 1[7000] [send] via NET/IB/0/GDRDMA comm 0x55ca738d7b10 nRanks 04

inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:6387 [0] /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/misc/ibvwrap.cc:150 NCCL WARN Call to ibv_reg_mr_iova2 failed with error Invalid argument
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:6387 [0] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/transport/net_ib.cc:943 -> 2
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:6387 [0] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/transport/net.cc:858 -> 2
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:6387 [0] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/proxy.cc:1316 -> 2

inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:6387 [0] /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/proxy.cc:1495 NCCL WARN [Service thread] Error encountered progressing operation=Connect, res=2, closing connection

inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/misc/socket.cc:57 NCCL WARN socketProgress: Connection closed by remote peer inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg<60531>
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/misc/socket.cc:791 -> 6

inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/proxy.cc:1153 NCCL WARN Socket recv failed while polling for opId=0x55ca739336c8
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8fhgg:7:7 [0] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/transport/net.cc:385 -> 3
*** SIGSEGV received at time=1741692641 on cpu 1 ***
inference-deepseek-rjmvpa-sllm-0-model-54f4c485bf-8(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] raise RuntimeError(f"NCCL error: {error_str}")
(RayWorkerWrapper pid=567, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] RuntimeError: NCCL error: internal error - please report this issue to the NCCL developers
(RayWorkerWrapper pid=574, ip=10.244.2.219)
(RayWorkerWrapper pid=574, ip=10.244.2.219)
(RayWorkerWrapper pid=574, ip=10.244.2.219)
(RayWorkerWrapper pid=574, ip=10.244.2.219) inference-deepseek-rjmvpa-head-0-model-854647bdfb-259lc:574:8763 [3] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/bu
(RayWorkerWrapper pid=560, ip=10.244.2.219)
(RayWorkerWrapper pid=560, ip=10.244.2.219)
(RayWorkerWrapper pid=560, ip=10.244.2.219)
(RayWorkerWrapper pid=560, ip=10.244.2.219) inference-deepseek-rjmvpa-head-0-model-854647bdfb-259lc:560:8753 [2] NCCL INFO /public/home/zhuww/zhoubo/temp/rccl/build/hipify/src/transport/net.cERROR 03-11 19:30:41 worker_base.py:609] Error executing method 'init_device'. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=574, ip=10.244.2.219) ERROR 03-11 19:30:41 worker_base.py:609] Error executing method 'init_device'. This might cause deadlock in distributed execution.
ERROR 03-11 19:30:41 worker_base.py:609] Error executing method 'init_device'. This might cause deadlock in distributed execution.
ERROR 03-11 19:30:41 worker_base.py:609] Traceback (most recent call last):
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 601, in execute_method
ERROR 03-11 19:30:41 worker_base.py:609] return run_method(target, method, args, kwargs)
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/utils.py", line 2313, in run_method
ERROR 03-11 19:30:41 worker_base.py:609] return func(*args, **kwargs)
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 166, in init_device
ERROR 03-11 19:30:41 worker_base.py:609] init_worker_distributed_environment(self.vllm_config, self.rank,
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 516, in init_worker_distributed_environment
ERROR 03-11 19:30:41 worker_base.py:609] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1103, in ensure_model_parallel_initialized
ERROR 03-11 19:30:41 worker_base.py:609] initialize_model_parallel(tensor_model_parallel_size,
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 1064, in initialize_model_parallel
ERROR 03-11 19:30:41 worker_base.py:609] _PP = init_model_parallel_group(group_ranks,
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 876, in init_model_parallel_group
ERROR 03-11 19:30:41 worker_base.py:609] return GroupCoordinator(
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 218, in __init__
ERROR 03-11 19:30:41 worker_base.py:609] self.pynccl_comm = PyNcclCommunicator(
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl.py", line 99, in __init__
ERROR 03-11 19:30:41 worker_base.py:609] self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 277, in ncclCommInitRank
ERROR 03-11 19:30:41 worker_base.py:609] self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
ERROR 03-11 19:30:41 worker_base.py:609] File "/usr/local/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 256, in NCCL_CHECK
ERROR 03-11 19:30:41 worker_base.py:609] raise RuntimeError(f"NCCL error: {error_str}")
ERROR 03-11 19:30:41 worker_base.py:609] RuntimeError: NCCL error: internal error - please report this issue to the NCCL developers
PC: @ 0x7f4416a7db14 (unknown) ncclProxyService()
@ 0x7f44924f4520 (unknown) (unknown)
[2025-03-11 19:30:41,554 E 7 6387] logging.cc:460: *** SIGSEGV received at time=1741692641 on cpu 1 ***
[2025-03-11 19:30:41,555 E 7 6387] logging.cc:460: PC: @ 0x7f4416a7db14 (unknown) ncclProxyService()
[2025-03-11 19:30:41,555 E 7 6387] logging.cc:460: @ 0x7f44924f4520 (unknown) (unknown)
Fatal Python error: Segmentation fault