框架组件
提问帖
有回复
BW1000 vllm0.11.0 qwen3-8B启动失败:RuntimeError: Gloo connectFullMesh failed
2026-01-16 17:10:44
130
机器:BW1000 gfx936 镜像:harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.11.0-ubuntu22.04-dtk25.04.2-1226-das1.7-py3.10-20251226 容器启动命令: docker run -it --network=host --ipc=host --name test --shm-size=16G --device=/dev/kfd --device=/dev/mkfd --device=/dev/dri --privileged -v /root:/root -v /opt/hyhal:/opt/hyhal --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined localhost:61001/cdc2/dcu_vllm:0.11.0-ubuntu22.04-dtk25.04.2-1226-das1.7-py3.10-20251226 /bin/bash 服务启动: VLLM_HOST_IP=127.0.0.1 MASTER_ADDR=127.0.0.1 MASTER_PORT=29500 GLOO_SOCKET_IFNAME=bond0 NCCL_SOCKET_IFNAME=bond0 HIP_VISIBLE_DEVICES=0,1 python3 -m vllm.entrypoints.openai.api_server --served-model-name qwen3 --model /root/models/Qwen3-8B --tensor-parallel-size 2 --port 9080 --enable-auto-tool-choice --tool-call-parser hermes --gpu-memory-utilization 0.9 --trust-remote-code 报错信息: WARNING: Logging before InitGoogleLogging() is written to STDERR E0116 17:08:02.158560 26638 ProcessGroupGloo.cpp:143] Gloo connectFullMesh failed with [/home/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:144] no error ERROR 01-16 17:08:02 [multiproc_executor.py:597] WorkerProc failed to start. ERROR 01-16 17:08:02 [multiproc_executor.py:597] Traceback (most recent call last): ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main ERROR 01-16 17:08:02 [multiproc_executor.py:597] worker = WorkerProc(*args, **kwargs) ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 430, in __init__ ERROR 01-16 17:08:02 [multiproc_executor.py:597] self.worker.init_device() ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 299, in init_device ERROR 01-16 17:08:02 [multiproc_executor.py:597] self.worker.init_device() # type: ignore ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 169, in init_device ERROR 01-16 17:08:02 [multiproc_executor.py:597] init_worker_distributed_environment(self.vllm_config, self.rank, ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 703, in init_worker_distributed_environment ERROR 01-16 17:08:02 [multiproc_executor.py:597] init_distributed_environment(parallel_config.world_size, rank, ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1085, in init_distributed_environment ERROR 01-16 17:08:02 [multiproc_executor.py:597] _WORLD = init_world_group(ranks, local_rank, backend) ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 910, in init_world_group ERROR 01-16 17:08:02 [multiproc_executor.py:597] return GroupCoordinator( ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 224, in __init__ ERROR 01-16 17:08:02 [multiproc_executor.py:597] cpu_group = torch.distributed.new_group(ranks, backend="gloo") ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 97, in wrapper ERROR 01-16 17:08:02 [multiproc_executor.py:597] func_return = func(*args, **kwargs) ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 4565, in new_group ERROR 01-16 17:08:02 [multiproc_executor.py:597] return _new_group_with_tag( ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 4648, in _new_group_with_tag ERROR 01-16 17:08:02 [multiproc_executor.py:597] pg, pg_store = _new_process_group_helper( ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1744, in _new_process_group_helper ERROR 01-16 17:08:02 [multiproc_executor.py:597] backend_class = ProcessGroupGloo( ERROR 01-16 17:08:02 [multiproc_executor.py:597] RuntimeError: Gloo connectFullMesh failed with [/home/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:144] no error
全部评论(4)
只看楼主
M
minfy
2楼
您好,使用命令ifconfig -a查看网络接口,然后把服务启动命令里修改成可以的网络接口后服务就可以正常启动了。
1/16
T
TTTzc
回复 @minfy
你好,我设置的bond0已经是通过fconfig -a查看网络接口后调整的启动命令,我在不开tp单卡部署的情况下没有问题,但是多卡tp下启动异常,有没有什么其他的原因
1/19
M
minfy
回复 @TTTzc
我这边目前多卡启动没问题的,请参考下面的启动命令,帮忙提供一下多卡的异常截图,确认是否和之前的报错相同。
VLLM_HOST_IP=127.0.0.1 MASTER_ADDR=127.0.0.1 MASTER_PORT=29500 GLOO_SOCKET_IFNAME=bond1 NCCL_SOCKET_IFNAME=bond1 HIP_VISIBLE_DEVICES=0,1 python3 -m vllm.entrypoints.openai.api_server --served-model-name qwen3 --model /root/workspace/models/Qwen3-8B --tensor-parallel-size 2 --port 9080 --enable-auto-tool-choice --tool-call-parser hermes --gpu-memory-utilization 0.9 --trust-remote-code
1/19
T
TTTzc
楼主
1楼
求助~~~
1/16
没有更多内容了
--
楼主
Level.1
0 帖子
楼主热帖
暂无数据