机器:BW1000 gfx936
镜像:harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.11.0-ubuntu22.04-dtk25.04.2-1226-das1.7-py3.10-20251226
容器启动命令:
docker run -it
--network=host
--ipc=host
--name test
--shm-size=16G
--device=/dev/kfd
--device=/dev/mkfd
--device=/dev/dri
--privileged
-v /root:/root
-v /opt/hyhal:/opt/hyhal
--group-add video
--cap-add=SYS_PTRACE
--security-opt seccomp=unconfined
localhost:61001/cdc2/dcu_vllm:0.11.0-ubuntu22.04-dtk25.04.2-1226-das1.7-py3.10-20251226
/bin/bash
服务启动:
VLLM_HOST_IP=127.0.0.1 MASTER_ADDR=127.0.0.1 MASTER_PORT=29500 GLOO_SOCKET_IFNAME=bond0 NCCL_SOCKET_IFNAME=bond0 HIP_VISIBLE_DEVICES=0,1 python3 -m vllm.entrypoints.openai.api_server
--served-model-name qwen3
--model /root/models/Qwen3-8B
--tensor-parallel-size 2
--port 9080
--enable-auto-tool-choice
--tool-call-parser hermes
--gpu-memory-utilization 0.9
--trust-remote-code
报错信息:
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0116 17:08:02.158560 26638 ProcessGroupGloo.cpp:143] Gloo connectFullMesh failed with [/home/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:144] no error
ERROR 01-16 17:08:02 [multiproc_executor.py:597] WorkerProc failed to start.
ERROR 01-16 17:08:02 [multiproc_executor.py:597] Traceback (most recent call last):
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main
ERROR 01-16 17:08:02 [multiproc_executor.py:597] worker = WorkerProc(*args, **kwargs)
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 430, in __init__
ERROR 01-16 17:08:02 [multiproc_executor.py:597] self.worker.init_device()
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/worker/worker_base.py", line 299, in init_device
ERROR 01-16 17:08:02 [multiproc_executor.py:597] self.worker.init_device() # type: ignore
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 169, in init_device
ERROR 01-16 17:08:02 [multiproc_executor.py:597] init_worker_distributed_environment(self.vllm_config, self.rank,
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 703, in init_worker_distributed_environment
ERROR 01-16 17:08:02 [multiproc_executor.py:597] init_distributed_environment(parallel_config.world_size, rank,
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1085, in init_distributed_environment
ERROR 01-16 17:08:02 [multiproc_executor.py:597] _WORLD = init_world_group(ranks, local_rank, backend)
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 910, in init_world_group
ERROR 01-16 17:08:02 [multiproc_executor.py:597] return GroupCoordinator(
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 224, in __init__
ERROR 01-16 17:08:02 [multiproc_executor.py:597] cpu_group = torch.distributed.new_group(ranks, backend="gloo")
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 97, in wrapper
ERROR 01-16 17:08:02 [multiproc_executor.py:597] func_return = func(*args, **kwargs)
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 4565, in new_group
ERROR 01-16 17:08:02 [multiproc_executor.py:597] return _new_group_with_tag(
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 4648, in _new_group_with_tag
ERROR 01-16 17:08:02 [multiproc_executor.py:597] pg, pg_store = _new_process_group_helper(
ERROR 01-16 17:08:02 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1744, in _new_process_group_helper
ERROR 01-16 17:08:02 [multiproc_executor.py:597] backend_class = ProcessGroupGloo(
ERROR 01-16 17:08:02 [multiproc_executor.py:597] RuntimeError: Gloo connectFullMesh failed with [/home/pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:144] no error