光合开发者社区-论坛

yangzhong

2楼

使用镜像和节点信息是什么？

2025/3/12

YoloVme50

回复 @yangzhong

您好，我们使用了dtk24.04作为基础镜像在里面安装了vllm-dtk，我们使用的k100 ai，在使用vllm部署qwq32b之后，某几次请求会导致发生这个错误导致服务中断

2025/3/12

YoloVme50

回复 @yangzhong

启动命令是 HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m vllm.entrypoints.openai.api_server --model /data1/cfl_workspace/models/Qwq-32B --tensor-parallel-size 8 --trust-remote-code --gpu-memory-utilization 0.9 --host 0.0.0.0 --port 7032 --max-model-len 12288

2025/3/12

yangzhong

回复 @yangzhong

你使用的是镜像pytorch2.4.1-ubuntu22.04-dtk25.04-rc4-vllm0.6.6-py3.10.tar.gz吗？建议换成docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.3.0-ubuntu22.04-dtk24.04.3-py3.10这个镜像试试，

启动vllm server

export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

vllm serve /data1/cfl_workspace/models/Qwq-32B --tensor-parallel-size 8 –-trust-remote-code --gpu-memory-utilization 0.9 --max-model-len 12288 –-host localhost –-port 8000

2025/3/12

全部6条回复

YoloVme50

楼主

1楼

E0311 18:38:24.553790 170421 ProcessGroupNCCL.cpp:474] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=18727, OpType=GATHER, NumelIn=19008, NumelOut=19008, Timeout(ms)=1800000) ran for 1800362 milliseconds before timing out. (RayWorkerVllm pid=169425) E0311 18:38:24.538865 170430 ProcessGroupNCCL.cpp:474] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=18727, OpType=GATHER, NumelIn=19008, NumelOut= 0, Timeout(ms)=1800000) ran for 1800347 milliseconds before timing out. (RayWorkerVllm pid=169935) WARNING: Logging before InitGoogleLogging() is written to STDERR [repeated 6x across cluster] (RayWorkerVllm pid=170165) I0311 17:47:28.209088 170165 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMI NG: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94094796433984 [repeated 20x across cluster]

2025/3/12

网

网友DVHXch

回复 @YoloVme50

解决了吗

2025/3/19