硬件为 K100AI 64G*8 docker-compose 配置文件如下 ```yml services: qwen3-5-35b-a3b-tool: image: image.sourcefind.cn:5000/dcu/admin/base/custom:pytorch2.9.1-ubuntu22.04-dtk26.04-0130-py3.10-20260204-qwen3_5 container_name: dcu-qwen3-5-35b-a3b-tool privileged: true devices: - /dev/kfd - /dev/dri - /dev/mkfd group_add: - video cap_add: - SYS_PTRACE security_opt: - seccomp:unconfined environment: - HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 - OMP_NUM_THREADS=256 - NCCL_MIN_NCHANNELS=16 - NCCL_MAX_NCHANNELS=16 - VLLM_USE_FLASH_ATTN_PA=0 - VLLM_USE_FUSED_RMS_ROPE=1 volumes: - my-pip-cache:/root/.cache/pip - /data/models/Qwen3.5-35B-A3B:/data/models/Qwen3.5-35B-A3B - /opt/hyhal:/opt/hyhal:ro - /sys/kernel/debug:/sys/kernel/debug:ro shm_size: '1024g' # command: bash # tty: true # stdin_open: true command: bash -c "pip install git+https://github.com/huggingface/transformers.git && \ pip install numpy==1.25.0 -i https://pypi.tuna.tsinghua.edu.cn/simple && \ export ALLREDUCE_STREAM_WITH_COMPUTE=1 && \ unset NCCL_ALGO && \ export NCCL_MIN_NCHANNELS=16 && \ export NCCL_MAX_NCHANNELS=16 && \ export NCCL_NET_GDR_READ=1 && \ export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 && \ export VLLM_SPEC_DECODE_EAGER=1 && \ export VLLM_MLA_DISABLE=0 && \ export VLLM_USE_FLASH_MLA=1 && \ export VLLM_RPC_TIMEOUT=1800000 && \ export VLLM_NUMA_BIND=1 && \ export VLLM_RANK0_NUMA=0 && \ export VLLM_RANK1_NUMA=1 && \ export VLLM_RANK2_NUMA=2 && \ export VLLM_RANK3_NUMA=3 && \ export VLLM_RANK4_NUMA=4 && \ export VLLM_RANK5_NUMA=5 && \ export VLLM_RANK6_NUMA=6 && \ export VLLM_RANK7_NUMA=7 && \ vllm serve /data/models/Qwen3.5-35B-A3B \ --served-model-name Qwen3.5-35B-A3B \ --port 8000 \ --host 0.0.0.0 \ --max-model-len 65535 \ --reasoning-parser qwen3 \ --gpu-memory-utilization 0.95 \ --tool-call-parser hermes \ --enable-auto-tool-choice \ --max-num-batched-tokens 32768 \ --tensor-parallel-size 8 \ --trust-remote-code --api-key sk-Wp1S**********************PZD86" ports: - "18000:8000" ulimits: nproc: 65535 nofile: soft: 65535 hard: 65535 volumes: my-pip-cache: ``` 启动后日志为： ``` (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/audio/translations, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/completions, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/completions/render, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/messages, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/models, Methods: GET (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /load, Methods: GET (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /version, Methods: GET (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /ping, Methods: GET (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /ping, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /invocations, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /classify, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/embeddings, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /score, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/score, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /rerank, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v1/rerank, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /v2/rerank, Methods: POST (APIServer pid=1) INFO 02-25 14:46:15 [launcher.py:46] Route: /pooling, Methods: POST (APIServer pid=1) INFO: Started server process [1] (APIServer pid=1) INFO: Waiting for application startup. (APIServer pid=1) INFO: Application startup complete. ``` 当调用模型后 提示 ``` frame #7: + 0x94ac3 (0x7f4d023faac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f4d0248c850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:33.528592 3120 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:34.527788 3150 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59586, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7fb9d7f6f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7fb9d057fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7fb9d05800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7fb9d0581adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7fb9d057da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7fb99f26c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7fb99aab0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7fb9e0c09ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7fb9e0c9b850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:34.527777 3132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59560, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f0bf036f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f0be897fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f0be89800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f0be8981adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f0be897da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f0bb766c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f0bb2eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f0bf8ff0ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f0bf9082850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:34.531567 3132 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:34.531563 3150 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:34.528102 3126 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59546, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f240416f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f23fbd7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f23fbd800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f23fbd81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f23fbd7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f23caa6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f23c62b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f240c570ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f240c602850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:34.531643 3126 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:34.528695 3120 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59542, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f4cf976f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f4cf1d7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f4cf1d800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f4cf1d81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f4cf1d7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f4cc0a6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f4cbc2b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f4d023faac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f4d0248c850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:34.532250 3120 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:35.531675 3132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59560, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f0bf036f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f0be897fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f0be89800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f0be8981adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f0be897da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f0bb766c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f0bb2eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f0bf8ff0ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f0bf9082850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:35.531680 3150 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59586, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7fb9d7f6f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7fb9d057fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7fb9d05800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7fb9d0581adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7fb9d057da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7fb99f26c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7fb99aab0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7fb9e0c09ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7fb9e0c9b850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:35.535321 3132 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:35.535344 3150 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:35.531817 3126 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59546, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f240416f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f23fbd7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f23fbd800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f23fbd81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f23fbd7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f23caa6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f23c62b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f240c570ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f240c602850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:35.535436 3126 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:35.532349 3120 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59542, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f4cf976f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f4cf1d7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f4cf1d800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f4cf1d81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f4cf1d7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f4cc0a6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f4cbc2b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f4d023faac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f4d0248c850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:35.536065 3120 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:36.535449 3150 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59586, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7fb9d7f6f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7fb9d057fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7fb9d05800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7fb9d0581adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7fb9d057da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7fb99f26c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7fb99aab0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7fb9e0c09ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7fb9e0c9b850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:36.535426 3132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59560, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f0bf036f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f0be897fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f0be89800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f0be8981adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f0be897da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f0bb766c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f0bb2eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f0bf8ff0ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f0bf9082850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:36.538957 3150 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:36.538977 3132 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:36.535539 3126 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59546, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f240416f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f23fbd7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f23fbd800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f23fbd81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f23fbd7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f23caa6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f23c62b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f240c570ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f240c602850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:36.539052 3126 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:36.536185 3120 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59542, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f4cf976f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f4cf1d7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f4cf1d800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f4cf1d81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f4cf1d7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f4cc0a6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f4cbc2b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f4d023faac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f4d0248c850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:36.539804 3120 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:37.539059 3150 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59586, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7fb9d7f6f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7fb9d057fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7fb9d05800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7fb9d0581adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7fb9d057da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7fb99f26c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7fb99aab0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7fb9e0c09ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7fb9e0c9b850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:37.542558 3150 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:37.539160 3132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59560, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f0bf036f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f0be897fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f0be89800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f0be8981adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f0be897da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f0bb766c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f0bb2eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f0bf8ff0ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f0bf9082850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:37.542618 3132 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:37.539234 3126 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59546, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f240416f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f23fbd7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f23fbd800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f23fbd81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f23fbd7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f23caa6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f23c62b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f240c570ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f240c602850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:37.542987 3126 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:37.539939 3120 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59542, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f4cf976f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f4cf1d7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f4cf1d800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f4cf1d81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f4cf1d7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f4cc0a6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f4cbc2b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f4d023faac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f4d0248c850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:37.543620 3120 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:38.542656 3150 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59586, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7fb9d7f6f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7fb9d057fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7fb9d05800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7fb9d0581adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7fb9d057da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7fb99f26c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7fb99aab0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7fb9e0c09ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7fb9e0c9b850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:38.546142 3150 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:38.542755 3132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59560, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f0bf036f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f0be897fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f0be89800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f0be8981adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f0be897da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f0bb766c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f0bb2eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f0bf8ff0ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f0bf9082850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:38.546368 3132 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:38.543217 3126 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59546, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f240416f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f23fbd7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f23fbd800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f23fbd81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f23fbd7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f23caa6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f23c62b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f240c570ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f240c602850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:38.546926 3126 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:38.543761 3120 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59542, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f4cf976f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f4cf1d7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f4cf1d800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f4cf1d81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f4cf1d7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f4cc0a6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f4cbc2b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f4d023faac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f4d0248c850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:38.547374 3120 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:39.546336 3150 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59586, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7fb9d7f6f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7fb9d057fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7fb9d05800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7fb9d0581adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7fb9d057da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7fb99f26c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7fb99aab0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7fb9e0c09ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7fb9e0c9b850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:39.546509 3132 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59560, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f0bf036f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f0be897fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f0be89800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f0be8981adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f0be897da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f0bb766c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f0bb2eb0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f0bf8ff0ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f0bf9082850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:39.550119 3150 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 5] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:39.550132 3132 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 6] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:39.547034 3126 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59546, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f240416f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f23fbd7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f23fbd800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f23fbd81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f23fbd7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f23caa6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f23c62b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f240c570ac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f240c602850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:39.550535 3126 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 1] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe W0225 14:53:39.547631 3120 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=35, addr=[localhost]:59542, remote=[localhost]:34949): Broken pipe Exception raised from sendBytes at /home/pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x88 (0x7f4cf976f9d8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so) frame #1: + 0x5b6fcce (0x7f4cf1d7fcce in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5b700a0 (0x7f4cf1d800a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5b71adb (0x7f4cf1d81adb in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector, std::allocator >, std::allocator, std::allocator > > > const&) + 0x30a (0x7f4cf1d7da8a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c4 (0x7f4cc0a6c3b4 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_hip.so) frame #6: + 0xdc253 (0x7f4cbc2b0253 in /lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x7f4d023faac3 in /lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x7f4d0248c850 in /lib/x86_64-linux-gnu/libc.so.6) W0225 14:53:39.551685 3120 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 7] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.15.1) with config: model='/data/models/Qwen3.5-35B-A3B', speculative_config=None, tokenizer='/data/models/Qwen3.5-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=65535, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen3.5-35B-A3B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none', '+sparse_attn_indexer', '+sparse_attn_indexer'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [32768], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}, (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['chatcmpl-9f4e70554ecbc0c4-8eff0cec'],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[None],num_computed_tokens=[249],num_output_tokens=[1]), num_scheduled_tokens={chatcmpl-9f4e70554ecbc0c4-8eff0cec: 1}, total_num_scheduled_tokens=1, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 0], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=null, ec_connector_metadata=null) (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=2, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.00023358346229085392, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=None, kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None) (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] EngineCore encountered a fatal error. (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] Traceback (most recent call last): (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 939, in run_engine_core (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] engine_core.run_busy_loop() (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 966, in run_busy_loop (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] self._process_engine_step() (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 999, in _process_engine_step (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] outputs, model_executed = self.step_fn() (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 486, in step_with_batch_queue (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] model_output = future.result() (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 80, in result (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] return super().result() (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] return self.__get_result() (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] raise self._exception (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 84, in wait_for_response (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] response = self.aggregate(get_response()) (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 351, in get_response (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] status, result = mq.dequeue( (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 616, in dequeue (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] with self.acquire_read(timeout, cancel, indefinite) as buf: (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] return next(self.gen) (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 531, in acquire_read (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] raise RuntimeError("cancelled") (EngineCore_DP0 pid=552) ERROR 02-25 14:53:40 [core.py:948] RuntimeError: cancelled (APIServer pid=1) ERROR 02-25 14:53:40 [async_llm.py:693] AsyncLLM output_handler failed. (APIServer pid=1) ERROR 02-25 14:53:40 [async_llm.py:693] Traceback (most recent call last): (APIServer pid=1) ERROR 02-25 14:53:40 [async_llm.py:693] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 649, in output_handler (APIServer pid=1) ERROR 02-25 14:53:40 [async_llm.py:693] outputs = await engine_core.get_output_async() (APIServer pid=1) ERROR 02-25 14:53:40 [async_llm.py:693] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 894, in get_output_async (APIServer pid=1) ERROR 02-25 14:53:40 [async_llm.py:693] raise self._format_exception(outputs) from None (APIServer pid=1) ERROR 02-25 14:53:40 [async_llm.py:693] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] Error in chat completion stream generator. (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] Traceback (most recent call last): (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 735, in chat_completion_stream_generator (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] async for res in result_generator: (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 572, in generate (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] out = q.get_nowait() or await q.get() (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/output_processor.py", line 80, in get (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] raise output (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 649, in output_handler (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] outputs = await engine_core.get_output_async() (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 894, in get_output_async (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] raise self._format_exception(outputs) from None (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] Error in chat completion stream generator. (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] Traceback (most recent call last): (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 735, in chat_completion_stream_generator (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] async for res in result_generator: (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 572, in generate (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] out = q.get_nowait() or await q.get() (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/output_processor.py", line 80, in get (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] raise output (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/chat_completion/serving.py", line 735, in chat_completion_stream_generator (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] async for res in result_generator: (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 572, in generate (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] out = q.get_nowait() or await q.get() (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/output_processor.py", line 80, in get (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] raise output (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 649, in output_handler (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] outputs = await engine_core.get_output_async() (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 894, in get_output_async (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] raise self._format_exception(outputs) from None (APIServer pid=1) ERROR 02-25 14:53:40 [serving.py:1377] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause. (EngineCore_DP0 pid=552) Process EngineCore_DP0: (EngineCore_DP0 pid=552) Traceback (most recent call last): (EngineCore_DP0 pid=552) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap (EngineCore_DP0 pid=552) self.run() (EngineCore_DP0 pid=552) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run (EngineCore_DP0 pid=552) self._target(*self._args, **self._kwargs) (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 950, in run_engine_core (EngineCore_DP0 pid=552) raise e (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 939, in run_engine_core (EngineCore_DP0 pid=552) engine_core.run_busy_loop() (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 966, in run_busy_loop (EngineCore_DP0 pid=552) self._process_engine_step() (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 999, in _process_engine_step (EngineCore_DP0 pid=552) outputs, model_executed = self.step_fn() (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 486, in step_with_batch_queue (EngineCore_DP0 pid=552) model_output = future.result() (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 80, in result (EngineCore_DP0 pid=552) return super().result() (EngineCore_DP0 pid=552) File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result (EngineCore_DP0 pid=552) return self.__get_result() (EngineCore_DP0 pid=552) File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result (EngineCore_DP0 pid=552) raise self._exception (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 84, in wait_for_response (EngineCore_DP0 pid=552) response = self.aggregate(get_response()) (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 351, in get_response (EngineCore_DP0 pid=552) status, result = mq.dequeue( (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 616, in dequeue (EngineCore_DP0 pid=552) with self.acquire_read(timeout, cancel, indefinite) as buf: (EngineCore_DP0 pid=552) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ (EngineCore_DP0 pid=552) return next(self.gen) (EngineCore_DP0 pid=552) File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 531, in acquire_read (EngineCore_DP0 pid=552) raise RuntimeError("cancelled") (EngineCore_DP0 pid=552) RuntimeError: cancelled (APIServer pid=1) INFO: Shutting down (APIServer pid=1) INFO: Waiting for application shutdown. (APIServer pid=1) INFO: Application shutdown complete. (APIServer pid=1) INFO: Finished server process [1] * 终端将被任务重用，按任意键关闭。