卡为K100_AI,一机8卡,使用的驱动是https://download.sourcefind.cn:65024/6/main/dtk-26.04%E9%A9%B1%E5%8A%A8, rock-6.3.30-V1.4.1a.run,加载模型为qwen3-32b,
使用镜像是image.sourcefind.cn:5000/dcu/admin/base/custom:pytorch2.9.1-ubuntu22.04-dtk26.04-0130-py3.10-20260204-qwen3\_5,
启动命令是:
vllm serve /data/qwen3-32b
--max-model-len 32768
--gpu-memory-utilization 0.9
--dtype auto
--tensor-parallel-size 2
--trust-remote-code
--reasoning-parser deepseek_r1
--enable-auto-tool-choice
--tool-call-parser hermes
启动时正常启动,但是请求过来后,显示:
/opt/hyhal/bin/hy-smi

===
还有一种是卡全找不到的:

查看系统日志显示:
=================
hycu 0000:0f:00.0 ... XID:160 ... PCIe link lost!
pcieport 0000:0c:00.0: DPC: ERR_FATAL detected
hycu 0000:0f:00.0 ... XID:12 ... NBIO DED ...
hycu 0000:0f:00.0 ... uncorrectable hardware errors detected in pcie_bif block
hycu 0000:0f:00.0 ... RAS controller interrupt triggered by NBIF error
hycu 0000:0f:00.0 ... XID:51 ... HCU fault isolation success!
而且是请求多张卡会导致多张卡一起出现这个问题,断电重启后可恢复,但是重启服务还是会导致这个问题,这个该怎么排查呢