光合开发者社区-论坛

数学库

提问帖

新人贴

K100_AI服务器在open欧拉22.03系统上安装驱动26.04，驱动起不来

2026-06-02 15:03:57

156

安装后提示successful，reboot之后，lsmod |grep hycu 或hydcu没有任何输出，执行 `modprobe hycu或hydcu，提示没有这个module，请问如何手动启动驱动，还有查看哪个日志能知道为啥驱动没有自动起来。` lspci 里面已经能识别c3000，是否代表已经能够正常识别dcu

回复 @panhb

vllm启动后报错
(Worker_TP0 pid=876)

(Worker_TP0 pid=876) INFO 06-04 10:12:32 [default_loader.py:291] Loading weights took 38.85 seconds

(Worker_TP0 pid=876) INFO 06-04 10:12:33 [gpu_model_runner.py:4596] Model loading took 7.64 GiB memory and 39.042913 seconds

(Worker_TP0 pid=876) INFO 06-04 10:12:54 [gpu_worker.py:356] Available KV cache memory: 48.28 GiB

(EngineCore_DP0 pid=610) INFO 06-04 10:12:54 [kv_cache_utils.py:1307] GPU KV cache size: 703,168 tokens

(EngineCore_DP0 pid=610) INFO 06-04 10:12:54 [kv_cache_utils.py:1312] Maximum concurrency for 4,096 tokens per request: 171.67x

Could not open /var/log/hylog/.

Invalid address access: (nil), Error code: 3.

>>>>>>>> KERNEL VMFault !!!! <<<<<<

>>>>>>>> PID: 876, SIGNAL: 0 !!!! <<<<<<

>>>>>>>> FOR DEVICE 0x558beb0cda20 <<<<<<

===========================================================================================

=========> HOSTQUEUE <0x7f0009501740>: VMFault HSA QUEUE ANALYSIS <=========

HOSTQUEUE <0x7f0009501740>: base_address 0x7f02d4200000

HOSTQUEUE <0x7f0009501740>: hsa queue id 5, stream id 24

HOSTQUEUE <0x7f0009501740>: get hsa queue W/R ptr: write index: 72, read index: 72

HOSTQUEUE <0x7f0009501740>: all command executed

===========================================================================================

=========> HOSTQUEUE <0x7f00095012f0>: VMFault HSA QUEUE ANALYSIS <=========

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [multiproc_executor.py:246] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] EngineCore failed to start.

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] Traceback (most recent call last):

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 938, in run_engine_core

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 692, in __init__

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] super().__init__(

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 113, in __init__

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 270, in _initialize_kv_caches

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] self.model_executor.initialize_from_config(kv_cache_configs)

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 116, in initialize_from_config

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] self.collective_rpc("compile_or_warm_up_model")

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 374, in collective_rpc

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] return aggregate(get_response())

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 351, in get_response

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] status, result = mq.dequeue(

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 616, in dequeue

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] with self.acquire_read(timeout, cancel, indefinite) as buf:

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] return next(self.gen)

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 531, in acquire_read

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] raise RuntimeError("cancelled")

(EngineCore_DP0 pid=610) ERROR 06-04 10:13:18 [core.py:947] RuntimeError: cancelled

全部评论（14）

只看楼主

_mmo8yl5uszbn

楼主

2楼

已经确认是只要使用张量并行参数，tp=n，这挂载进容器的n张卡就会出问题，smi看不到这些卡，之前曾看到信息卡被quarentined，使用官方镜像0.15.1，求救大神有思路吗

6/3

热

热心网友

回复 @_mmo8yl5uszbn

试过018镜像吗

6/4

1楼

后续发现浪潮服务器断电reboot之后驱动能够识别所有DCU, 出现故障的时刻是在vllm serve测试时从单卡转为设置tp=2，然后出现所有显卡掉卡，重装系统重装驱动也找不到，直到断电重启。暂无法确定故障原因在哪。后续继续多卡测试看是否会复现故障。

panhb

发下起容器命令吧，以及驱动版本hy-smi --showdriverversion

Driver Version: 6.3.30-V1.4.1a

1.启动vllm镜像：

sudo docker run --rm \

-it \

--privileged \

--name=vllm_test \

--device=/dev/kfd \

--device=/dev/dri \

--device=/dev/mkfd \

--security-opt seccomp=unconfined \

--cap-add=SYS_PTRACE \

--ipc=host \

--network host \

--shm-size=128G \

--group-add 39 \

-e HIP_VISIBLE_DEVICES=4,5 \

-v /opt/hyhal:/opt/hyhal \

-v /home/apt/models:/models \

-v /home/apt/code:/code \

harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.15.1-ubuntu22.04-dtk26.04-py3.10 /bin/bash

2.在容器中启动vllm服务：

vllm serve /models/qwen3-8B_cx \

--host 0.0.0.0 \

--port 8000 \

--max-model-len 4096 \

--served-model-name qwen3-8B_cx \

--tensor-parallel-size 2 \

--data-parallel-size 1 \

--gpu-memory-utilization 0.9 \

--max-num-seqs 20 \

--trust-remote-code \

--enforce-eager

全部11条回复

没有更多内容了