hy-smi 显示其中一张卡Error: Device 0000:a3:00.0 is isolation 启动服务命令docker run -it --network=host --ipc=host --shm-size=16G --device=/dev/kfd --device=/dev/mkfd --device=/dev/dri -v /opt/hyhal:/opt/hyhal --group-add video --cap-add=SYS_PTRACE -p 8000:8000 -v /root/workspace/Qwen:/workspace/Qwen --security-opt seccomp=unconfined image.sourcefind.cn:5000/dcu/admin/base/vllm:0.9.2-ubuntu22.04-dtk25.04.2-py3.10 /bin/bash vllm报错 ERROR 10-24 19:48:05 [core.py:591] EngineCore encountered a fatal error. ERROR 10-24 19:48:05 [core.py:591] Traceback (most recent call last): ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc ERROR 10-24 19:48:05 [core.py:591] result = get_response(w, dequeue_timeout) ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 198, in get_response ERROR 10-24 19:48:05 [core.py:591] status, result = w.worker_response_mq.dequeue( ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 507, in dequeue ERROR 10-24 19:48:05 [core.py:591] with self.acquire_read(timeout, cancel) as buf: ERROR 10-24 19:48:05 [core.py:591] File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ ERROR 10-24 19:48:05 [core.py:591] return next(self.gen) ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 469, in acquire_read ERROR 10-24 19:48:05 [core.py:591] raise TimeoutError ERROR 10-24 19:48:05 [core.py:591] TimeoutError ERROR 10-24 19:48:05 [core.py:591] ERROR 10-24 19:48:05 [core.py:591] The above exception was the direct cause of the following exception: ERROR 10-24 19:48:05 [core.py:591] ERROR 10-24 19:48:05 [core.py:591] Traceback (most recent call last): ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 582, in run_engine_core ERROR 10-24 19:48:05 [core.py:591] engine_core.run_busy_loop() ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 609, in run_busy_loop ERROR 10-24 19:48:05 [core.py:591] self._process_engine_step() ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 634, in _process_engine_step ERROR 10-24 19:48:05 [core.py:591] outputs, model_executed = self.step_fn() ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 239, in step ERROR 10-24 19:48:05 [core.py:591] model_output = self.execute_model(scheduler_output) ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 223, in execute_model ERROR 10-24 19:48:05 [core.py:591] raise err ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 214, in execute_model ERROR 10-24 19:48:05 [core.py:591] return self.model_executor.execute_model(scheduler_output) ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 158, in execute_model ERROR 10-24 19:48:05 [core.py:591] (output, ) = self.collective_rpc( ERROR 10-24 19:48:05 [core.py:591] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 221, in collective_rpc ERROR 10-24 19:48:05 [core.py:591] raise TimeoutError(f"RPC call to {method} timed out.") from e ERROR 10-24 19:48:05 [core.py:591] TimeoutError: RPC call to execute_model timed out. ERROR 10-24 19:48:05 [async_llm.py:419] AsyncLLM output_handler failed. ERROR 10-24 19:48:05 [async_llm.py:419] Traceback (most recent call last): ERROR 10-24 19:48:05 [async_llm.py:419] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 378, in output_handler ERROR 10-24 19:48:05 [async_llm.py:419] outputs = await engine_core.get_output_async() ERROR 10-24 19:48:05 [async_llm.py:419] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 740, in get_output_async ERROR 10-24 19:48:05 [async_llm.py:419] raise self._format_exception(outputs) from None ERROR 10-24 19:48:05 [async_llm.py:419] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.