DCU Z100的GPU，在用vllm0.6.2进行推理时，会报如下错误：

mWorkerProcess pid=739) INFO 12-28 15:50:27 model_runner.py:1041] Starting to load model /app/models/glm-4-9b-chat...
(VllmWorkerProcess pid=739) INFO 12-28 15:50:27 selector.py:121] Using ROCmFlashAttention backend.
INFO 12-28 15:50:27 selector.py:121] Using ROCmFlashAttention backend.
Loading safetensors checkpoint shards: 0% Completed | 0/10 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 10% Completed | 1/10 [00:02<00:18, 2.05s/it]
Loading safetensors checkpoint shards: 20% Completed | 2/10 [00:03<00:15, 1.93s/it]
Loading safetensors checkpoint shards: 30% Completed | 3/10 [00:05<00:13, 1.97s/it]
Loading safetensors checkpoint shards: 40% Completed | 4/10 [00:07<00:11, 1.96s/it]
Loading safetensors checkpoint shards: 50% Completed | 5/10 [00:09<00:09, 1.93s/it]
Loading safetensors checkpoint shards: 60% Completed | 6/10 [00:11<00:07, 1.95s/it]
Loading safetensors checkpoint shards: 70% Completed | 7/10 [00:13<00:05, 1.96s/it]
Loading safetensors checkpoint shards: 80% Completed | 8/10 [00:15<00:03, 1.94s/it]
Loading safetensors checkpoint shards: 90% Completed | 9/10 [00:17<00:01, 1.91s/it]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:18<00:00, 1.80s/it]
Loading safetensors checkpoint shards: 100% Completed | 10/10 [00:18<00:00, 1.90s/it]

(VllmWorkerProcess pid=739) INFO 12-28 15:50:52 model_runner.py:1052] Loading model weights took 8.8289 GB
INFO 12-28 15:50:52 model_runner.py:1052] Loading model weights took 8.8289 GB
I1228 15:50:53.405133 739 ProcessGroupNCCL.cpp:1991] [PG 2 Rank 1] ProcessGroupNCCL created ncclComm_ 0x5598102dcf20 on CUDA device:
I1228 15:50:53.405187 739 ProcessGroupNCCL.cpp:1996] [PG 2 Rank 1] NCCL_DEBUG: N/A
I1228 15:50:53.405314 666 ProcessGroupNCCL.cpp:1991] [PG 2 Rank 0] ProcessGroupNCCL created ncclComm_ 0x5642bb0d48b0 on CUDA device:
I1228 15:50:53.405349 666 ProcessGroupNCCL.cpp:1996] [PG 2 Rank 0] NCCL_DEBUG: N/A
INFO 12-28 15:51:02 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241228-155102.pkl...
INFO 12-28 15:51:02 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20241228-155102.pkl.
Process SpawnProcess-1:
ERROR 12-28 15:51:02 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 739 died, exit code: -15
INFO 12-28 15:51:02 multiproc_worker_utils.py:124] Killing local vLLM worker processes
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1622, in execute_model
hidden_or_intermediate_states = model_executable(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/chatglm.py", line 633, in forward
hidden_states = self.transformer(input_ids, positions, kv_caches,
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/chatglm.py", line 564, in forward
hidden_states = self.encoder(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/chatglm.py", line 474, in forward
hidden_states = layer(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/chatglm.py", line 409, in forward
attention_output = self.self_attention(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/chatglm.py", line 308, in forward
context_layer = self.attn(
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/attention/layer.py", line 98, in forward
return self.impl.forward(query,
File "/usr/local/lib/python3.10/site-packages/vllm/attention/backends/rocm_flash_attn.py", line 572, in forward
out = self.attn_func(
File "/usr/local/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 1125, in flash_attn_varlen_func
return FlashAttnVarlenFunc.apply(
File "/usr/local/lib/python3.10/site-packages/torch/autograd/function.py", line 598, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/usr/local/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 621, in forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _flash_attn_varlen_forward(
File "/usr/local/lib/python3.10/site-packages/flash_attn/flash_attn_interface.py", line 89, in _flash_attn_varlen_forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.varlen_fwd(
RuntimeError: HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3.
Compile with TORCH_USE_HIP_DSA  to enable device-side assertions.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 388, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
File "/usr/local/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 138, in from_engine_args
return cls(
File "/usr/local/lib/python3.10/site-packages/vllm/engine/multiprocessing/engine.py", line 78, in init
self.engine = LLMEngine(*args,
File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 339, in init
self._initialize_kv_caches()
File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 474, in _initialize_kv_caches
self.model_executor.determine_num_available_blocks())
File "/usr/local/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 39, in determine_num_available_blocks
num_blocks = self._run_workers("determine_num_available_blocks", )
File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 185, in _run_workers
driver_worker_output = driver_worker_method(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 223, in determine_num_available_blocks
self.model_runner.profile_run()
File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 1263, in profile_run
self.execute_model(model_input, kv_caches, intermediate_tensors)
File "/usr/local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner_base.py", line 152, in _wrapper
raise type(err)(
RuntimeError: Error in model execution (input dumped to /tmp/err_execute_model_input_20241228-155102.pkl): HIP error: invalid device function
HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing AMD_SERIALIZE_KERNEL=3.
Compile with TORCH_USE_HIP_DSA  to enable device-side assertions.

I1228 15:51:04.099032 666 ProcessGroupNCCL.cpp:1126] [PG 0 Rank 0] ProcessGroupNCCL destructor entered.
I1228 15:51:04.099105 666 ProcessGroupNCCL.cpp:1111] [PG 0 Rank 0] Launching ProcessGroupNCCL abort asynchrounously.
I1228 15:51:04.099432 666 ProcessGroupNCCL.cpp:1016] [PG 0 Rank 0] future is successfully executed for: ProcessGroup abort
I1228 15:51:04.099455 666 ProcessGroupNCCL.cpp:1117] [PG 0 Rank 0] ProcessGroupNCCL aborts successfully.
I1228 15:51:04.099597 666 ProcessGroupNCCL.cpp:1149] [PG 0 Rank 0] ProcessGroupNCCL watchdog thread joined.
I1228 15:51:04.099885 666 ProcessGroupNCCL.cpp:1153] [PG 0 Rank 0] ProcessGroupNCCL heart beat monitor thread joined.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 571, in <module> 
uvloop.run(run_server(args))
File "/usr/local/lib/python3.10/site-packages/uvloop/init.py", line 82, in run
return loop.run_until_complete(wrapper())
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.10/site-packages/uvloop/init.py", line 61, in wrapper
return await main
File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 538, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/local/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 105, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/local/lib/python3.10/contextlib.py", line 199, in aenter
return await anext(self.gen)
File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 192, in build_async_engine_client_from_engine_args
raise RuntimeError(
RuntimeError: Engine process failed to start