2025年12月15日11:29:41更新
使用 https://github.com/hikafeng/dcu-vllm/blob/main/docker/docker-compose-qwen3-32b-tool.yml 
但报错：IndexError: max(): Expected reduction dim 2 to have non-zero size.
/usr/local/lib/python3.10/dist-packages/vllm/attention/backends/rocm_flash_attn.py:1088: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:663.)
  sub_out = torch.nn.functional.scaled_dot_product_attention(
ERROR 12-15 11:27:58 [engine.py:476] max(): Expected reduction dim 2 to have non-zero size.

2025年12月15日10:25:26更新
回退到vllm0.8.5 可以启动可以回复，但是乱码
docker stop qwen3_vllm || true && docker rm qwen3_vllm || true && docker run -d --name qwen3_vllm --privileged --shm-size=64G --device=/dev/kfd --device=/dev/dri/ --device=/dev/mkfd --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 --ipc=host --network host --group-add video -v /sys/kernel/debug:/sys/kernel/debug -v /opt/hyhal:/opt/hyhal -v /ssddata/aimodel:/ssddata/aimodel -e HSA_OVERRIDE_GFX_VERSION=9.2.6 -e ROCBLAS_NO_TENSILE=1 -e OMP_NUM_THREADS=24 -e LD_LIBRARY_PATH=/opt/hygon/dtk/lib64:$LD_LIBRARY_PATH harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.8.5-ubuntu22.04-dtk25.04.1-rc5-das1.6-py3.10-20250724 /bin/bash -ic "source ~/.bashrc && vllm  serve /ssddata/aimodel/Qwen3-32B --enforce-eager --dtype float16 --trust-remote-code --tensor-parallel-size 2 --max-model-len 10240 --gpu-memory-utilization 0.9 --block-size 64 --enable-reasoning --reasoning-parser deepseek_r1 --host 0.0.0.0 --port 15888 --api-key APIKEY20251212  --served-model-name Qwen3-32B"

2025年12月15日09:37:58更新

1、添加HIP_VISIBLE_DEVICES=0,1 无效
2、使用最新docker镜像：harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.9.2-ubuntu22.04-dtk25.04.2-rocblas1121-das1.7-py3.10-20251203 报新的错误

rocBLAS error: No Tensile solution found for { a: 0x7f7bdc400000, a_type: "f16_r", b: 0x7f7405a00000, b_type: "f16_r", c: 0x7f7337400000, c_type: "f16_r", d: 0x7f7337400000, d_type: "f16_r", compute_type: "f32_r", transA: 'N', transB: 'N', M: 5120, N: 10240, K: 5120, alpha: 1, row_stride_a: 1, col_stride_a: 5120, row_stride_b: 1, col_stride_b: 5120, row_stride_c: 1, col_stride_c: 5120, row_stride_d: 1, col_stride_d: 5120, beta: 0, batch_count: 1, strided_batch: true, stride_a: 1, stride_b: 1, stride_c: 1, stride_d: 1, offset_a: 0, offset_b: 0, offset_c: 0, offset_d: 0, atomics_mode: atomics_allowed }

This message will be only be displayed once, unless the ROCBLAS_VERBOSE_TENSILE_ERROR environment variable is set.

rocBLAS error: No Tensile solution found for { a: 0x7f5760c00000, a_type: "f16_r", b: 0x7f4f89a00000, b_type: "f16_r", c: 0x7f4ebf400000, c_type: "f16_r", d: 0x7f4ebf400000, d_type: "f16_r", compute_type: "f32_r", transA: 'N', transB: 'N', M: 5120, N: 10240, K: 5120, alpha: 1, row_stride_a: 1, col_stride_a: 5120, row_stride_b: 1, col_stride_b: 5120, row_stride_c: 1, col_stride_c: 5120, row_stride_d: 1, col_stride_d: 5120, beta: 0, batch_count: 1, strided_batch: true, stride_a: 1, stride_b: 1, stride_c: 1, stride_d: 1, offset_a: 0, offset_b: 0, offset_c: 0, offset_d: 0, atomics_mode: atomics_allowed }

This message will be only be displayed once, unless the ROCBLAS_VERBOSE_TENSILE_ERROR environment variable is set.
ERROR 12-15 09:32:58 [engine.py:476] CUDA error: HIPBLAS_STATUS_NOT_SUPPORTED when calling `rocBLASStatusToHIPStatus(rocblas_gemm_ex( (rocblas_handle)handle, hipOperationToRocOperation(opa), hipOperationToRocOperation(opb), m, n, k, &falpha, a, rocblas_datatype_f16_r, lda, b, rocblas_datatype_f16_r, ldb, &fbeta, c, rocblas_datatype_f16_r, ldc, c, rocblas_datatype_f16_r, ldc, rocblas_datatype_f32_r, rocblas_gemm_algo_standard, 0, flag))`

基础环境：

操作系统：OpenEuler 22.03

docker镜像：image.sourcefind.cn:5000/dcu/admin/base/vllm:0.9.2-ubuntu22.04-dtk25.04.1-rc5-rocblas104381-0915-das1.6-py3.10-20250916-rc2

具体dcu型号：
[root@localhost]# lspci -nn | grep -i co-proc
03:00.0 Co-processor [0b40]: Chengdu C-3000 IC Design Co., Ltd. KONGMING [1d94:62b7] (rev 01)
a3:00.0 Co-processor [0b40]: Chengdu C-3000 IC Design Co., Ltd. KONGMING [1d94:62b7] (rev 01)

[root@localhost]# rocminfo |grep gfx 
  Name:                    gfx926                             
      Name:                    amdgcn-amd-amdhsa--gfx926:sramecc+:xnack-
  Name:                    gfx926                             
      Name:                    amdgcn-amd-amdhsa--gfx926:sramecc+:xnack-
      
驱动版本：
[root@localhost]# hy-smi --showdriverversion

================================= System Management Interface ==================================
================================================================================================
Driver Version: 6.3.16-V1.1.0a
================================================================================================
======================================== End of SMI Log ========================================

docker启动命令：

docker run -d --name qwen3_vllm --privileged --shm-size=64G --device=/dev/kfd --device=/dev/dri/ --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --ulimit memlock=-1:-1 --ipc=host --network host --group-add video -v /opt/hyhal:/opt/hyhal -v /ssddata/aimodel:/ssddata/aimodel image.sourcefind.cn:5000/dcu/admin/base/vllm:0.9.2-ubuntu22.04-dtk25.04.1-rc5-rocblas104381-0915-das1.6-py3.10-20250916-rc2 /bin/bash 


vllm  serve /ssddata/aimodel/Qwen3-8B --enforce-eager --dtype float16 --trust-remote-code --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --block-size 64 --host 0.0.0.0 --port 15888 --api-key APIKEY20251212 --served-model-name Qwen3-8B"

docker运行日志：

root@localhost:/workspace# vllm  serve /ssddata/aimodel/Qwen3-8B --enforce-eager --dtype float16 --trust-remote-code --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --block-size 64 --host 0.0.0.0 --port 15888 --api-key APIKEY20251212 --served-model-name Qwen3-8B
INFO 12-14 14:54:14 [__init__.py:245] Automatically detected platform rocm.
INFO 12-14 14:54:19 [api_server.py:1395] vLLM API server version 0.9.2
INFO 12-14 14:54:19 [cli_args.py:325] non-default args: {'host': '0.0.0.0', 'port': 15888, 'api_key': 'APIKEY20251212', 'model': '/ssddata/aimodel/Qwen3-8B', 'trust_remote_code': True, 'dtype': 'float16', 'enforce_eager': True, 'served_model_name': ['Qwen3-8B'], 'tensor_parallel_size': 2}
INFO 12-14 14:54:31 [config.py:851] This model supports multiple tasks: {'classify', 'embed', 'generate', 'reward'}. Defaulting to 'generate'.
`torch_dtype` is deprecated! Use `dtype` instead!
WARNING 12-14 14:54:31 [config.py:3403] Casting torch.bfloat16 to torch.float16.
INFO 12-14 14:54:31 [config.py:1488] Using max model len 40960
INFO 12-14 14:54:31 [config.py:2301] Chunked prefill is enabled with max_num_batched_tokens=2048.
WARNING 12-14 14:54:31 [rocm.py:348] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 12-14 14:54:39 [__init__.py:245] Automatically detected platform rocm.
INFO 12-14 14:54:43 [core.py:529] Waiting for init message from front-end.
INFO 12-14 14:54:43 [core.py:71] Initializing a V1 LLM engine (v0.9.2) with config: model='/ssddata/aimodel/Qwen3-8B', speculative_config=None, tokenizer='/ssddata/aimodel/Qwen3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-8B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
WARNING 12-14 14:54:43 [multiproc_worker_utils.py:307] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 12-14 14:54:43 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 16777216, 10, 'psm_a528590a'), local_subscribe_addr='ipc:///tmp/09806666-f4d1-432b-b9f4-9b97457ee84d', remote_subscribe_addr=None, remote_addr_ipv6=False)
INFO 12-14 14:54:50 [__init__.py:245] Automatically detected platform rocm.
INFO 12-14 14:54:50 [__init__.py:245] Automatically detected platform rocm.
WARNING 12-14 14:54:53 [worker_base.py:42] VLLM_RANK1_NUMA is unset or set incorrectly, vllm will not bind to numa! VLLM_RANK1_NUMA = -1
INFO 12-14 14:54:53 [worker_base.py:654] ########## 576 process(rank1) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191}
INFO 12-14 14:54:53 [worker_base.py:655] ########## 576 process(rank1) is running on memnode(s): {0, 1, 2, 3, 4, 5, 6, 7}
(VllmWorker rank=1 pid=576) INFO 12-14 14:54:53 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_64d55cc7'), local_subscribe_addr='ipc:///tmp/673bf2da-2658-4be0-b461-069fddd27aeb', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 12-14 14:54:54 [worker_base.py:42] VLLM_RANK0_NUMA is unset or set incorrectly, vllm will not bind to numa! VLLM_RANK0_NUMA = -1
INFO 12-14 14:54:54 [worker_base.py:654] ########## 575 process(rank0) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191}
INFO 12-14 14:54:54 [worker_base.py:655] ########## 575 process(rank0) is running on memnode(s): {0, 1, 2, 3, 4, 5, 6, 7}
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:54 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_0771107b'), local_subscribe_addr='ipc:///tmp/24e9d6e8-633b-4269-9226-1943c671f069', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:54 [__init__.py:1170] Found nccl from library librccl.so.1
(VllmWorker rank=1 pid=576) INFO 12-14 14:54:54 [__init__.py:1170] Found nccl from library librccl.so.1
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:54 [pynccl.py:70] vLLM is using nccl==2.18.3
(VllmWorker rank=1 pid=576) INFO 12-14 14:54:54 [pynccl.py:70] vLLM is using nccl==2.18.3
Error opening NUMA node file: No such file or directory
Error opening NUMA node file: No such file or directory
(VllmWorker rank=1 pid=576) WARNING 12-14 14:54:55 [custom_all_reduce.py:146] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=0 pid=575) WARNING 12-14 14:54:55 [custom_all_reduce.py:146] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:55 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_7169de94'), local_subscribe_addr='ipc:///tmp/89da0d03-6cdb-4281-a251-06992be867a8', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=1 pid=576) INFO 12-14 14:54:55 [parallel_state.py:1077] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:55 [parallel_state.py:1077] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorker rank=1 pid=576) INFO 12-14 14:54:55 [gpu_model_runner.py:1788] Starting to load model /ssddata/aimodel/Qwen3-8B...
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:55 [gpu_model_runner.py:1788] Starting to load model /ssddata/aimodel/Qwen3-8B...
(VllmWorker rank=1 pid=576) INFO 12-14 14:54:55 [gpu_model_runner.py:1793] Loading model from scratch...
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:55 [gpu_model_runner.py:1793] Loading model from scratch...
(VllmWorker rank=1 pid=576) INFO 12-14 14:54:55 [rocm.py:279] Using Flash Attention backend on V1 engine. (only supports block size 64)
(VllmWorker rank=0 pid=575) INFO 12-14 14:54:55 [rocm.py:279] Using Flash Attention backend on V1 engine. (only supports block size 64)
Loading safetensors checkpoint shards:   0% Completed | 0/5 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  20% Completed | 1/5 [00:07<00:31,  7.83s/it]
Loading safetensors checkpoint shards:  40% Completed | 2/5 [00:13<00:20,  6.80s/it]
Loading safetensors checkpoint shards:  60% Completed | 3/5 [00:20<00:13,  6.74s/it]
Loading safetensors checkpoint shards:  80% Completed | 4/5 [00:22<00:04,  4.64s/it]
(VllmWorker rank=1 pid=576) INFO 12-14 14:55:22 [default_loader.py:272] Loading weights took 27.08 seconds
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:27<00:00,  4.91s/it]
Loading safetensors checkpoint shards: 100% Completed | 5/5 [00:27<00:00,  5.48s/it]
(VllmWorker rank=0 pid=575) 
(VllmWorker rank=0 pid=575) INFO 12-14 14:55:23 [default_loader.py:272] Loading weights took 27.54 seconds
(VllmWorker rank=1 pid=576) INFO 12-14 14:55:23 [gpu_model_runner.py:1819] Model loading took 7.6394 GiB and 27.322518 seconds
(VllmWorker rank=0 pid=575) INFO 12-14 14:55:23 [gpu_model_runner.py:1819] Model loading took 7.6394 GiB and 27.773470 seconds
(VllmWorker rank=0 pid=575) INFO 12-14 14:55:27 [gpu_worker.py:239] Available KV cache memory: 47.97 GiB
(VllmWorker rank=1 pid=576) INFO 12-14 14:55:27 [gpu_worker.py:239] Available KV cache memory: 47.97 GiB
INFO 12-14 14:55:28 [kv_cache_utils.py:716] GPU KV cache size: 698,624 tokens
INFO 12-14 14:55:28 [kv_cache_utils.py:720] Maximum concurrency for 40,960 tokens per request: 17.06x
INFO 12-14 14:55:28 [kv_cache_utils.py:716] GPU KV cache size: 698,624 tokens
INFO 12-14 14:55:28 [kv_cache_utils.py:720] Maximum concurrency for 40,960 tokens per request: 17.06x
INFO 12-14 14:55:28 [core.py:174] init engine (profile, create kv cache, warmup model) took 4.84 seconds
INFO 12-14 14:55:29 [loggers.py:137] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 10916
WARNING 12-14 14:55:29 [config.py:1408] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
INFO 12-14 14:55:29 [serving_chat.py:125] Using default chat sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 12-14 14:55:29 [serving_completion.py:72] Using default completion sampling params from model: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
INFO 12-14 14:55:29 [api_server.py:1457] Starting vLLM API server 0 on http://0.0.0.0:15888
INFO 12-14 14:55:29 [launcher.py:29] Available routes are:
INFO 12-14 14:55:29 [launcher.py:37] Route: /openapi.json, Methods: GET, HEAD
INFO 12-14 14:55:29 [launcher.py:37] Route: /docs, Methods: GET, HEAD
INFO 12-14 14:55:29 [launcher.py:37] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 12-14 14:55:29 [launcher.py:37] Route: /redoc, Methods: GET, HEAD
INFO 12-14 14:55:29 [launcher.py:37] Route: /health, Methods: GET
INFO 12-14 14:55:29 [launcher.py:37] Route: /load, Methods: GET
INFO 12-14 14:55:29 [launcher.py:37] Route: /ping, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /ping, Methods: GET
INFO 12-14 14:55:29 [launcher.py:37] Route: /tokenize, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /detokenize, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/models, Methods: GET
INFO 12-14 14:55:29 [launcher.py:37] Route: /version, Methods: GET
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/chat/completions, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/completions, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/embeddings, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /pooling, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /classify, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /score, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/score, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/audio/transcriptions, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/audio/translations, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /rerank, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v1/rerank, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /v2/rerank, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /invocations, Methods: POST
INFO 12-14 14:55:29 [launcher.py:37] Route: /metrics, Methods: GET
INFO:     Started server process [53]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO 12-14 14:55:37 [chat_utils.py:444] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
INFO 12-14 14:55:37 [logger.py:43] Received request chatcmpl-1d6df3109e434eacb0438b7c2c50f9d9: prompt: '<|im_start|>user\n你好<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=800, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None.
INFO:     172.17.0.2:50556 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO 12-14 14:55:37 [async_llm.py:270] Added request chatcmpl-1d6df3109e434eacb0438b7c2c50f9d9.
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 320, in execute_model
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1396, in execute_model
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     model_output = self.model(
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3.py", line 311, in forward
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 179, in __call__
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self.forward(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 379, in forward
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     hidden_states, residual = layer(
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522] WorkerProc hit an exception.
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3.py", line 224, in forward
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522] Traceback (most recent call last):
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     hidden_states = self.self_attn(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 517, in worker_busy_loop
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     output = func(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 320, in execute_model
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3.py", line 155, in forward
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     output = self.model_runner.execute_model(scheduler_output,
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     attn_output = self.attn(q, k, v)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return func(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1396, in execute_model
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     model_output = self.model(
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/layer.py", line 243, in forward
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     torch.ops.vllm.unified_attention_with_output(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1116, in __call__
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._op(*args, **(kwargs or {}))
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3.py", line 311, in forward
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/layer.py", line 452, in unified_attention_with_output
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     hidden_states = self.model(input_ids, positions, intermediate_tensors,
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     self.impl.forward(self,
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 179, in __call__
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/attention/backends/flash_attn.py", line 644, in forward
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self.forward(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     vllm_flash_attn_varlen_func(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen2.py", line 379, in forward
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 1199, in vllm_flash_attn_varlen_func
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     hidden_states, residual = layer(
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.vllm_mha_varlen_fwd(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522] RuntimeError: HIP error: invalid device function
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522] For debugging consider passing AMD_SERIALIZE_KERNEL=3
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3.py", line 224, in forward
(VllmWorker rank=1 pid=576) ERROR 12-14 14:55:37 [multiproc_executor.py:522] 
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     hidden_states = self.self_attn(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3.py", line 155, in forward
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     attn_output = self.attn(q, k, v)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._call_impl(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return forward_call(*args, **kwargs)
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/layer.py", line 243, in forward
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     torch.ops.vllm.unified_attention_with_output(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1116, in __call__
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     return self._op(*args, **(kwargs or {}))
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/attention/layer.py", line 452, in unified_attention_with_output
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     self.impl.forward(self,
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/attention/backends/flash_attn.py", line 644, in forward
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     vllm_flash_attn_varlen_func(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]   File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 1199, in vllm_flash_attn_varlen_func
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522]     out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = flash_attn_cuda.vllm_mha_varlen_fwd(
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522] RuntimeError: HIP error: invalid device function
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522] For debugging consider passing AMD_SERIALIZE_KERNEL=3
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
(VllmWorker rank=0 pid=575) ERROR 12-14 14:55:37 [multiproc_executor.py:522] 
ERROR 12-14 14:55:37 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.9.2) with config: model='/ssddata/aimodel/Qwen3-8B', speculative_config=None, tokenizer='/ssddata/aimodel/Qwen3-8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-8B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}, 
ERROR 12-14 14:55:37 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=chatcmpl-1d6df3109e434eacb0438b7c2c50f9d9,prompt_token_ids_len=9,mm_inputs=[],mm_hashes=[],mm_positions=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=800, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([1],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[], resumed_from_preemption=[], new_token_ids=[], new_block_ids=[], num_computed_tokens=[]), num_scheduled_tokens={chatcmpl-1d6df3109e434eacb0438b7c2c50f9d9: 9}, total_num_scheduled_tokens=9, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[1], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
ERROR 12-14 14:55:37 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, kv_cache_usage=0.00018321729571268985, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=9, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
ERROR 12-14 14:55:37 [core.py:591] EngineCore encountered a fatal error.
ERROR 12-14 14:55:37 [core.py:591] Traceback (most recent call last):
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 582, in run_engine_core
ERROR 12-14 14:55:37 [core.py:591]     engine_core.run_busy_loop()
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 609, in run_busy_loop
ERROR 12-14 14:55:37 [core.py:591]     self._process_engine_step()
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 634, in _process_engine_step
ERROR 12-14 14:55:37 [core.py:591]     outputs, model_executed = self.step_fn()
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 239, in step
ERROR 12-14 14:55:37 [core.py:591]     model_output = self.execute_model(scheduler_output)
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 223, in execute_model
ERROR 12-14 14:55:37 [core.py:591]     raise err
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 214, in execute_model
ERROR 12-14 14:55:37 [core.py:591]     return self.model_executor.execute_model(scheduler_output)
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 158, in execute_model
ERROR 12-14 14:55:37 [core.py:591]     (output, ) = self.collective_rpc(
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc
ERROR 12-14 14:55:37 [core.py:591]     result = get_response(w, dequeue_timeout)
ERROR 12-14 14:55:37 [core.py:591]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 202, in get_response
ERROR 12-14 14:55:37 [core.py:591]     raise RuntimeError(
ERROR 12-14 14:55:37 [core.py:591] RuntimeError: Worker failed with error 'HIP error: invalid device function
ERROR 12-14 14:55:37 [core.py:591] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
ERROR 12-14 14:55:37 [core.py:591] For debugging consider passing AMD_SERIALIZE_KERNEL=3
ERROR 12-14 14:55:37 [core.py:591] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.
ERROR 12-14 14:55:37 [core.py:591] ', please check the stack trace above for the root cause
ERROR 12-14 14:55:37 [async_llm.py:419] AsyncLLM output_handler failed.
ERROR 12-14 14:55:37 [async_llm.py:419] Traceback (most recent call last):
ERROR 12-14 14:55:37 [async_llm.py:419]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 378, in output_handler
ERROR 12-14 14:55:37 [async_llm.py:419]     outputs = await engine_core.get_output_async()
ERROR 12-14 14:55:37 [async_llm.py:419]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 740, in get_output_async
ERROR 12-14 14:55:37 [async_llm.py:419]     raise self._format_exception(outputs) from None
ERROR 12-14 14:55:37 [async_llm.py:419] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO 12-14 14:55:37 [async_llm.py:345] Request chatcmpl-1d6df3109e434eacb0438b7c2c50f9d9 failed (engine dead).
ERROR 12-14 14:55:37 [serving_chat.py:948] Error in chat completion stream generator.
ERROR 12-14 14:55:37 [serving_chat.py:948] Traceback (most recent call last):
ERROR 12-14 14:55:37 [serving_chat.py:948]   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 518, in chat_completion_stream_generator
ERROR 12-14 14:55:37 [serving_chat.py:948]     async for res in result_generator:
ERROR 12-14 14:55:37 [serving_chat.py:948]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 326, in generate
ERROR 12-14 14:55:37 [serving_chat.py:948]     out = q.get_nowait() or await q.get()
ERROR 12-14 14:55:37 [serving_chat.py:948]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
ERROR 12-14 14:55:37 [serving_chat.py:948]     raise output
ERROR 12-14 14:55:37 [serving_chat.py:948]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 378, in output_handler
ERROR 12-14 14:55:37 [serving_chat.py:948]     outputs = await engine_core.get_output_async()
ERROR 12-14 14:55:37 [serving_chat.py:948]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 740, in get_output_async
ERROR 12-14 14:55:37 [serving_chat.py:948]     raise self._format_exception(outputs) from None
ERROR 12-14 14:55:37 [serving_chat.py:948] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [53]