在论坛里有个大佬分享的启动命令如下：

docker run -dit --name=dcm38 --network=host --privileged --shm-size 60g -v /opt/hyhal:/opt/hyhal:ro --device=/dev/kfd --device=/dev/dri --device=/dev/mkfd --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -u root -v /home/xts/models:/models -v /home/xts/bf/dtk-26.04:/dtk-26.04 -e HIP_VISIBLE_DEVICES=0,1 dcu/admin/base/vllm:0.18.1-ubuntu22.04-dtk26.04-py3.10  bash

vllm serve /models/Qwen3.8-27B --host 0.0.0.0 -tp 2 --max-model-len 33792 --max-num-seq 16 --gpu-memory-utilization 0.95 --block-size 16 --speculative-config.method mtp --speculative-config.num_speculative_tokens 3

使用这个命令 dcm38容器运行正常，vllm报错，请官方或大佬指点下。

INFO 09-03 01:53:35 [__init__.py:44] Available plugins for group vllm.platform_plugins:

INFO 09-03 01:53:35 [__init__.py:46] - hcu -> vllm_hcu:hcu_platform_plugin

INFO 09-03 01:53:35 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.

INFO 09-03 01:53:35 [__init__.py:239] Platform plugin hcu is activated

[aiter] WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'"

WARNING 09-03 01:54:33 [registry.py:911] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

WARNING 09-03 01:54:33 [registry.py:911] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

WARNING 09-03 01:54:33 [registry.py:911] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_mtp:DeepSeekMTP.

WARNING 09-03 01:54:33 [registry.py:911] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:GlmMoeDsaForCausalLM.

WARNING 09-03 01:54:33 [registry.py:911] Model architecture Glm4MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.glm4_moe:Glm4MoeForCausalLM.

(APIServer pid=28) INFO 09-03 01:54:34 [utils.py:297]

(APIServer pid=28) INFO 09-03 01:54:34 [utils.py:297]        █     █     █▄   ▄█

(APIServer pid=28) INFO 09-03 01:54:34 [utils.py:297]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.18.1

(APIServer pid=28) INFO 09-03 01:54:34 [utils.py:297]   █▄█▀ █     █     █     █  model   /models/Qwen3.8-27B

(APIServer pid=28) INFO 09-03 01:54:34 [utils.py:297]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀

(APIServer pid=28) INFO 09-03 01:54:34 [utils.py:297]

(APIServer pid=28) INFO 09-03 01:54:34 [utils.py:233] non-default args: {'model_tag': '/models/Qwen3.8-27B', 'host': '0.0.0.0', 'model': '/models/Qwen3.8-27B', 'max_model_len': 33792, 'tensor_parallel_size': 2, 'block_size': 16, 'gpu_memory_utilization': 0.95, 'max_num_seqs': 16, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}

(APIServer pid=28) `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead

(APIServer pid=28) `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead

(APIServer pid=28) `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead

(APIServer pid=28) INFO 09-03 01:55:12 [model.py:533] Resolved architecture: Qwen3_5ForConditionalGeneration

(APIServer pid=28) INFO 09-03 01:55:12 [model.py:1585] Using max model len 33792

(APIServer pid=28) `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead

(APIServer pid=28) `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead

(APIServer pid=28) `layer_type_validation` is deprecated and will be removed in v5.20. Use `PreTrainedConfig.validate_layer_type` instead

(APIServer pid=28) INFO 09-03 01:55:50 [model.py:533] Resolved architecture: Qwen3_5MTP

(APIServer pid=28) INFO 09-03 01:55:50 [model.py:1585] Using max model len 262144

(APIServer pid=28) WARNING 09-03 01:55:50 [speculative.py:557] num_speculative_tokens (3) exceeds available MTP layers (1); later draft steps will reuse MTP layers, which may lower acceptance rate.

(APIServer pid=28) INFO 09-03 01:55:50 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.

(APIServer pid=28) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.

(APIServer pid=28) INFO 09-03 01:55:50 [config.py:227] Setting attention block size to 800 tokens to ensure that attention page size is >= mamba page size.

(APIServer pid=28) INFO 09-03 01:55:50 [config.py:258] Padding mamba page size by 0.25% to ensure that mamba page size and attention page size are exactly equal.

(APIServer pid=28) INFO 09-03 01:55:50 [vllm.py:776] Asynchronous scheduling is enabled.

(APIServer pid=28) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.

INFO 09-03 01:56:08 [__init__.py:44] Available plugins for group vllm.platform_plugins:

INFO 09-03 01:56:08 [__init__.py:46] - hcu -> vllm_hcu:hcu_platform_plugin

INFO 09-03 01:56:08 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.

INFO 09-03 01:56:08 [__init__.py:239] Platform plugin hcu is activated

[aiter] WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'"

(EngineCore pid=902) WARNING 09-03 01:56:36 [registry.py:911] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

(EngineCore pid=902) WARNING 09-03 01:56:36 [registry.py:911] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

(EngineCore pid=902) WARNING 09-03 01:56:36 [registry.py:911] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_mtp:DeepSeekMTP.

(EngineCore pid=902) WARNING 09-03 01:56:36 [registry.py:911] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:GlmMoeDsaForCausalLM.

(EngineCore pid=902) WARNING 09-03 01:56:36 [registry.py:911] Model architecture Glm4MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.glm4_moe:Glm4MoeForCausalLM.

(EngineCore pid=902) INFO 09-03 01:56:37 [core.py:104] Initializing a V1 LLM engine (v0.18.1) with config: model='/models/Qwen3.8-27B', speculative_config=SpeculativeConfig(method='mtp', model='/models/Qwen3.8-27B', num_spec_tokens=3, enable_multi_layers_mtp=False), tokenizer='/models/Qwen3.8-27B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=33792, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=/models/Qwen3.8-27B, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}

(EngineCore pid=902) WARNING 09-03 01:56:37 [multiproc_executor.py:997] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.

(EngineCore pid=902) /usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py:133: UserWarning: Failed to get the IP address, using 0.0.0.0 by default.The value can be set by the environment variable VLLM_HOST_IP or HOST_IP.

(EngineCore pid=902)   mq_connect_ip = get_ip()

(EngineCore pid=902) INFO 09-03 01:56:37 [multiproc_executor.py:134] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=0.0.0.0 (local), world_size=2, local_world_size=2

INFO 09-03 01:56:40 [__init__.py:44] Available plugins for group vllm.platform_plugins:

INFO 09-03 01:56:40 [__init__.py:46] - hcu -> vllm_hcu:hcu_platform_plugin

INFO 09-03 01:56:40 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.

INFO 09-03 01:56:40 [__init__.py:44] Available plugins for group vllm.platform_plugins:

INFO 09-03 01:56:40 [__init__.py:46] - hcu -> vllm_hcu:hcu_platform_plugin

INFO 09-03 01:56:40 [__init__.py:49] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.

INFO 09-03 01:56:40 [__init__.py:239] Platform plugin hcu is activated

INFO 09-03 01:56:40 [__init__.py:239] Platform plugin hcu is activated

[aiter] WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'"

[aiter] WARNING: NUMA balancing is enabled, which may cause errors. It is recommended to disable NUMA balancing by running "sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'"

WARNING 09-03 01:57:07 [registry.py:911] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_mtp:DeepSeekMTP.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:GlmMoeDsaForCausalLM.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture Glm4MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.glm4_moe:Glm4MoeForCausalLM.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture DeepseekV32ForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:DeepseekV3ForCausalLM.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_mtp:DeepSeekMTP.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture GlmMoeDsaForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.deepseek_v2:GlmMoeDsaForCausalLM.

WARNING 09-03 01:57:07 [registry.py:911] Model architecture Glm4MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_hcu.models.glm4_moe:Glm4MoeForCausalLM.

`Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.

`Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.

(Worker pid=1169) INFO 09-03 01:57:11 [parallel_state.py:1395] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:60225 backend=nccl

(Worker pid=1170) INFO 09-03 01:57:11 [parallel_state.py:1395] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:60225 backend=nccl

(Worker pid=1169) INFO 09-03 01:57:12 [pynccl.py:111] vLLM is using nccl==2.22.3

(Worker pid=1170) WARNING:vllm_hcu.distributed.device_communicators.custom_all_reduce:We are using PCIe's custom allreduce.If the performance is poor, we can add --disable-custom-all-reduce in the instruction.

(Worker pid=1169) WARNING:vllm_hcu.distributed.device_communicators.custom_all_reduce:We are using PCIe's custom allreduce.If the performance is poor, we can add --disable-custom-all-reduce in the instruction.

Could not open /var/log/hylog/.

Could not open /var/log/hylog/.

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852] WorkerProc failed to start.

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852] Traceback (most recent call last):

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_main

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     worker = WorkerProc(*args, **kwargs)

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     return func(*args, **kwargs)

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 614, in __init__

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852] WorkerProc failed to start.

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.worker.init_device()

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852] Traceback (most recent call last):

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/worker_base.py", line 312, in init_device

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 821, in worker_main

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.worker.init_device()  # type: ignore

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     worker = WorkerProc(*args, **kwargs)

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/v1/worker.py", line 106, in init_device

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     init_worker_distributed_environment(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     return func(*args, **kwargs)

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 1061, in init_worker_distributed_environment

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 614, in __init__

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     ensure_model_parallel_initialized(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.worker.init_device()

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1749, in ensure_model_parallel_initialized

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/worker_base.py", line 312, in init_device

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     initialize_model_parallel(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.worker.init_device()  # type: ignore

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1564, in initialize_model_parallel

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/v1/worker.py", line 106, in init_device

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     _TP = init_model_parallel_group(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     init_worker_distributed_environment(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1154, in init_model_parallel_group

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 1061, in init_worker_distributed_environment

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     return GroupCoordinator(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     ensure_model_parallel_initialized(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 373, in __init__

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1749, in ensure_model_parallel_initialized

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.device_communicator = device_comm_cls(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     initialize_model_parallel(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 101, in __init__

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1564, in initialize_model_parallel

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.ca_comm = CustomAllreduce(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     _TP = init_model_parallel_group(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 175, in __init__

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 1154, in init_model_parallel_group

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.meta_ptrs = self.create_shared_buffer(

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     return GroupCoordinator(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 313, in create_shared_buffer

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/parallel_state.py", line 373, in __init__

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     pointers.append(torch.ops.hcu_ops.open_mem_handle(h))

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.device_communicator = device_comm_cls(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1209, in __call__

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 101, in __init__

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     return self._op(*args, **kwargs)

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.ca_comm = CustomAllreduce(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852] torch.AcceleratorError: HIP error: invalid device pointer

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 175, in __init__

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852] Search for `hipErrorInvalidDevicePointer' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     self.meta_ptrs = self.create_shared_buffer(

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 313, in create_shared_buffer

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852] For debugging consider passing AMD_SERIALIZE_KERNEL=3

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     pointers.append(torch.ops.hcu_ops.open_mem_handle(h))

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]   File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1209, in __call__

(Worker pid=1170) ERROR 09-03 01:57:13 [multiproc_executor.py:852]

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]     return self._op(*args, **kwargs)

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852] torch.AcceleratorError: HIP error: invalid device pointer

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852] Search for `hipErrorInvalidDevicePointer' in https://rocm.docs.amd.com/projects/HIP/en/latest/index.html for more information.

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852] For debugging consider passing AMD_SERIALIZE_KERNEL=3

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

(Worker pid=1169) ERROR 09-03 01:57:13 [multiproc_executor.py:852]

(Worker pid=1170) Exception ignored in: <function CustomAllreduce.__del__ at 0x77de88043640>

(Worker pid=1170) Traceback (most recent call last):

(Worker pid=1170)   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 293, in __del__

(Worker pid=1169) Exception ignored in: <function CustomAllreduce.__del__ at 0x7e8e847b3640>

(Worker pid=1169) Traceback (most recent call last):

(Worker pid=1169)   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 293, in __del__

(Worker pid=1170)     self.close()

(Worker pid=1170)   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 285, in close

(Worker pid=1169)     self.close()

(Worker pid=1169)   File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/distributed/device_communicators/custom_all_reduce.py", line 285, in close

(Worker pid=1170)     if not self.disabled and self._ptr:

(Worker pid=1170) AttributeError: 'CustomAllreduce' object has no attribute '_ptr'

(Worker pid=1169)     if not self.disabled and self._ptr:

(Worker pid=1169) AttributeError: 'CustomAllreduce' object has no attribute '_ptr'

WARNING: Logging before InitGoogleLogging() is written to STDERR

W0903 01:57:15.091552  1169 ProcessGroupNCCL.cpp:1548] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100] EngineCore failed to start.

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100] Traceback (most recent call last):

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1074, in run_engine_core

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     return func(*args, **kwargs)

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 840, in __init__

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     super().__init__(

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 113, in __init__

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     self.model_executor = executor_class(vllm_config)

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 101, in __init__

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     super().__init__(vllm_config)

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     return func(*args, **kwargs)

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 103, in __init__

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     self._init_executor()

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 190, in _init_executor

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     self.workers = WorkerProc.wait_for_ready(unready_workers)

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 731, in wait_for_ready

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100]     raise e from None

(EngineCore pid=902) ERROR 09-03 01:57:18 [core.py:1100] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.

(EngineCore pid=902) Process EngineCore:

(EngineCore pid=902) Traceback (most recent call last):

(EngineCore pid=902)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap

(EngineCore pid=902)     self.run()

(EngineCore pid=902)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run

(EngineCore pid=902)     self._target(*self._args, **self._kwargs)

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1104, in run_engine_core

(EngineCore pid=902)     raise e

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1074, in run_engine_core

(EngineCore pid=902)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(EngineCore pid=902)     return func(*args, **kwargs)

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 840, in __init__

(EngineCore pid=902)     super().__init__(

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 113, in __init__

(EngineCore pid=902)     self.model_executor = executor_class(vllm_config)

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 101, in __init__

(EngineCore pid=902)     super().__init__(vllm_config)

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(EngineCore pid=902)     return func(*args, **kwargs)

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 103, in __init__

(EngineCore pid=902)     self._init_executor()

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 190, in _init_executor

(EngineCore pid=902)     self.workers = WorkerProc.wait_for_ready(unready_workers)

(EngineCore pid=902)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 731, in wait_for_ready

(EngineCore pid=902)     raise e from None

(EngineCore pid=902) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.

(APIServer pid=28) Traceback (most recent call last):

(APIServer pid=28)   File "/usr/local/bin/vllm", line 6, in <module>

(APIServer pid=28)     sys.exit(main())

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main

(APIServer pid=28)     args.dispatch_function(args)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 118, in cmd

(APIServer pid=28)     uvloop.run(run_server(args))

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 69, in run

(APIServer pid=28)     return loop.run_until_complete(wrapper())

(APIServer pid=28)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 48, in wrapper

(APIServer pid=28)     return await main

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 656, in run_server

(APIServer pid=28)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker

(APIServer pid=28)     async with build_async_engine_client(

(APIServer pid=28)   File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__

(APIServer pid=28)     return await anext(self.gen)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client

(APIServer pid=28)     async with build_async_engine_client_from_engine_args(

(APIServer pid=28)   File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__

(APIServer pid=28)     return await anext(self.gen)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args

(APIServer pid=28)     async_llm = AsyncLLM.from_vllm_config(

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config

(APIServer pid=28)     return cls(

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 154, in __init__

(APIServer pid=28)     self.engine_core = EngineCoreClient.make_async_mp_client(

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(APIServer pid=28)     return func(*args, **kwargs)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client

(APIServer pid=28)     return AsyncMPClient(*client_args)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper

(APIServer pid=28)     return func(*args, **kwargs)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 924, in __init__

(APIServer pid=28)     super().__init__(

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 583, in __init__

(APIServer pid=28)     with launch_core_engines(

(APIServer pid=28)   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__

(APIServer pid=28)     next(self.gen)

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 972, in launch_core_engines

(APIServer pid=28)     wait_for_engine_startup(

(APIServer pid=28)   File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup

(APIServer pid=28)     raise RuntimeError(

(APIServer pid=28) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}