pytorch version 2.9.1

1 DCU were detected in the container.

NOTE: Make sure the user is in the 39 group.

For more information, please go to https://developer.sourcefind.cn/

=== 确认环境变量 ===

VLLM_USE_V1:

HSA_OVERRIDE_GFX_VERSION: 9.2.8

=== [2] 启动 vLLM ===

(APIServer pid=1) INFO 03-09 17:03:53 [utils.py:325]

(APIServer pid=1) INFO 03-09 17:03:53 [utils.py:325]

(APIServer pid=1) INFO 03-09 17:03:53 [utils.py:325] version 0.15.1

(APIServer pid=1) INFO 03-09 17:03:53 [utils.py:325] model /models/Qwen3.5-9B

(APIServer pid=1) INFO 03-09 17:03:53 [utils.py:325]

(APIServer pid=1) INFO 03-09 17:03:53 [utils.py:325]

(APIServer pid=1) INFO 03-09 17:03:53 [utils.py:261] non-default args: {'model_tag': '/models/Qwen3.5-9B', 'api_server_count': 1, 'host': '0.0.0.0', 'model': '/models/Qwen3.5-9B', 'trust_remote_code': True, 'max_model_len': 4096, 'served_model_name': ['qwen3.5-9b'], 'gpu_memory_utilization': 0.5}

(APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.

(APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.

(APIServer pid=1) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}

(APIServer pid=1) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}

(APIServer pid=1) INFO 03-09 17:04:24 [model.py:548] Resolved architecture: Qwen3_5ForConditionalGeneration

(APIServer pid=1) INFO 03-09 17:04:24 [model.py:1573] Using max model len 4096

(APIServer pid=1) INFO 03-09 17:04:25 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=10240.

(APIServer pid=1) INFO 03-09 17:04:25 [config.py:504] Setting attention block size to 272 tokens to ensure that attention page size is >= mamba page size.

(APIServer pid=1) INFO 03-09 17:04:25 [config.py:535] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal.

(APIServer pid=1) INFO 03-09 17:04:25 [vllm.py:624] Asynchronous scheduling is enabled.

(APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.

(APIServer pid=1) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}

(APIServer pid=1) Unrecognized keys in `rope_parameters` for 'rope_type'='default': {'mrope_interleaved', 'mrope_section'}

(EngineCore_DP0 pid=476) INFO 03-09 17:04:57 [core.py:96] Initializing a V1 LLM engine (v0.15.1) with config: model='/models/Qwen3.5-9B', speculative_config=None, tokenizer='/models/Qwen3.5-9B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.5-9b, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none', '+sparse_attn_indexer'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [10240], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.PIECEWISE: 1>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None, 'static_all_moe_layers': []}

(EngineCore_DP0 pid=476) INFO 03-09 17:05:01 [parallel_state.py:1216] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.20.0.2:36271 backend=nccl

(EngineCore_DP0 pid=476) INFO 03-09 17:05:01 [parallel_state.py:1427] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A

Could not open /var/log/hylog/.

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] EngineCore failed to start.

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] Traceback (most recent call last):

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 937, in run_engine_core

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 691, in __init__

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] super().__init__(

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 105, in __init__

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.model_executor = executor_class(vllm_config)

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self._init_executor()

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.driver_worker.init_device()

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.worker.init_device() # type: ignore

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 262, in init_device

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device)

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 510, in __init__

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.input_batch = InputBatch(

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_input_batch.py", line 148, in __init__

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.block_table = MultiGroupBlockTable(

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 290, in __init__

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.block_tables = [

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 291, in <listcomp>

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] BlockTable(

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 68, in __init__

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.block_table = self._make_buffer(

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 248, in _make_buffer

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] return CpuGpuBuffer(

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/utils.py", line 118, in __init__

(EngineCore_DP0 pid=476) Process EngineCore_DP0:

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] self.gpu = torch.zeros_like(self.cpu, device=device)

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] torch.AcceleratorError: HIP error: invalid device function

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] For debugging consider passing AMD_SERIALIZE_KERNEL=3

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946] Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

(EngineCore_DP0 pid=476) ERROR 03-09 17:05:02 [core.py:946]

(EngineCore_DP0 pid=476) Traceback (most recent call last):

(EngineCore_DP0 pid=476) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap

(EngineCore_DP0 pid=476) self.run()

(EngineCore_DP0 pid=476) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run

(EngineCore_DP0 pid=476) self._target(*self._args, **self._kwargs)

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 950, in run_engine_core

(EngineCore_DP0 pid=476) raise e

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 937, in run_engine_core

(EngineCore_DP0 pid=476) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 691, in __init__

(EngineCore_DP0 pid=476) super().__init__(

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 105, in __init__

(EngineCore_DP0 pid=476) self.model_executor = executor_class(vllm_config)

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 101, in __init__

(EngineCore_DP0 pid=476) self._init_executor()

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/uniproc_executor.py", line 47, in _init_executor

(EngineCore_DP0 pid=476) self.driver_worker.init_device()

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/worker_base.py", line 326, in init_device

(EngineCore_DP0 pid=476) self.worker.init_device() # type: ignore

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 262, in init_device

(EngineCore_DP0 pid=476) self.model_runner = GPUModelRunnerV1(self.vllm_config, self.device)

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 510, in __init__

(EngineCore_DP0 pid=476) self.input_batch = InputBatch(

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_input_batch.py", line 148, in __init__

(EngineCore_DP0 pid=476) self.block_table = MultiGroupBlockTable(

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 290, in __init__

(EngineCore_DP0 pid=476) self.block_tables = [

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 291, in <listcomp>

(EngineCore_DP0 pid=476) BlockTable(

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 68, in __init__

(EngineCore_DP0 pid=476) self.block_table = self._make_buffer(

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/block_table.py", line 248, in _make_buffer

(EngineCore_DP0 pid=476) return CpuGpuBuffer(

(EngineCore_DP0 pid=476) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/utils.py", line 118, in __init__

(EngineCore_DP0 pid=476) self.gpu = torch.zeros_like(self.cpu, device=device)

(EngineCore_DP0 pid=476) torch.AcceleratorError: HIP error: invalid device function

(EngineCore_DP0 pid=476) Search for `hipErrorInvalidDeviceFunction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html for more information.

(EngineCore_DP0 pid=476) HIP kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

(EngineCore_DP0 pid=476) For debugging consider passing AMD_SERIALIZE_KERNEL=3

(EngineCore_DP0 pid=476) Compile with `TORCH_USE_HIP_DSA` to enable device-side assertions.

(EngineCore_DP0 pid=476)

WARNING: Logging before InitGoogleLogging() is written to STDERR

W0309 17:05:03.840453 476 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

(APIServer pid=1) Traceback (most recent call last):

(APIServer pid=1) File "/usr/local/bin/vllm", line 6, in <module>

(APIServer pid=1) sys.exit(main())

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main

(APIServer pid=1) args.dispatch_function(args)

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 111, in cmd

(APIServer pid=1) uvloop.run(run_server(args))

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 69, in run

(APIServer pid=1) return loop.run_until_complete(wrapper())

(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 48, in wrapper

(APIServer pid=1) return await main

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 919, in run_server

(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 938, in run_server_worker

(APIServer pid=1) async with build_async_engine_client(

(APIServer pid=1) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__

(APIServer pid=1) return await anext(self.gen)

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 147, in build_async_engine_client

(APIServer pid=1) async with build_async_engine_client_from_engine_args(

(APIServer pid=1) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__

(APIServer pid=1) return await anext(self.gen)

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 188, in build_async_engine_client_from_engine_args

(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config(

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 228, in from_vllm_config

(APIServer pid=1) return cls(

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 155, in __init__

(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client(

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client

(APIServer pid=1) return AsyncMPClient(*client_args)

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 819, in __init__

(APIServer pid=1) super().__init__(

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 479, in __init__

(APIServer pid=1) with launch_core_engines(vllm_config, executor_class, log_stats) as (

(APIServer pid=1) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__

(APIServer pid=1) next(self.gen)

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 933, in launch_core_engines

(APIServer pid=1) wait_for_engine_startup(

(APIServer pid=1) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 992, in wait_for_engine_startup

(APIServer pid=1) raise RuntimeError(

(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}