Qwen3.5-35B-A3B-Channel-INT8-w8a8 在BW100单卡 vllm 0.21无法启动，帮忙看下什么问题 https://github.com/HYGON-AI/inference-cookbook-das/blob/main/docs/model-deployment/vllm/qwen3.5.md#qwen35-35b-a3b-channel-int8-w8a8-ifb-bw1000-1x-vllm-021 docker run --rm --name vllm-hcu --network=g7_hy_network -p 8000:8000 --ipc=host --shm-size=16G --device=/dev/kfd --device=/dev/mkfd --device=/dev/dri -v /opt/hyhal:/opt/hyhal -v /mnt/Dev/model_vlm:/root/models --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -e HIP_VISIBLE_DEVICES=1 harbor.sourcefind.cn:5443/hcu/admin/base/vllm:0.21.0-ubuntu22.04-dtk2604-py3.10 vllm serve /root/models/Qwen3.5-35B-A3B-Channel-INT8-w8a8 --served-model-name Qwen3.5-35B-A3B-Channel-INT8-w8a8 --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.9 -tp 1 --trust-remote-code --max-num-batched-tokens 16384 -q slimquant_marlin --speculative-config.method mtp --speculative-config.num_speculative_tokens 3 --speculative-config.quantization slimquant_marlin --attention-backend FLASH_ATTN_CUSTOM --max-num-seqs 128 --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 mba page size. (EngineCore pid=1275) INFO 09-19 12:13:07 [interface.py:669] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. (EngineCore pid=1275) INFO:vllm_hcu.v1.hcu_model_runner:Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size. (EngineCore pid=1275) INFO 09-19 12:13:23 [backends.py:1089] Using cache directory: /root/.cache/vllm/torch_compile_cache/3ebdb94d36/rank_0_0/backbone for vLLM's torch.compile (EngineCore pid=1275) INFO 09-19 12:13:23 [backends.py:1148] Dynamo bytecode transform time: 11.26 s (EngineCore pid=1275) INFO 09-19 12:13:46 [backends.py:378] Cache the graph of compile range (1, 16384) for later use (EngineCore pid=1275) INFO 09-19 12:18:18 [backends.py:393] Compiling a graph for compile range (1, 16384) takes 295.20 s (EngineCore pid=1275) INFO 09-19 12:18:24 [decorators.py:708] saved AOT compiled function to /root/.cache/vllm/torch_compile_cache/torch_aot_compile/df298745ea01a3ef0721af6d2050ba5cc745945ef44e62c87674b3a26a5747e9/rank_0_0/model (EngineCore pid=1275) INFO 09-19 12:18:24 [monitor.py:53] torch.compile took 312.24 s in total (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] EngineCore failed to start. (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] Traceback (most recent call last): (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1182, in run_engine_core (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return func(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 948, in __init__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] super().__init__( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 129, in __init__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return func(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 251, in _initialize_kv_caches (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self.collective_rpc("determine_available_memory") (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] result = run_method(self.driver_worker, method, args, kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return func(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return func(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] self.model_runner.profile_run() (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/v1/hcu_model_runner.py", line 6387, in profile_run (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] hidden_states, last_hidden_states = self._dummy_run( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return func(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/v1/hcu_model_runner.py", line 6045, in _dummy_run (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] outputs = self.model( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/cuda_graph.py", line 254, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self.runnable(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self._call_impl(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return forward_call(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_5.py", line 658, in forward (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] hidden_states = self.language_model.model( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 670, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] output = self.aot_compiled_fn(self, *args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/aot_compile.py", line 124, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self.fn(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 503, in forward (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] def forward( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/caching.py", line 217, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self.optimized_call(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "", line 187, in execution_fn (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/cuda_graph.py", line 254, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self.runnable(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/piecewise_backend.py", line 380, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return range_entry.runnable(*args) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/standalone_compile.py", line 122, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self._compiled_fn(*args) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return fn(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py", line 1148, in forward (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return compiled_fn(full_args) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 357, in runtime_wrapper (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] all_outs = call_func_at_runtime_with_args( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 134, in call_func_at_runtime_with_args (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] out = normalize_as_list(f(args)) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1962, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self.compiled_fn(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 531, in wrapper (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return compiled_fn(runtime_args) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 729, in inner_fn (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] outs = compiled_fn(args) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/output_code.py", line 638, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self.current_callable(inputs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/utils.py", line 3220, in run (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] out = model(new_inputs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/root/.cache/vllm/torch_compile_cache/torch_aot_compile/df298745ea01a3ef0721af6d2050ba5cc745945ef44e62c87674b3a26a5747e9/inductor_cache/qf/cqfplwbws3pd7qxvjshae6wa6dj2iwkzp5oudydqwsgf4saqsuiu.py", line 1112, in call (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] buf13 = torch.ops.vllm.moe_forward_shared.default(buf10, buf11, buf12, None, None, None, None, None, 'language_model.model.layers.0.mlp.experts', 0) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 819, in __call__ (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return self._op(*args, **kwargs) (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 152, in _moe_forward_shared (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return layer.runner._forward_impl( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 915, in _forward_impl (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] shared_output, hidden_states = self._apply_quant_method( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 625, in _apply_quant_method (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] fused_out = self._quant_method.apply( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe_marlin.py", line 946, in apply (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] return fused_experts_impl_int8_marlin( (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] File "/usr/local/lib/python3.10/dist-packages/lightop/_lmslim_native/layers/fused_moe/int8_marlin.py", line 276, in fused_experts_impl_int8_marlin (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] assert status, f'moe marlin unsupport this size E:{E}, N:{N}, K:{K}' (EngineCore pid=1275) ERROR 09-19 12:19:36 [core.py:1208] AssertionError: moe marlin unsupport this size E:256, N:512, K:2048 (EngineCore pid=1275) Process EngineCore: (EngineCore pid=1275) Traceback (most recent call last): (EngineCore pid=1275) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap (EngineCore pid=1275) self.run() (EngineCore pid=1275) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run (EngineCore pid=1275) self._target(*self._args, **self._kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1212, in run_engine_core (EngineCore pid=1275) raise e (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1182, in run_engine_core (EngineCore pid=1275) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1275) return func(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 948, in __init__ (EngineCore pid=1275) super().__init__( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 129, in __init__ (EngineCore pid=1275) kv_cache_config = self._initialize_kv_caches(vllm_config) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=1275) return func(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 251, in _initialize_kv_caches (EngineCore pid=1275) available_gpu_memory = self.model_executor.determine_available_memory() (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory (EngineCore pid=1275) return self.collective_rpc("determine_available_memory") (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/uniproc_executor.py", line 93, in collective_rpc (EngineCore pid=1275) result = run_method(self.driver_worker, method, args, kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method (EngineCore pid=1275) return func(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=1275) return func(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory (EngineCore pid=1275) self.model_runner.profile_run() (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/v1/hcu_model_runner.py", line 6387, in profile_run (EngineCore pid=1275) hidden_states, last_hidden_states = self._dummy_run( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context (EngineCore pid=1275) return func(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/v1/hcu_model_runner.py", line 6045, in _dummy_run (EngineCore pid=1275) outputs = self.model( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/cuda_graph.py", line 254, in __call__ (EngineCore pid=1275) return self.runnable(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1776, in _wrapped_call_impl (EngineCore pid=1275) return self._call_impl(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1787, in _call_impl (EngineCore pid=1275) return forward_call(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_5.py", line 658, in forward (EngineCore pid=1275) hidden_states = self.language_model.model( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/decorators.py", line 670, in __call__ (EngineCore pid=1275) output = self.aot_compiled_fn(self, *args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/aot_compile.py", line 124, in __call__ (EngineCore pid=1275) return self.fn(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 503, in forward (EngineCore pid=1275) def forward( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/caching.py", line 217, in __call__ (EngineCore pid=1275) return self.optimized_call(*args, **kwargs) (EngineCore pid=1275) File "", line 187, in execution_fn (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/cuda_graph.py", line 254, in __call__ (EngineCore pid=1275) return self.runnable(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/compilation/piecewise_backend.py", line 380, in __call__ (EngineCore pid=1275) return range_entry.runnable(*args) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/standalone_compile.py", line 122, in __call__ (EngineCore pid=1275) return self._compiled_fn(*args) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn (EngineCore pid=1275) return fn(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/aot_autograd.py", line 1148, in forward (EngineCore pid=1275) return compiled_fn(full_args) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 357, in runtime_wrapper (EngineCore pid=1275) all_outs = call_func_at_runtime_with_args( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 134, in call_func_at_runtime_with_args (EngineCore pid=1275) out = normalize_as_list(f(args)) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1962, in __call__ (EngineCore pid=1275) return self.compiled_fn(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 531, in wrapper (EngineCore pid=1275) return compiled_fn(runtime_args) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 729, in inner_fn (EngineCore pid=1275) outs = compiled_fn(args) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/output_code.py", line 638, in __call__ (EngineCore pid=1275) return self.current_callable(inputs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/utils.py", line 3220, in run (EngineCore pid=1275) out = model(new_inputs) (EngineCore pid=1275) File "/root/.cache/vllm/torch_compile_cache/torch_aot_compile/df298745ea01a3ef0721af6d2050ba5cc745945ef44e62c87674b3a26a5747e9/inductor_cache/qf/cqfplwbws3pd7qxvjshae6wa6dj2iwkzp5oudydqwsgf4saqsuiu.py", line 1112, in call (EngineCore pid=1275) buf13 = torch.ops.vllm.moe_forward_shared.default(buf10, buf11, buf12, None, None, None, None, None, 'language_model.model.layers.0.mlp.experts', 0) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 819, in __call__ (EngineCore pid=1275) return self._op(*args, **kwargs) (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 152, in _moe_forward_shared (EngineCore pid=1275) return layer.runner._forward_impl( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 915, in _forward_impl (EngineCore pid=1275) shared_output, hidden_states = self._apply_quant_method( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 625, in _apply_quant_method (EngineCore pid=1275) fused_out = self._quant_method.apply( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/vllm_hcu/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe_marlin.py", line 946, in apply (EngineCore pid=1275) return fused_experts_impl_int8_marlin( (EngineCore pid=1275) File "/usr/local/lib/python3.10/dist-packages/lightop/_lmslim_native/layers/fused_moe/int8_marlin.py", line 276, in fused_experts_impl_int8_marlin (EngineCore pid=1275) assert status, f'moe marlin unsupport this size E:{E}, N:{N}, K:{K}' (EngineCore pid=1275) AssertionError: moe marlin unsupport this size E:256, N:512, K:2048 WARNING: Logging before InitGoogleLogging() is written to STDERR W0919 12:19:38.086376 1275 ProcessGroupNCCL.cpp:1548] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) (APIServer pid=16) Traceback (most recent call last): (APIServer pid=16) File "/usr/local/bin/vllm", line 6, in (APIServer pid=16) sys.exit(main()) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 92, in main (APIServer pid=16) args.dispatch_function(args) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd (APIServer pid=16) uvloop.run(run_server(args)) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 69, in run (APIServer pid=16) return loop.run_until_complete(wrapper()) (APIServer pid=16) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 48, in wrapper (APIServer pid=16) return await main (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 693, in run_server (APIServer pid=16) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 707, in run_server_worker (APIServer pid=16) async with build_async_engine_client( (APIServer pid=16) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ (APIServer pid=16) return await anext(self.gen) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client (APIServer pid=16) async with build_async_engine_client_from_engine_args( (APIServer pid=16) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ (APIServer pid=16) return await anext(self.gen) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args (APIServer pid=16) async_llm = AsyncLLM.from_vllm_config( (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config (APIServer pid=16) return cls( (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 146, in __init__ (APIServer pid=16) self.engine_core = EngineCoreClient.make_async_mp_client( (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=16) return func(*args, **kwargs) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client (APIServer pid=16) return AsyncMPClient(*client_args) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (APIServer pid=16) return func(*args, **kwargs) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 900, in __init__ (APIServer pid=16) super().__init__( (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 535, in __init__ (APIServer pid=16) with launch_core_engines( (APIServer pid=16) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ (APIServer pid=16) next(self.gen) (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 1128, in launch_core_engines (APIServer pid=16) wait_for_engine_startup( (APIServer pid=16) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 1187, in wait_for_engine_startup (APIServer pid=16) raise RuntimeError( (APIServer pid=16) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}