0
1
0
0
K100-AI 推理qwen3-30B-A3B-W8A8量化报错，求指导
2026-06-07 15:04:24
196
使用论坛提供的custom-vllm018镜像推理qwen3-30B量化模型报错，官方0.15.1镜像也不行，DTK26.04，求教当前是否支持qwen3量化版本。
sudo docker run --rm
-it
--name=vllm_test
--device=/dev/kfd
--device=/dev/dri/card1 --device=/dev/dri/renderD128
--device=/dev/mkfd
--security-opt seccomp=unconfined
--cap-add=SYS_PTRACE
--ipc=host
--network host
--shm-size=128G
--group-add 39
-v /opt/hyhal:/opt/hyhal
-v /home/apt/new_disk/LLM_models:/models
-v /home/apt/code:/code
harbor.sourcefind.cn:5443/dcu/admin/base/custom:vllm018-ubuntu22.04-dtk26.04-gemma4-0413-k100ai /bin/bash
vllm serve /models/qwen3-30B-A3B
--host 0.0.0.0
--port 8000
--max-model-len 4096
--served-model-name qwen3-30B-A3B
--tensor-parallel-size 1
--gpu-memory-utilization 0.9
--max-num-seqs 20
--trust-remote-code
--enforce-eager
--
楼主
0 点赞
0 帖子
楼主热帖
暂无数据常见问题新帖
暂无数据
(APIServer pid=985) INFO 06-05 16:08:34 [utils.py:233] non-default args: {'model_tag': '/models/qwen3-30B-A3B', 'host': '0.0.0.0', 'model': '/models/qwen3-30B-A3B', 'trust_remote_code': True, 'max_model_len': 4096, 'enforce_eager': True, 'served_model_name': ['qwen3-30B-A3B'], 'enable_expert_parallel': True, 'disable_custom_all_reduce': True, 'max_num_seqs': 20}
(APIServer pid=985) INFO 06-05 16:08:35 [compilation.py:289] Enabled custom fusions: norm_quant, act_quant
/qwen3-30B-A3B(EngineCore pid=1257) INFO 06-05 16:08:55 [core.py:103] Initializing a V1 LLM engine (v0.18.1) with config: model='/models/qwen3-30B-A3B', speculative_config=None, tokenizer='/models/qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3-30B-A3B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+sparse_attn_indexer', 'all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
(EngineCore pid=1257) INFO 06-05 16:08:58 [rocm.py:496] Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN'].
(EngineCore pid=1257) INFO 06-05 16:08:58 [unquantized.py:186] Using TRITON backend for Unquantized MoE
(EngineCore pid=1257) WARNING 06-05 16:08:58 [compilation.py:1136] Op 'sparse_attn_indexer' not present in model, enabling with '+sparse_attn_indexer' has no effect
Loading safetensors checkpoint shards: 0% Completed | 0/8 [00:00<?, ?it/s]
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] EngineCore failed to start.
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] Traceback (most recent call last):
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] return func(*args, **kwargs)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] super().__init__(
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 112, in __init__
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] self.model_executor = executor_class(vllm_config)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] return func(*args, **kwargs)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] self._init_executor()
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/uniproc_executor.py", line 50, in _init_executor
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] self.driver_worker.load_model()
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 335, in load_model
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] self.model_runner.load_model(load_dummy_weights=dummy_weights)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] return func(*args, **kwargs)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4497, in load_model
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] self.model = model_loader.load_model(
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] return func(*args, **kwargs)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 62, in load_model
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] self.load_weights(model, model_config)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] return func(*args, **kwargs)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 381, in load_weights
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 782, in load_weights
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] return loader.load_weights(weights)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] return original_load_weights(self, weights, *args, **kwargs)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 355, in load_weights
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 302, in _load_module
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] yield from self._load_module(
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 275, in _load_module
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] loaded_params = module_load_weights(weights)
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 630, in load_weights
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] param = params_dict[name_mapped]
(EngineCore pid=1257) ERROR 06-05 16:08:59 [core.py:1099] KeyError: 'layers.0.mlp.experts.w2_weight_offset'
(EngineCore pid=1257) Process EngineCore:
(EngineCore pid=1257) Traceback (most recent call last):
(EngineCore pid=1257) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=1257) self.run()
(EngineCore pid=1257) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore pid=1257) self._target(*self._args, **self._kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1103, in run_engine_core
(EngineCore pid=1257) raise e
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 1073, in run_engine_core
(EngineCore pid=1257) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) return func(*args, **kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 839, in __init__
(EngineCore pid=1257) super().__init__(
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 112, in __init__
(EngineCore pid=1257) self.model_executor = executor_class(vllm_config)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) return func(*args, **kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 103, in __init__
(EngineCore pid=1257) self._init_executor()
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/uniproc_executor.py", line 50, in _init_executor
(EngineCore pid=1257) self.driver_worker.load_model()
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 335, in load_model
(EngineCore pid=1257) self.model_runner.load_model(load_dummy_weights=dummy_weights)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) return func(*args, **kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4497, in load_model
(EngineCore pid=1257) self.model = model_loader.load_model(
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) return func(*args, **kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 62, in load_model
(EngineCore pid=1257) self.load_weights(model, model_config)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1257) return func(*args, **kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 381, in load_weights
(EngineCore pid=1257) loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 782, in load_weights
(EngineCore pid=1257) return loader.load_weights(weights)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(EngineCore pid=1257) return original_load_weights(self, weights, *args, **kwargs)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 355, in load_weights
(EngineCore pid=1257) autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 302, in _load_module
(EngineCore pid=1257) yield from self._load_module(
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 275, in _load_module
(EngineCore pid=1257) loaded_params = module_load_weights(weights)
(EngineCore pid=1257) File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_moe.py", line 630, in load_weights
(EngineCore pid=1257) param = params_dict[name_mapped]
(EngineCore pid=1257) KeyError: 'layers.0.mlp.experts.w2_weight_offset'
Loading safetensors checkpoint shards: 0% Completed | 0/8 [00:01<?, ?it/s]
(EngineCore pid=1257)
W0605 16:09:01.534838 1257 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=985) Traceback (most recent call last):
(APIServer pid=985) File "/usr/local/bin/vllm", line 6, in <module>
(APIServer pid=985) sys.exit(main())
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=985) args.dispatch_function(args)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 118, in cmd
(APIServer pid=985) uvloop.run(run_server(args))
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 69, in run
(APIServer pid=985) return loop.run_until_complete(wrapper())
(APIServer pid=985) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=985) return await main
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 656, in run_server
(APIServer pid=985) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 670, in run_server_worker
(APIServer pid=985) async with build_async_engine_client(
(APIServer pid=985) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
(APIServer pid=985) return await anext(self.gen)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 103, in build_async_engine_client
(APIServer pid=985) async with build_async_engine_client_from_engine_args(
(APIServer pid=985) File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
(APIServer pid=985) return await anext(self.gen)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 144, in build_async_engine_client_from_engine_args
(APIServer pid=985) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=985) return cls(
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=985) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=985) return func(*args, **kwargs)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=985) return AsyncMPClient(*client_args)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=985) return func(*args, **kwargs)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=985) super().__init__(
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=985) with launch_core_engines(
(APIServer pid=985) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
(APIServer pid=985) next(self.gen)
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=985) wait_for_engine_startup(
(APIServer pid=985) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=985) raise RuntimeError(
(APIServer pid=985) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}