光合开发者社区-论坛

nohup: ignoring input INFO 01-13 08:30:17 [__init__.py:217] Automatically detected platform rocm. [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:24 [api_server.py:1839] vLLM API server version 0.11.0 [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:24 [utils.py:233] non-default args: {'model_tag': '/root/models/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', 'port': 9102, 'model': '/root/models/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', 'trust_remote_code': True, 'max_model_len': 4096, 'served_model_name': ['Qwen3-Next-80B-A3B-Instruct-AWQ-4bit'], 'gpu_memory_utilization': 0.92, 'enable_prefix_caching': False, 'max_num_batched_tokens': 4096, 'max_num_seqs': 64, 'enable_chunked_prefill': False} [1;36m(APIServer pid=67)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:24 [model.py:555] Resolved architecture: Qwen3NextForCausalLM [1;36m(APIServer pid=67)[0;0m `torch_dtype` is deprecated! Use `dtype` instead! [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:24 [model.py:1520] Using max model len 4096 [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:24 [config.py:297] Hybrid or mamba-based model detected: disabling prefix caching since it is not yet supported. [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:24 [config.py:308] Hybrid or mamba-based model detected: setting cudagraph mode to FULL_AND_PIECEWISE in order to optimize performance. [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:25 [config.py:376] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size. [1;36m(APIServer pid=67)[0;0m INFO 01-13 08:30:25 [config.py:397] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. [1;36m(APIServer pid=67)[0;0m The tokenizer you are loading from '/root/models/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue. INFO 01-13 08:30:28 [__init__.py:217] Automatically detected platform rocm. [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:34 [core.py:644] Waiting for init message from front-end. [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:34 [core.py:78] Initializing a V1 LLM engine (v0.11.0) with config: model='/root/models/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', speculative_config=None, tokenizer='/root/models/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-Next-80B-A3B-Instruct-AWQ-4bit, enable_prefix_caching=False, chunked_prefill_enabled=False, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":128,"local_cache_dir":null} [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:34 [worker_base.py:287] ########## 295 process(rank0) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15} [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:34 [worker_base.py:288] ########## 295 process(rank0) is running on memnode(s): {0} [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:34 [parallel_state.py:1212] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:36 [gpu_model_runner.py:2608] Starting to load model /root/models/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit... [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:37 [gpu_model_runner.py:2640] Loading model from scratch... [1;36m(EngineCore_DP0 pid=295)[0;0m `torch_dtype` is deprecated! Use `dtype` instead! [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:37 [compressed_tensors_moe.py:128] Using CompressedTensorsWNA16MoEMethod [1;36m(EngineCore_DP0 pid=295)[0;0m INFO 01-13 08:30:37 [rocm.py:266] Using Triton backend on V1 engine. [1;36m(EngineCore_DP0 pid=295)[0;0m Loading safetensors checkpoint shards: 0% Completed | 0/10 [00:00<?, ?it/s] [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] EngineCore failed to start. [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] Traceback (most recent call last): [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 699, in run_engine_core [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] engine_core = EngineCoreProc(*args, **kwargs) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 498, in __init__ [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] super().__init__(vllm_config, executor_class, log_stats, [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 84, in __init__ [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] self.model_executor = executor_class(vllm_config) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 54, in __init__ [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] self._init_executor() [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 55, in _init_executor [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] self.collective_rpc("load_model") [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] return [run_method(self.driver_worker, method, args, kwargs)] [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/utils/__init__.py", line 3291, in run_method [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] return func(*args, **kwargs) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 213, in load_model [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] self.model_runner.load_model(eep_scale_up=eep_scale_up) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2641, in load_model [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] self.model = model_loader.load_model( [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] self.load_weights(model, model_config) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 264, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] loaded_weights = model.load_weights( [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 1185, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] return loader.load_weights(weights) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 294, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] autoloaded_weights = set(self._load_module("", self.module, weights)) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 252, in _load_module [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] yield from self._load_module(prefix, [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 225, in _load_module [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] loaded_params = module_load_weights(weights) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 1024, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] weight_loader(param, loaded_weight) [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/mamba/mamba_mixer2.py", line 193, in loader [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] param.data[ [1;36m(EngineCore_DP0 pid=295)[0;0m ERROR 01-13 08:30:39 [core.py:708] RuntimeError: The expanded size of the tensor (8192) must match the existing size (4) at non-singleton dimension 2. Target sizes: [4, 1, 8192]. Tensor sizes: [2048, 1, 4] [1;36m(EngineCore_DP0 pid=295)[0;0m Process EngineCore_DP0: [1;36m(EngineCore_DP0 pid=295)[0;0m Traceback (most recent call last): [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [1;36m(EngineCore_DP0 pid=295)[0;0m self.run() [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [1;36m(EngineCore_DP0 pid=295)[0;0m self._target(*self._args, **self._kwargs) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 712, in run_engine_core [1;36m(EngineCore_DP0 pid=295)[0;0m raise e [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 699, in run_engine_core [1;36m(EngineCore_DP0 pid=295)[0;0m engine_core = EngineCoreProc(*args, **kwargs) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 498, in __init__ [1;36m(EngineCore_DP0 pid=295)[0;0m super().__init__(vllm_config, executor_class, log_stats, [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 84, in __init__ [1;36m(EngineCore_DP0 pid=295)[0;0m self.model_executor = executor_class(vllm_config) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 54, in __init__ [1;36m(EngineCore_DP0 pid=295)[0;0m self._init_executor() [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 55, in _init_executor [1;36m(EngineCore_DP0 pid=295)[0;0m self.collective_rpc("load_model") [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc [1;36m(EngineCore_DP0 pid=295)[0;0m return [run_method(self.driver_worker, method, args, kwargs)] [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/utils/__init__.py", line 3291, in run_method [1;36m(EngineCore_DP0 pid=295)[0;0m return func(*args, **kwargs) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 213, in load_model [1;36m(EngineCore_DP0 pid=295)[0;0m self.model_runner.load_model(eep_scale_up=eep_scale_up) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2641, in load_model [1;36m(EngineCore_DP0 pid=295)[0;0m self.model = model_loader.load_model( [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model [1;36m(EngineCore_DP0 pid=295)[0;0m self.load_weights(model, model_config) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 264, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m loaded_weights = model.load_weights( [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 1185, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m return loader.load_weights(weights) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 294, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m autoloaded_weights = set(self._load_module("", self.module, weights)) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 252, in _load_module [1;36m(EngineCore_DP0 pid=295)[0;0m yield from self._load_module(prefix, [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 225, in _load_module [1;36m(EngineCore_DP0 pid=295)[0;0m loaded_params = module_load_weights(weights) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 1024, in load_weights [1;36m(EngineCore_DP0 pid=295)[0;0m weight_loader(param, loaded_weight) [1;36m(EngineCore_DP0 pid=295)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/mamba/mamba_mixer2.py", line 193, in loader [1;36m(EngineCore_DP0 pid=295)[0;0m param.data[ [1;36m(EngineCore_DP0 pid=295)[0;0m RuntimeError: The expanded size of the tensor (8192) must match the existing size (4) at non-singleton dimension 2. Target sizes: [4, 1, 8192]. Tensor sizes: [2048, 1, 4] [1;36m(EngineCore_DP0 pid=295)[0;0m Loading safetensors checkpoint shards: 0% Completed | 0/10 [00:01<?, ?it/s] [1;36m(EngineCore_DP0 pid=295)[0;0m WARNING: Logging before InitGoogleLogging() is written to STDERR W0113 08:30:41.337307 295 ProcessGroupNCCL.cpp:1279] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [1;36m(APIServer pid=67)[0;0m Traceback (most recent call last): [1;36m(APIServer pid=67)[0;0m File "/usr/local/bin/vllm", line 7, in <module> [1;36m(APIServer pid=67)[0;0m sys.exit(main()) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main [1;36m(APIServer pid=67)[0;0m args.dispatch_function(args) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 57, in cmd [1;36m(APIServer pid=67)[0;0m uvloop.run(run_server(args)) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 69, in run [1;36m(APIServer pid=67)[0;0m return loop.run_until_complete(wrapper()) [1;36m(APIServer pid=67)[0;0m File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 48, in wrapper [1;36m(APIServer pid=67)[0;0m return await main [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1884, in run_server [1;36m(APIServer pid=67)[0;0m await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1902, in run_server_worker [1;36m(APIServer pid=67)[0;0m async with build_async_engine_client( [1;36m(APIServer pid=67)[0;0m File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ [1;36m(APIServer pid=67)[0;0m return await anext(self.gen) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 180, in build_async_engine_client [1;36m(APIServer pid=67)[0;0m async with build_async_engine_client_from_engine_args( [1;36m(APIServer pid=67)[0;0m File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ [1;36m(APIServer pid=67)[0;0m return await anext(self.gen) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 225, in build_async_engine_client_from_engine_args [1;36m(APIServer pid=67)[0;0m async_llm = AsyncLLM.from_vllm_config( [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/utils/__init__.py", line 1586, in inner [1;36m(APIServer pid=67)[0;0m return fn(*args, **kwargs) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 207, in from_vllm_config [1;36m(APIServer pid=67)[0;0m return cls( [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 134, in __init__ [1;36m(APIServer pid=67)[0;0m self.engine_core = EngineCoreClient.make_async_mp_client( [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client [1;36m(APIServer pid=67)[0;0m return AsyncMPClient(*client_args) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 769, in __init__ [1;36m(APIServer pid=67)[0;0m super().__init__( [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 448, in __init__ [1;36m(APIServer pid=67)[0;0m with launch_core_engines(vllm_config, executor_class, [1;36m(APIServer pid=67)[0;0m File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ [1;36m(APIServer pid=67)[0;0m next(self.gen) [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 732, in launch_core_engines [1;36m(APIServer pid=67)[0;0m wait_for_engine_startup( [1;36m(APIServer pid=67)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup [1;36m(APIServer pid=67)[0;0m raise RuntimeError("Engine core initialization failed. " [1;36m(APIServer pid=67)[0;0m RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

nohup: ignoring input INFO 01-13 08:38:38 [__init__.py:217] Automatically detected platform rocm. [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [api_server.py:1839] vLLM API server version 0.11.0 [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [utils.py:233] non-default args: {'model_tag': '/root/models/Qwen3-Next-80B-A3B-Instruct', 'port': 9102, 'model': '/root/models/Qwen3-Next-80B-A3B-Instruct', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 4096, 'served_model_name': ['Qwen3-Next-80B-A3B-Instruct'], 'tensor_parallel_size': 4, 'block_size': 128, 'gpu_memory_utilization': 0.8, 'enable_prefix_caching': False, 'max_num_batched_tokens': 4096, 'max_num_seqs': 64, 'enable_chunked_prefill': False} [1;36m(APIServer pid=696)[0;0m The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [model.py:555] Resolved architecture: Qwen3NextForCausalLM [1;36m(APIServer pid=696)[0;0m `torch_dtype` is deprecated! Use `dtype` instead! [1;36m(APIServer pid=696)[0;0m WARNING 01-13 08:38:44 [model.py:1743] Casting torch.bfloat16 to torch.float16. [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [model.py:1520] Using max model len 4096 [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [config.py:297] Hybrid or mamba-based model detected: disabling prefix caching since it is not yet supported. [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [config.py:308] Hybrid or mamba-based model detected: setting cudagraph mode to FULL_AND_PIECEWISE in order to optimize performance. [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [config.py:376] Setting attention block size to 272 tokens to ensure that attention page size is >= mamba page size. [1;36m(APIServer pid=696)[0;0m INFO 01-13 08:38:44 [config.py:397] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal. INFO 01-13 08:38:48 [__init__.py:217] Automatically detected platform rocm. [1;36m(EngineCore_DP0 pid=924)[0;0m INFO 01-13 08:38:54 [core.py:644] Waiting for init message from front-end. [1;36m(EngineCore_DP0 pid=924)[0;0m INFO 01-13 08:38:54 [core.py:78] Initializing a V1 LLM engine (v0.11.0) with config: model='/root/models/Qwen3-Next-80B-A3B-Instruct', speculative_config=None, tokenizer='/root/models/Qwen3-Next-80B-A3B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-Next-80B-A3B-Instruct, enable_prefix_caching=False, chunked_prefill_enabled=False, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":128,"local_cache_dir":null} [1;36m(EngineCore_DP0 pid=924)[0;0m WARNING 01-13 08:38:54 [multiproc_executor.py:720] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. [1;36m(EngineCore_DP0 pid=924)[0;0m INFO 01-13 08:38:54 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 16777216, 10, 'psm_92c084da'), local_subscribe_addr='ipc:///tmp/9ecaf01a-fa7d-42db-af01-78e0c8932a5c', remote_subscribe_addr=None, remote_addr_ipv6=False) INFO 01-13 08:38:57 [__init__.py:217] Automatically detected platform rocm. INFO 01-13 08:38:57 [__init__.py:217] Automatically detected platform rocm. INFO 01-13 08:38:57 [__init__.py:217] Automatically detected platform rocm. INFO 01-13 08:38:57 [__init__.py:217] Automatically detected platform rocm. INFO 01-13 08:39:03 [worker_base.py:287] ########## 1216 process(rank2) is running on CPU(s): {32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47} INFO 01-13 08:39:03 [worker_base.py:288] ########## 1216 process(rank2) is running on memnode(s): {2} INFO 01-13 08:39:03 [worker_base.py:287] ########## 1215 process(rank1) is running on CPU(s): {16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31} INFO 01-13 08:39:03 [worker_base.py:288] ########## 1215 process(rank1) is running on memnode(s): {1} INFO 01-13 08:39:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_80fbe412'), local_subscribe_addr='ipc:///tmp/8eea5e2a-f0ca-4da1-a426-ec9d56777c30', remote_subscribe_addr=None, remote_addr_ipv6=False) INFO 01-13 08:39:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_cf88c5b3'), local_subscribe_addr='ipc:///tmp/928f3c31-8967-403e-8f9b-b972fd12564a', remote_subscribe_addr=None, remote_addr_ipv6=False) INFO 01-13 08:39:03 [worker_base.py:287] ########## 1214 process(rank0) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15} INFO 01-13 08:39:03 [worker_base.py:288] ########## 1214 process(rank0) is running on memnode(s): {0} INFO 01-13 08:39:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_baf17a69'), local_subscribe_addr='ipc:///tmp/50245094-3aa6-4996-95d3-3482841377eb', remote_subscribe_addr=None, remote_addr_ipv6=False) INFO 01-13 08:39:04 [worker_base.py:287] ########## 1217 process(rank3) is running on CPU(s): {48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63} INFO 01-13 08:39:04 [worker_base.py:288] ########## 1217 process(rank3) is running on memnode(s): {3} INFO 01-13 08:39:04 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_a07e59a9'), local_subscribe_addr='ipc:///tmp/8718adee-8cd1-41a6-8382-83bc702086c2', remote_subscribe_addr=None, remote_addr_ipv6=False) INFO 01-13 08:39:04 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:04 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:04 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:04 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:04 [pynccl.py:103] vLLM is using nccl==2.22.3 INFO 01-13 08:39:04 [pynccl.py:103] vLLM is using nccl==2.22.3 INFO 01-13 08:39:04 [pynccl.py:103] vLLM is using nccl==2.22.3 INFO 01-13 08:39:04 [pynccl.py:103] vLLM is using nccl==2.22.3 WARNING 01-13 08:39:05 [custom_all_reduce.py:153] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. WARNING 01-13 08:39:05 [custom_all_reduce.py:153] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. WARNING 01-13 08:39:05 [custom_all_reduce.py:153] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. WARNING 01-13 08:39:05 [custom_all_reduce.py:153] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. INFO 01-13 08:39:05 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_673c1daa'), local_subscribe_addr='ipc:///tmp/9abee234-7d0f-4f27-a16d-b65eca61e92c', remote_subscribe_addr=None, remote_addr_ipv6=False) INFO 01-13 08:39:05 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:05 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:05 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:05 [pynccl.py:103] vLLM is using nccl==2.22.3 INFO 01-13 08:39:05 [pynccl.py:103] vLLM is using nccl==2.22.3 INFO 01-13 08:39:05 [pynccl.py:103] vLLM is using nccl==2.22.3 INFO 01-13 08:39:05 [__init__.py:1398] Found nccl from library librccl.so.1 INFO 01-13 08:39:05 [pynccl.py:103] vLLM is using nccl==2.22.3 INFO 01-13 08:39:06 [parallel_state.py:1212] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 INFO 01-13 08:39:06 [parallel_state.py:1212] rank 3 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3 INFO 01-13 08:39:06 [parallel_state.py:1212] rank 1 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1 INFO 01-13 08:39:06 [parallel_state.py:1212] rank 2 in world size 4 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2 [1;36m(Worker_TP3 pid=1217)[0;0m INFO 01-13 08:39:06 [gpu_model_runner.py:2608] Starting to load model /root/models/Qwen3-Next-80B-A3B-Instruct... [1;36m(Worker_TP1 pid=1215)[0;0m INFO 01-13 08:39:06 [gpu_model_runner.py:2608] Starting to load model /root/models/Qwen3-Next-80B-A3B-Instruct... [1;36m(Worker_TP2 pid=1216)[0;0m INFO 01-13 08:39:07 [gpu_model_runner.py:2608] Starting to load model /root/models/Qwen3-Next-80B-A3B-Instruct... [1;36m(Worker_TP0 pid=1214)[0;0m INFO 01-13 08:39:07 [gpu_model_runner.py:2608] Starting to load model /root/models/Qwen3-Next-80B-A3B-Instruct... [1;36m(Worker_TP3 pid=1217)[0;0m INFO 01-13 08:39:07 [gpu_model_runner.py:2640] Loading model from scratch... [1;36m(Worker_TP1 pid=1215)[0;0m INFO 01-13 08:39:07 [gpu_model_runner.py:2640] Loading model from scratch... [1;36m(Worker_TP2 pid=1216)[0;0m INFO 01-13 08:39:07 [gpu_model_runner.py:2640] Loading model from scratch... [1;36m(Worker_TP3 pid=1217)[0;0m `torch_dtype` is deprecated! Use `dtype` instead! [1;36m(Worker_TP1 pid=1215)[0;0m `torch_dtype` is deprecated! Use `dtype` instead! [1;36m(Worker_TP0 pid=1214)[0;0m INFO 01-13 08:39:07 [gpu_model_runner.py:2640] Loading model from scratch... [1;36m(Worker_TP2 pid=1216)[0;0m `torch_dtype` is deprecated! Use `dtype` instead! [1;36m(Worker_TP0 pid=1214)[0;0m `torch_dtype` is deprecated! Use `dtype` instead! [1;36m(Worker_TP3 pid=1217)[0;0m INFO 01-13 08:39:07 [rocm.py:266] Using Triton backend on V1 engine. [1;36m(Worker_TP1 pid=1215)[0;0m INFO 01-13 08:39:07 [rocm.py:266] Using Triton backend on V1 engine. [1;36m(Worker_TP2 pid=1216)[0;0m INFO 01-13 08:39:07 [rocm.py:266] Using Triton backend on V1 engine. [1;36m(Worker_TP0 pid=1214)[0;0m INFO 01-13 08:39:07 [rocm.py:266] Using Triton backend on V1 engine. [1;36m(Worker_TP0 pid=1214)[0;0m Loading safetensors checkpoint shards: 0% Completed | 0/41 [00:00<?, ?it/s] [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] WorkerProc failed to start. [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] Traceback (most recent call last): [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] worker = WorkerProc(*args, **kwargs) [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 437, in __init__ [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] self.worker.load_model() [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 213, in load_model [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] self.model_runner.load_model(eep_scale_up=eep_scale_up) [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 2641, in load_model [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] self.model = model_loader.load_model( [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 50, in load_model [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] self.load_weights(model, model_config) [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 264, in load_weights [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] loaded_weights = model.load_weights( [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 1185, in load_weights [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] return loader.load_weights(weights) [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 294, in load_weights [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] autoloaded_weights = set(self._load_module("", self.module, weights)) [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 252, in _load_module [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] yield from self._load_module(prefix, [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 225, in _load_module [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] loaded_params = module_load_weights(weights) [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/qwen3_next.py", line 1024, in load_weights [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] weight_loader(param, loaded_weight) [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/mamba/mamba_mixer2.py", line 193, in loader [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] param.data[ [1;36m(Worker_TP3 pid=1217)[0;0m ERROR 01-13 08:39:09 [multiproc_executor.py:597] RuntimeError: The expanded size of the tensor (2048) must match the existing size (4) at non-singleton dimension 2. Target sizes: [4, 1, 2048]. Tensor sizes: [512, 1, 4] [1;36m(Worker_TP3 pid=1217)[0;0m INFO 01-13 08:39:09 [multiproc_executor.py:558] Parent process exited, terminating worker [1;36m(Worker_TP1 pid=1215)[0;0m INFO 01-13 08:39:09 [multiproc_executor.py:558] Parent process exited, terminating worker [1;36m(Worker_TP0 pid=1214)[0;0m INFO 01-13 08:39:09 [multiproc_executor.py:558] Parent process exited, terminating worker [1;36m(Worker_TP0 pid=1214)[0;0m Loading safetensors checkpoint shards: 0% Completed | 0/41 [00:01<?, ?it/s] [1;36m(Worker_TP0 pid=1214)[0;0m [1;36m(Worker_TP2 pid=1216)[0;0m INFO 01-13 08:39:09 [multiproc_executor.py:558] Parent process exited, terminating worker WARNING: Logging before InitGoogleLogging() is written to STDERR W0113 08:39:11.373036 1214 ProcessGroupNCCL.cpp:1279] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] EngineCore failed to start. [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] Traceback (most recent call last): [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 699, in run_engine_core [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] engine_core = EngineCoreProc(*args, **kwargs) [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 498, in __init__ [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] super().__init__(vllm_config, executor_class, log_stats, [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 84, in __init__ [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] self.model_executor = executor_class(vllm_config) [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 54, in __init__ [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] self._init_executor() [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 106, in _init_executor [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] self.workers = WorkerProc.wait_for_ready(unready_workers) [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 509, in wait_for_ready [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] raise e from None [1;36m(EngineCore_DP0 pid=924)[0;0m ERROR 01-13 08:39:13 [core.py:708] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. [1;36m(EngineCore_DP0 pid=924)[0;0m Process EngineCore_DP0: [1;36m(EngineCore_DP0 pid=924)[0;0m Traceback (most recent call last): [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap [1;36m(EngineCore_DP0 pid=924)[0;0m self.run() [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run [1;36m(EngineCore_DP0 pid=924)[0;0m self._target(*self._args, **self._kwargs) [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 712, in run_engine_core [1;36m(EngineCore_DP0 pid=924)[0;0m raise e [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 699, in run_engine_core [1;36m(EngineCore_DP0 pid=924)[0;0m engine_core = EngineCoreProc(*args, **kwargs) [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 498, in __init__ [1;36m(EngineCore_DP0 pid=924)[0;0m super().__init__(vllm_config, executor_class, log_stats, [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 84, in __init__ [1;36m(EngineCore_DP0 pid=924)[0;0m self.model_executor = executor_class(vllm_config) [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 54, in __init__ [1;36m(EngineCore_DP0 pid=924)[0;0m self._init_executor() [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 106, in _init_executor [1;36m(EngineCore_DP0 pid=924)[0;0m self.workers = WorkerProc.wait_for_ready(unready_workers) [1;36m(EngineCore_DP0 pid=924)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 509, in wait_for_ready [1;36m(EngineCore_DP0 pid=924)[0;0m raise e from None [1;36m(EngineCore_DP0 pid=924)[0;0m Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause. [1;36m(APIServer pid=696)[0;0m Traceback (most recent call last): [1;36m(APIServer pid=696)[0;0m File "/usr/local/bin/vllm", line 7, in <module> [1;36m(APIServer pid=696)[0;0m sys.exit(main()) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 54, in main [1;36m(APIServer pid=696)[0;0m args.dispatch_function(args) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 57, in cmd [1;36m(APIServer pid=696)[0;0m uvloop.run(run_server(args)) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 69, in run [1;36m(APIServer pid=696)[0;0m return loop.run_until_complete(wrapper()) [1;36m(APIServer pid=696)[0;0m File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 48, in wrapper [1;36m(APIServer pid=696)[0;0m return await main [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1884, in run_server [1;36m(APIServer pid=696)[0;0m await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1902, in run_server_worker [1;36m(APIServer pid=696)[0;0m async with build_async_engine_client( [1;36m(APIServer pid=696)[0;0m File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ [1;36m(APIServer pid=696)[0;0m return await anext(self.gen) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 180, in build_async_engine_client [1;36m(APIServer pid=696)[0;0m async with build_async_engine_client_from_engine_args( [1;36m(APIServer pid=696)[0;0m File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__ [1;36m(APIServer pid=696)[0;0m return await anext(self.gen) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 225, in build_async_engine_client_from_engine_args [1;36m(APIServer pid=696)[0;0m async_llm = AsyncLLM.from_vllm_config( [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/utils/__init__.py", line 1586, in inner [1;36m(APIServer pid=696)[0;0m return fn(*args, **kwargs) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 207, in from_vllm_config [1;36m(APIServer pid=696)[0;0m return cls( [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 134, in __init__ [1;36m(APIServer pid=696)[0;0m self.engine_core = EngineCoreClient.make_async_mp_client( [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 102, in make_async_mp_client [1;36m(APIServer pid=696)[0;0m return AsyncMPClient(*client_args) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 769, in __init__ [1;36m(APIServer pid=696)[0;0m super().__init__( [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 448, in __init__ [1;36m(APIServer pid=696)[0;0m with launch_core_engines(vllm_config, executor_class, [1;36m(APIServer pid=696)[0;0m File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ [1;36m(APIServer pid=696)[0;0m next(self.gen) [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 732, in launch_core_engines [1;36m(APIServer pid=696)[0;0m wait_for_engine_startup( [1;36m(APIServer pid=696)[0;0m File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup [1;36m(APIServer pid=696)[0;0m raise RuntimeError("Engine core initialization failed. " [1;36m(APIServer pid=696)[0;0m RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} /usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

#!/bin/bash -x export HIP_VISIBLE_DEVICES=0 export VLLM_NUMA_BIND=1 export VLLM_RANK0_NUMA=0 export VLLM_RANK1_NUMA=1 export VLLM_RANK2_NUMA=2 export VLLM_RANK3_NUMA=3 export VLLM_RANK4_NUMA=4 export VLLM_RANK5_NUMA=5 export VLLM_RANK6_NUMA=6 export VLLM_RANK7_NUMA=7 #VLLM_USE_V1=0 export NCCL_MIN_NCHANNELS=16 export NCCL_MAX_NCHANNELS=16 export NCCL_MIN_P2P_NCHANNELS=4 export NCCL_MAX_P2P_NCHANNELS=4 export NCCL_NCHANNELS_PER_PEER=4 export NCCL_RINGS="0 1 3 2 4 6 5 7" export NCCL_P2P_NVL_CHUNKSIZE=131072 export HSA_FORCE_FINE_GRAIN_PCIE=1 #K100AI需要 export NCCL_P2P_LEVEL=SYS export NCCL_LAUNCH_MODE=GROUP export ALLREDUCE_STREAM_WITH_COMPUTE=1 #export W4A16_MOE_CUDA=1 #export VLLM_FUSED_MOE_CHUNK_SIZE=8192 export LMSLIM_USE_LIGHTOP=0 export VLLM_RPC_TIMEOUT=1800000 export VLLM_WORKER_MULTIPROC_METHOD=spawn export VLLM_ZERO_OVERHEAD=1 # zero export HIP_USE_GRAPH_QUEUE_POOL=1 export VLLM_USE_GLOBAL_CACHE13=1 export VLLM_USE_TRITON_AWQ=1 vllm serve /root/models/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit \ --trust-remote-code \ --served-model-name Qwen3-Next-80B-A3B-Instruct-AWQ-4bit \ --max-model-len 4096 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.92 \ --max-num-seqs 64 \ --block-size 64 \ --no-enable-chunked-prefill \ --max-num-batched-tokens 4096 \ --no-enable-prefix-caching \ --port 9102

#!/bin/bash -x export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export VLLM_NUMA_BIND=1 export VLLM_RANK0_NUMA=0 export VLLM_RANK1_NUMA=1 export VLLM_RANK2_NUMA=2 export VLLM_RANK3_NUMA=3 export VLLM_RANK4_NUMA=4 export VLLM_RANK5_NUMA=5 export VLLM_RANK6_NUMA=6 export VLLM_RANK7_NUMA=7 #VLLM_USE_V1=0 export NCCL_MIN_NCHANNELS=16 export NCCL_MAX_NCHANNELS=16 export NCCL_MIN_P2P_NCHANNELS=4 export NCCL_MAX_P2P_NCHANNELS=4 export NCCL_NCHANNELS_PER_PEER=4 export NCCL_RINGS="0 1 3 2 4 6 5 7" export NCCL_P2P_NVL_CHUNKSIZE=131072 export HSA_FORCE_FINE_GRAIN_PCIE=1 #K100AI需要 export NCCL_P2P_LEVEL=SYS export NCCL_LAUNCH_MODE=GROUP export ALLREDUCE_STREAM_WITH_COMPUTE=1 #export W4A16_MOE_CUDA=1 #export VLLM_FUSED_MOE_CHUNK_SIZE=8192 export LMSLIM_USE_LIGHTOP=0 export VLLM_RPC_TIMEOUT=1800000 export VLLM_WORKER_MULTIPROC_METHOD=spawn export VLLM_ZERO_OVERHEAD=1 # zero export HIP_USE_GRAPH_QUEUE_POOL=1 export VLLM_USE_GLOBAL_CACHE13=1 #export VLLM_USE_TRITON_AWQ=1 VLLM_USE_V1=0 vllm serve /root/models/Qwen3-Next-80B-A3B-Instruct \ --trust-remote-code \ --served-model-name Qwen3-Next-80B-A3B-Instruct \ --max-model-len 4096 \ --tensor-parallel-size 8 \ --gpu-memory-utilization 0.95 \ --max-num-seqs 64 \ --block-size 128 \ --no-enable-chunked-prefill \ --max-num-batched-tokens 4096 \ --no-enable-prefix-caching \ --dtype float16 \ --port 9102

ydd

2楼

你好，请教一下，现在支持Qwen3-VL-8B-Instruct的微调的多个lora模型的推理吗？vllm的版本要在0.12以上

7/29

aaatest

回复 @ydd

我没试过哎

8/2

yangzhong

1楼

https://forum.sourcefind.cn/thread/bf75705e-e481-11f0-b9e4-0242ac150003?postId=a10c7798-f01a-11f0-a600-0242ac150003
vllm0.11.0推理Qwen3-Next-80B-A3B-Instruct目前存在精度问题，还在排查解决中，可以参见上述帖子，解决后会第一时间回复

1/13

aaatest

回复 @yangzhong

应该是我部署的时候漏加环境变量了

“qwen3-next需要手动设置环境变量：

export VLLM_USE_NN=0

export TRITON_MOVE_LOAD_TOFRONT_DOT=0”，之前下载镜像的时候看到过，部署的时候给忘了，我再试下估计就可以了

1/13