光合开发者社区-论坛

数学库

8 张 K100_AI 部署 meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 报错

2025-10-21 09:42:37

126

环境： 1、DCU：8 * k100_ai 2、镜像：image.sourcefind.cn:5000/dcu/admin/base/vllm:0.9.2-ubuntu22.04-dtk25.04.1-rc5-rocblas101839-0811-das1.6-py3.10-20250913-rc1 启动命令： vllm serve \--model=/mnt/model/Llama-4-Maverick-17B-128E-Instruct-FP8 --tensor-parallel-size=8 --served-model-name=llama4-fp8 --max-model-len=32768 --gpu_memory_utilization=0.92 报错信息： ERROR 10-21 08:36:04 [multiproc_executor.py:135] Worker proc VllmWorker-1 died unexpectedly, shutting down executor. Process EngineCore_0: Traceback (most recent call last): File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 593, in run_engine_core raise e File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 580, in run_engine_core engine_core = EngineCoreProc(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 407, in init super().__init__(vllm_config, executor_class, log_stats, File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 84, in init self._initialize_kv_caches(vllm_config) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 144, in _initialize_kv_caches available_gpu_memory = self.model_executor.determine_available_memory() File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/abstract.py", line 76, in determine_available_memory output = self.collective_rpc("determine_available_memory") File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 215, in collective_rpc result = get_response(w, dequeue_timeout) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 202, in get_response raise RuntimeError( RuntimeError: Worker failed with error 'CompressedTensorsW8A8Fp8MoEMethod.apply() got an unexpected keyword argument 'use_nn_moe'', please check the stack trace above for the root cause Traceback (most recent call last): File "/usr/local/bin/vllm", line 8, in module sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/main.py", line 65, in main args.dispatch_function(args) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/cli/serve.py", line 55, in cmd uvloop.run(run_server(args)) File "/usr/local/lib/python3.10/dist-packages/uvloop/init.py", line 82, in run return loop.run_until_complete(wrapper()) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete File "/usr/local/lib/python3.10/dist-packages/uvloop/init.py", line 61, in wrapper return await main File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1431, in run_server await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 1451, in run_server_worker async with build_async_engine_client(args, client_config) as engine_client: File "/usr/lib/python3.10/contextlib.py", line 199, in aenter return await anext(self.gen) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 158, in build_async_engine_client async with build_async_engine_client_from_engine_args( File "/usr/lib/python3.10/contextlib.py", line 199, in aenter return await anext(self.gen) File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/api_server.py", line 194, in build_async_engine_client_from_engine_args async_llm = AsyncLLM.from_vllm_config( File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 162, in from_vllm_config return cls( File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/async_llm.py", line 124, in init self.engine_core = EngineCoreClient.make_async_mp_client( File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 96, in make_async_mp_client return AsyncMPClient(*client_args) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 666, in init super().init( File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core_client.py", line 403, in init with launch_core_engines(vllm_config, executor_class, File "/usr/lib/python3.10/contextlib.py", line 142, in exit next(self.gen) File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 434, in launch_core_engines wait_for_engine_startup( File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/utils.py", line 484, in wait_for_engine_startup raise RuntimeError("Engine core initialization failed. " RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

zhufzh1

1楼

您好目前K100_AI暂时不支持FP8数据类型的模型推理

2025/10/21

网

网友yfri9U

回复 @zhufzh1

您好，部署 RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16 也会报错，INT4 量化，模型地址如下：https://modelscope.cn/models/RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16/summary 启动命令： vllm serve --model=/mnt/model/Llama-4-Maverick-17B-128E-Instruct-INT4 --tensor-parallel-size=4 --served-model-name=llama4-int --max-model-len=32768 --gpu_memory_utilization=0.92 报错信息： (VllmWorker rank=3 pid=576) INFO 10-22 08:39:18 [compressed_tensors_wNa16.py:95] Using MacheteLinearKernel for CompressedTensorsWNA16 (VllmWorker rank=3 pid=576) WARNING 10-22 08:39:18 [config.py:444] MoE DP setup unable to determine quantization scheme or unsupported quantization type. This model will not run with DP enabled. (VllmWorker rank=3 pid=576) INFO 10-22 08:39:18 [compressed_tensors_moe.py:80] Using CompressedTensorsWNA16MoEMethod Loading safetensors checkpoint shards: 9% 4/44 [00:00<00:01, 31.87it/s](VllmWorker rank=1 pid=574) INFO 10-22 08:39:18 [gpu_model_runner.py:1793] Loading model from scratch... (VllmWorker rank=1 pid=574) INFO 10-22 08:39:18 [rocm.py:279] Using Flash Attention backend on V1 engine. (only supports block size 64) Loading safetensors checkpoint shards: 27% 12/44 [00:00<00:01, 31.03it/s](VllmWorker rank=1 pid=574) INFO 10-22 08:39:19 [compressed_tensors_wNa16.py:95] Using MacheteLinearKernel for CompressedTensorsWNA16 (VllmWorker rank=1 pid=574) WARNING 10-22 08:39:19 [config.py:444] MoE DP setup unable to determine quantization scheme or unsupported quantization type. This model will not run with DP enabled. (VllmWorker rank=1 pid=574) INFO 10-22 08:39:19 [compressed_tensors_moe.py:80] Using CompressedTensorsWNA16MoEMethod Loading safetensors checkpoint shards: 100% 44/44 [00:01<00:00, 29.01it/s] (VllmWorker rank=0 pid=573) INFO 10-22 08:43:07 [default_loader.py:272] Loading weights took 228.58 seconds (VllmWorker rank=2 pid=575) INFO 10-22 08:43:07 [default_loader.py:272] Loading weights took 228.93 seconds (VllmWorker rank=1 pid=574) INFO 10-22 08:43:07 [default_loader.py:272] Loading weights took 228.32 seconds (VllmWorker rank=3 pid=576) INFO 10-22 08:43:07 [default_loader.py:272] Loading weights took 228.82 seconds (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] WorkerProc failed to start. (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] Traceback (most recent call last): (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 461, in worker_main (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] worker = WorkerProc(*args, **kwargs) (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 358, in __init__ (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] self.worker.load_model() (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 192, in load_model (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] self.model_runner.load_model() (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1794, in load_model (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] self.model = model_loader.load_model( (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 42, in load_model (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] process_weights_after_loading(model, model_config, target_device) (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/utils.py", line 116, in process_weights_after_loading (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] quant_method.process_weights_after_loading(module) (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 623, in process_weights_after_loading (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] n=layer.weight.shape[0] (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1931, in __getattr__ (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] raise AttributeError( (VllmWorker rank=0 pid=573) ERROR 10-22 08:43:07 [multiproc_executor.py:487] AttributeError:

2025/10/22

网

网友yfri9U

回复 @网友yfri9U

接上面报错： (VllmWorker rank=3 pid=598) ERROR 10-22 08:59:50 [multiproc_executor.py:487] AttributeError: 'MergedColumnParallelLinear' object has no attribute 'weight' I1022 08:59:50.734885 598 ProcessGroupNCCL.cpp:1263] [PG ID 0 PG GUID 0 Rank 3] ProcessGroupNCCL destructor entered. I1022 08:59:50.734935 598 ProcessGroupNCCL.cpp:1247] [PG ID 0 PG GUID 0 Rank 3] Launching ProcessGroupNCCL abort asynchrounously. I1022 08:59:50.735107 598 ProcessGroupNCCL.cpp:1133] [PG ID 0 PG GUID 0 Rank 3] future is successfully executed for: ProcessGroup abort I1022 08:59:50.735111 598 ProcessGroupNCCL.cpp:1254] [PG ID 0 PG GUID 0 Rank 3] ProcessGroupNCCL aborts successfully. I1022 08:59:50.735138 598 ProcessGroupNCCL.cpp:1284] [PG ID 0 PG GUID 0 Rank 3] ProcessGroupNCCL watchdog thread joined. I1022 08:59:50.735160 598 ProcessGroupNCCL.cpp:1288] [PG ID 0 PG GUID 0 Rank 3] ProcessGroupNCCL heart beat monitor thread joined. I1022 08:59:53.404451 597 ProcessGroupNCCL.cpp:1263] [PG ID 0 PG GUID 0 Rank 2] ProcessGroupNCCL destructor entered. I1022 08:59:53.404501 597 ProcessGroupNCCL.cpp:1247] [PG ID 0 PG GUID 0 Rank 2] Launching ProcessGroupNCCL abort asynchrounously. I1022 08:59:53.404664 597 ProcessGroupNCCL.cpp:1133] [PG ID 0 PG GUID 0 Rank 2] future is successfully executed for: ProcessGroup abort I1022 08:59:53.404670 597 ProcessGroupNCCL.cpp:1254] [PG ID 0 PG GUID 0 Rank 2] ProcessGroupNCCL aborts successfully. I1022 08:59:53.404700 597 ProcessGroupNCCL.cpp:1284] [PG ID 0 PG GUID 0 Rank 2] ProcessGroupNCCL watchdog thread joined. I1022 08:59:53.404719 597 ProcessGroupNCCL.cpp:1288] [PG ID 0 PG GUID 0 Rank 2] ProcessGroupNCCL heart beat monitor thread joined. I1022 08:59:53.409129 596 ProcessGroupNCCL.cpp:1263] [PG ID 0 PG GUID 0 Rank 1] ProcessGroupNCCL destructor entered. I1022 08:59:53.409163 596 ProcessGroupNCCL.cpp:1247] [PG ID 0 PG GUID 0 Rank 1] Launching ProcessGroupNCCL abort asynchrounously. I1022 08:59:53.409260 596 ProcessGroupNCCL.cpp:1133] [PG ID 0 PG GUID 0 Rank 1] future is successfully executed for: ProcessGroup abort I1022 08:59:53.409265 596 ProcessGroupNCCL.cpp:1254] [PG ID 0 PG GUID 0 Rank 1] ProcessGroupNCCL aborts successfully. I1022 08:59:53.409297 596 ProcessGroupNCCL.cpp:1284] [PG ID 0 PG GUID 0 Rank 1] ProcessGroupNCCL watchdog thread joined. I1022 08:59:53.409309 596 ProcessGroupNCCL.cpp:1288] [PG ID 0 PG GUID 0 Rank 1] ProcessGroupNCCL heart beat monitor thread joined. I1022 08:59:53.686990 595 ProcessGroupNCCL.cpp:1263] [PG ID 0 PG GUID 0 Rank 0] ProcessGroupNCCL destructor entered. W1022 08:59:53.687047 595 ProcessGroupNCCL.cpp:1267] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) I1022 08:59:53.687062 595 ProcessGroupNCCL.cpp:1247] [PG ID 0 PG GUID 0 Rank 0] Launching ProcessGroupNCCL abort asynchrounously. I1022 08:59:53.687188 595 ProcessGroupNCCL.cpp:1133] [PG ID 0 PG GUID 0 Rank 0] future is successfully executed for: ProcessGroup abort I1022 08:59:53.687193 595 ProcessGroupNCCL.cpp:1254] [PG ID 0 PG GUID 0 Rank 0] ProcessGroupNCCL aborts successfully. I1022 08:59:53.687217 595 ProcessGroupNCCL.cpp:1284] [PG ID 0 PG GUID 0 Rank 0] ProcessGroupNCCL watchdog thread joined. I1022 08:59:53.687238 595 ProcessGroupNCCL.cpp:1288] [PG ID 0 PG GUID 0 Rank 0] ProcessGroupNCCL heart beat monitor thread joined. ERROR 10-22 08:59:54 [core.py:589] EngineCore failed to start. ERROR 10-22 08:59:54 [core.py:589] Traceback (most recent call last): ERROR 10-22 08:59:54 [core.py:589] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 580, in run_engine_core ERROR 10-22 08:59:54 [core.py:589] engine_core = EngineCoreProc(*args, **kwargs) ERROR 10-22 08:59:54 [core.py:589] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 407, in __init__ ERROR 10-22 08:59:54 [core.py:589] super().__init__(vllm_config, executor_class, log_stats, ERROR 10-22 08:59:54 [core.py:589] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 77, in __init__ ERROR 10-22 08:59:54 [core.py:589] self.model_executor = executor_class(vllm_config) ERROR 10-22 08:59:54 [core.py:589] File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 53, in __init__ ERROR 10-22 08:59:54 [core.py:589] self._init_executor() ERROR 10-22 08:59:54 [core.py:589] File "/usr/local/lib/python3.10/dist-packages/vllm/v1/executor/multiproc_executor.py", line 93, in _init_executor ERROR 10-22 08:59:54 [core.py:589] self.workers = WorkerProc.wait_for_ready(unready_workers) ER

2025/10/22