2
性能优化
提问帖
新人贴
BW C3000 vllm 0.9.2 性能问题
2026-02-10 10:17:14
175
vllm 只用的das1.7官方打包好的 0.9.5版本3.10 dtk 24.0.2 scnet上的docker镜像 ```python INFO 02-10 10:07:05 [chat_utils.py:444] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this. INFO 02-10 10:07:05 [logger.py:43] Received request chatcmpl-22cb633b1c7c493994340e896d96d93a: prompt: '<|im_start|>user\n你好<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.9, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None. INFO 02-10 10:07:05 [async_llm.py:270] Added request chatcmpl-22cb633b1c7c493994340e896d96d93a. INFO 02-10 10:07:09 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% INFO: 127.0.0.1:38150 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 02-10 10:07:19 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 02-10 10:07:23 [logger.py:43] Received request chatcmpl-dc0114924de64378a948ca72e4d821d6: prompt: '<|im_start|>user\n你好<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.9, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None. INFO: 127.0.0.1:47462 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 02-10 10:07:23 [async_llm.py:270] Added request chatcmpl-dc0114924de64378a948ca72e4d821d6. INFO 02-10 10:07:29 [loggers.py:118] Engine 000: Avg prompt throughput: 0.9 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% INFO 02-10 10:07:39 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 02-10 10:07:49 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% ^CINFO 02-10 10:09:35 [launcher.py:80] Shutting down FastAPI HTTP server. ``` 运行的Qwen3-32b BF16的精度，TP=2 速度比之前的K100AI感觉还要慢，我不理解？ 我看正常抓了Cuda graph为什么这么慢？才5，6token一秒？ ``` root@crdnotebook-2021034352754171905:~# vllm serve /root/private_data/SothisAI/model/Aihub/Qwen3-32B/main/Qwen3-32B/ --served-model-name Qwen3-32B --tensor-parallel-size 2 --max-model-len 8192 --dtype bfloat16 --host 0.0.0.0 --port 8020 INFO 02-10 10:00:38 [__init__.py:245] Automatically detected platform rocm. INFO 02-10 10:00:42 [api_server.py:1395] vLLM API server version 0.9.2 INFO 02-10 10:00:42 [cli_args.py:325] non-default args: {'host': '0.0.0.0', 'port': 8020, 'model': '/root/private_data/SothisAI/model/Aihub/Qwen3-32B/main/Qwen3-32B/', 'dtype': 'bfloat16', 'max_model_len': 8192, 'served_model_name': ['Qwen3-32B'], 'tensor_parallel_size': 2} INFO 02-10 10:00:52 [config.py:850] This model supports multiple tasks: {'embed', 'generate', 'classify', 'reward'}. Defaulting to 'generate'. `torch_dtype` is deprecated! Use `dtype` instead! INFO 02-10 10:00:52 [config.py:1488] Using max model len 8192 INFO 02-10 10:00:55 [config.py:2301] Chunked prefill is enabled with max_num_batched_tokens=2048. INFO 02-10 10:01:00 [__init__.py:245] Automatically detected platform rocm. INFO 02-10 10:01:03 [core.py:529] Waiting for init message from front-end. INFO 02-10 10:01:03 [core.py:71] Initializing a V1 LLM engine (v0.9.2) with config: model='/root/private_data/SothisAI/model/Aihub/Qwen3-32B/main/Qwen3-32B/', speculative_config=None, tokenizer='/root/private_data/SothisAI/model/Aihub/Qwen3-32B/main/Qwen3-32B/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-32B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null} WARNING 02-10 10:01:03 [multiproc_worker_utils.py:307] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. INFO 02-10 10:01:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 16777216, 10, 'psm_dd022587'), local_subscribe_addr='ipc:///tmp/1d9c10a8-af22-48b0-97a3-95a0d58b571c', remote_subscribe_addr=None, remote_addr_ipv6=False) INFO 02-10 10:01:08 [__init__.py:245] Automatically detected platform rocm. INFO 02-10 10:01:08 [__init__.py:245] Automatically detected platform rocm. WARNING 02-10 10:01:12 [worker_base.py:42] VLLM_RANK1_NUMA is unset or set incorrectly, vllm will not bind to numa! VLLM_RANK1_NUMA = -1 INFO 02-10 10:01:12 [worker_base.py:654] ########## 19719 process(rank1) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127} INFO 02-10 10:01:12 [worker_base.py:655] ########## 19719 process(rank1) is running on memnode(s): {0, 1, 2, 3, 4, 5, 6, 7} (VllmWorker rank=1 pid=19719) INFO 02-10 10:01:12 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_76fc6038'), local_subscribe_addr='ipc:///tmp/f2b0a094-0162-4b4d-b4b8-8488cb5a0b11', remote_subscribe_addr=None, remote_addr_ipv6=False) WARNING 02-10 10:01:12 [worker_base.py:42] VLLM_RANK0_NUMA is unset or set incorrectly, vllm will not bind to numa! VLLM_RANK0_NUMA = -1 INFO 02-10 10:01:12 [worker_base.py:654] ########## 19718 process(rank0) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127} INFO 02-10 10:01:12 [worker_base.py:655] ########## 19718 process(rank0) is running on memnode(s): {0, 1, 2, 3, 4, 5, 6, 7} (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:12 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_79193597'), local_subscribe_addr='ipc:///tmp/76de98b1-2f5c-4e78-a62c-6b9d7be92218', remote_subscribe_addr=None, remote_addr_ipv6=False) (VllmWorker rank=1 pid=19719) INFO 02-10 10:01:13 [__init__.py:1171] Found nccl from library librccl.so.1 (VllmWorker rank=1 pid=19719) INFO 02-10 10:01:13 [pynccl.py:70] vLLM is using nccl==2.22.3 (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:13 [__init__.py:1171] Found nccl from library librccl.so.1 (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:13 [pynccl.py:70] vLLM is using nccl==2.22.3 (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:14 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_06430cfc'), local_subscribe_addr='ipc:///tmp/584ffea2-c0a3-4f2f-b08f-9f4f861d24b4', remote_subscribe_addr=None, remote_addr_ipv6=False) (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:14 [parallel_state.py:1077] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 (VllmWorker rank=1 pid=19719) INFO 02-10 10:01:14 [parallel_state.py:1077] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1 (VllmWorker rank=1 pid=19719) INFO 02-10 10:01:14 [gpu_model_runner.py:1819] Starting to load model /root/private_data/SothisAI/model/Aihub/Qwen3-32B/main/Qwen3-32B/... (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:14 [gpu_model_runner.py:1819] Starting to load model /root/private_data/SothisAI/model/Aihub/Qwen3-32B/main/Qwen3-32B/... (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:14 [gpu_model_runner.py:1824] Loading model from scratch... (VllmWorker rank=0 pid=19718) INFO 02-10 10:01:14 [rocm.py:288] Using Flash Attention backend on V1 engine. (only supports block size 64) (VllmWorker rank=1 pid=19719) INFO 02-10 10:01:14 [gpu_model_runner.py:1824] Loading model from scratch... (VllmWorker rank=1 pid=19719) INFO 02-10 10:01:14 [rocm.py:288] Using Flash Attention backend on V1 engine. (only supports block size 64) Loading safetensors checkpoint shards: 0% Completed | 0/17 [00:00user\n你好<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.9, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None. INFO 02-10 10:07:05 [async_llm.py:270] Added request chatcmpl-22cb633b1c7c493994340e896d96d93a. INFO 02-10 10:07:09 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% INFO: 127.0.0.1:38150 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 02-10 10:07:19 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 02-10 10:07:23 [logger.py:43] Received request chatcmpl-dc0114924de64378a948ca72e4d821d6: prompt: '<|im_start|>user\n你好<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=0.9, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1024, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, prompt_embeds shape: None, lora_request: None, prompt_adapter_request: None. INFO: 127.0.0.1:47462 - "POST /v1/chat/completions HTTP/1.1" 200 OK INFO 02-10 10:07:23 [async_llm.py:270] Added request chatcmpl-dc0114924de64378a948ca72e4d821d6. INFO 02-10 10:07:29 [loggers.py:118] Engine 000: Avg prompt throughput: 0.9 tokens/s, Avg generation throughput: 6.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0% INFO 02-10 10:07:39 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 5.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% INFO 02-10 10:07:49 [loggers.py:118] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0% ^CINFO 02-10 10:09:35 [launcher.py:80] Shutting down FastAPI HTTP server. ^C^CINFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. ```
全部评论（8）
只看楼主
D
dhtest
楼主
2楼
拉了镜像来部署，看起来性能好多了
2/10
M
maming
回复 @dhtest
好的，那就行

2/10
D
dhtest
回复 @maming
不过最新的镜像跑qwen next 80b还是有问题
2/10
M
maming
回复 @dhtest
qwen3-next需要手动设置环境变量
export VLLM_USE_NN=0
export TRITON_MOVE_LOAD_TOFRONT_DOT=0,可以根据社区文档查看docker pull harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.11.0-ubuntu22.04-dtk25.04.2-1226-das1.7-py3.10-20251226
2/11
D
dhtest
楼主
1楼
```

root@crdnotebook-2021034352754171905-dhtest2-zhang-12320:~# vllm serve /root/private_data/Qwen3-30A3-FP8/ --served-model-name Qwen3-32B --tensor-parallel-size 2 --max-model-len 8192 --dtype bfloat16 --host 0.0.0.0 --port 8020

^CTraceback (most recent call last):

  File "/usr/local/bin/vllm", line 3, in <module>

    from vllm.entrypoints.cli.main import main

  File "/usr/local/lib/python3.10/dist-packages/vllm/__init__.py", line 14, in <module>

    import vllm.env_override # noqa: F401

  File "/usr/local/lib/python3.10/dist-packages/vllm/env_override.py", line 5, in <module>

    import torch

  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 2493, in <module>

    from torch import _meta_registrations

  File "/usr/local/lib/python3.10/dist-packages/torch/_meta_registrations.py", line 10, in <module>

    from torch._decomp import (

  File "/usr/local/lib/python3.10/dist-packages/torch/_decomp/__init__.py", line 249, in <module>

    import torch._decomp.decompositions

  File "/usr/local/lib/python3.10/dist-packages/torch/_decomp/decompositions.py", line 284, in <module>

    @register_decomposition(aten.silu_backward)

  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1236, in __getattr__

    opoverloadpacket = OpOverloadPacket(

  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1017, in __init__

    self._has_torchbind_op_overload = any(

  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1018, in <genexpr>

    _has_script_object_arg(schema) for schema in self._schemas.values()

  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 1003, in _has_script_object_arg

    return any(isinstance(arg.type, torch.ClassType) for arg in schema.arguments)

KeyboardInterrupt

^C

root@crdnotebook-2021034352754171905-dhtest2-zhang-12320:~# vllm serve /root/private_data/Qwen3-30A3-FP8/ --served-model-name Qwen3-32B --tensor-parallel-size 2 --max-model-len 8192 --host 0.0.0.0 --port 8020

INFO 02-10 10:18:27 [__init__.py:245] Automatically detected platform rocm.

INFO 02-10 10:18:32 [api_server.py:1395] vLLM API server version 0.9.2

INFO 02-10 10:18:32 [cli_args.py:325] non-default args: {'host': '0.0.0.0', 'port': 8020, 'model': '/root/private_data/Qwen3-30A3-FP8/', 'max_model_len': 8192, 'served_model_name': ['Qwen3-32B'], 'tensor_parallel_size': 2}

INFO 02-10 10:18:43 [config.py:850] This model supports multiple tasks: {'classify', 'reward', 'embed', 'generate'}. Defaulting to 'generate'.

`torch_dtype` is deprecated! Use `dtype` instead!

INFO 02-10 10:18:43 [config.py:1488] Using max model len 8192

INFO 02-10 10:18:45 [config.py:2301] Chunked prefill is enabled with max_num_batched_tokens=2048.

INFO 02-10 10:18:51 [__init__.py:245] Automatically detected platform rocm.

INFO 02-10 10:18:54 [core.py:529] Waiting for init message from front-end.

INFO 02-10 10:18:54 [core.py:71] Initializing a V1 LLM engine (v0.9.2) with config: model='/root/private_data/Qwen3-30A3-FP8/', speculative_config=None, tokenizer='/root/private_data/Qwen3-30A3-FP8/', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-32B, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}

WARNING 02-10 10:18:54 [multiproc_worker_utils.py:307] Reducing Torch parallelism from 128 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.

INFO 02-10 10:18:54 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1], buffer_handle=(2, 16777216, 10, 'psm_4764e83f'), local_subscribe_addr='ipc:///tmp/3b160b93-2605-488a-8447-bb68c8e61bcc', remote_subscribe_addr=None, remote_addr_ipv6=False)

INFO 02-10 10:18:59 [__init__.py:245] Automatically detected platform rocm.

INFO 02-10 10:18:59 [__init__.py:245] Automatically detected platform rocm.

WARNING 02-10 10:19:03 [worker_base.py:42] VLLM_RANK1_NUMA is unset or set incorrectly, vllm will not bind to numa! VLLM_RANK1_NUMA = -1

INFO 02-10 10:19:03 [worker_base.py:654] ########## 24035 process(rank1) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}

INFO 02-10 10:19:03 [worker_base.py:655] ########## 24035 process(rank1) is running on memnode(s): {0, 1, 2, 3, 4, 5, 6, 7}

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_db1734c9'), local_subscribe_addr='ipc:///tmp/b5aaa839-5bc0-4793-b629-e9aece861d2e', remote_subscribe_addr=None, remote_addr_ipv6=False)

WARNING 02-10 10:19:03 [worker_base.py:42] VLLM_RANK0_NUMA is unset or set incorrectly, vllm will not bind to numa! VLLM_RANK0_NUMA = -1

INFO 02-10 10:19:03 [worker_base.py:654] ########## 24034 process(rank0) is running on CPU(s): {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127}

INFO 02-10 10:19:03 [worker_base.py:655] ########## 24034 process(rank0) is running on memnode(s): {0, 1, 2, 3, 4, 5, 6, 7}

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:03 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_56984c80'), local_subscribe_addr='ipc:///tmp/118f7f54-dc68-4cc5-81ad-0e5f14c962ff', remote_subscribe_addr=None, remote_addr_ipv6=False)

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:04 [__init__.py:1171] Found nccl from library librccl.so.1

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:04 [__init__.py:1171] Found nccl from library librccl.so.1

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:04 [pynccl.py:70] vLLM is using nccl==2.22.3

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:04 [pynccl.py:70] vLLM is using nccl==2.22.3

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:05 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1], buffer_handle=(1, 4194304, 6, 'psm_54097d34'), local_subscribe_addr='ipc:///tmp/0a27ea5d-d6fe-4809-a9c3-76e85f000d75', remote_subscribe_addr=None, remote_addr_ipv6=False)

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:05 [parallel_state.py:1077] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:05 [parallel_state.py:1077] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:05 [gpu_model_runner.py:1819] Starting to load model /root/private_data/Qwen3-30A3-FP8/...

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:05 [gpu_model_runner.py:1819] Starting to load model /root/private_data/Qwen3-30A3-FP8/...

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:05 [gpu_model_runner.py:1824] Loading model from scratch...

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:05 [gpu_model_runner.py:1824] Loading model from scratch...

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:05 [rocm.py:288] Using Flash Attention backend on V1 engine. (only supports block size 64)

(VllmWorker rank=0 pid=24034) WARNING 02-10 10:19:05 [fp8.py:488] CutlassBlockScaledGroupedGemm not supported on the current platform.

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:05 [rocm.py:288] Using Flash Attention backend on V1 engine. (only supports block size 64)

(VllmWorker rank=1 pid=24035) WARNING 02-10 10:19:05 [fp8.py:488] CutlassBlockScaledGroupedGemm not supported on the current platform.

Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s]

Loading safetensors checkpoint shards: 25% Completed | 1/4 [00:01<00:04, 1.36s/it]

Loading safetensors checkpoint shards: 50% Completed | 2/4 [00:16<00:19, 9.58s/it]

Loading safetensors checkpoint shards: 75% Completed | 3/4 [00:33<00:12, 12.87s/it]

Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:49<00:00, 14.14s/it]

Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:49<00:00, 12.39s/it]

(VllmWorker rank=0 pid=24034)

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:55 [default_loader.py:272] Loading weights took 49.78 seconds

(VllmWorker rank=0 pid=24034) INFO 02-10 10:19:56 [gpu_model_runner.py:1850] Model loading took 14.6560 GiB and 50.236842 seconds

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:57 [default_loader.py:272] Loading weights took 51.18 seconds

(VllmWorker rank=1 pid=24035) INFO 02-10 10:19:58 [gpu_model_runner.py:1850] Model loading took 14.6560 GiB and 51.630359 seconds

(VllmWorker rank=0 pid=24034) INFO 02-10 10:20:15 [backends.py:508] Using cache directory: /root/.cache/vllm/torch_compile_cache/13d7c404f9/rank_0_0/backbone for vLLM's torch.compile

(VllmWorker rank=0 pid=24034) INFO 02-10 10:20:15 [backends.py:519] Dynamo bytecode transform time: 16.69 s

(VllmWorker rank=1 pid=24035) INFO 02-10 10:20:15 [backends.py:508] Using cache directory: /root/.cache/vllm/torch_compile_cache/13d7c404f9/rank_1_0/backbone for vLLM's torch.compile

(VllmWorker rank=1 pid=24035) INFO 02-10 10:20:15 [backends.py:519] Dynamo bytecode transform time: 16.99 s

(VllmWorker rank=0 pid=24034) INFO 02-10 10:20:22 [backends.py:181] Cache the graph of shape None for later use

(VllmWorker rank=1 pid=24035) INFO 02-10 10:20:23 [backends.py:181] Cache the graph of shape None for later use

(VllmWorker rank=1 pid=24035) INFO 02-10 10:21:51 [backends.py:193] Compiling a graph for general shape takes 95.26 s

(VllmWorker rank=0 pid=24034) INFO 02-10 10:21:52 [backends.py:193] Compiling a graph for general shape takes 95.83 s

(VllmWorker rank=1 pid=24035) WARNING 02-10 10:21:56 [fp8_utils.py:561] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=2048,device_name=BW,dtype=fp8_w8a8,block_shape=[128,128].json

Unsupported input argument type.

UNREACHABLE executed at /home/triton/third_party/amd/lib/TritonAMDGPUTransforms/MfmaGroup.cpp:34!

(VllmWorker rank=0 pid=24034) WARNING 02-10 10:21:57 [fp8_utils.py:561] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=2560,K=2048,device_name=BW,dtype=fp8_w8a8,block_shape=[128,128].json

Unsupported input argument type.

UNREACHABLE executed at /home/triton/third_party/amd/lib/TritonAMDGPUTransforms/MfmaGroup.cpp:34!

ERROR 02-10 10:22:02 [multiproc_executor.py:135] Worker proc VllmWorker-1 died unexpectedly, shutting down executor.


``` 


FP8在BW卡上看起来支持又不支持

2/10
M
maming
回复 @dhtest
您好，docker pull harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.9.2-ubuntu22.04-dtk25.04.2-1226-das1.7-py3.10-20251226 是使用的这个镜像吗
2/10
D
dhtest
回复 @maming
你好，不是的，scnet只有 PyTorch / 2.5.1 / py3.10-ubuntu22.04 / dtk25.04.2， 我自己从https://download.sourcefind.cn:65024/4/main/vllm/DAS1.7 上装的vllm，scnet提供的jupyterlab-qwen3-openwebui:pytorch2.4.1-dtk25.04-py3.10-model速度也是类似很慢

scnet的环境似乎有一点问题
2/10
D
dhtest
回复 @maming
我感觉像nccl之类互联有问题
2/10
没有更多内容了
--
楼主
Level.3
0 帖子
楼主热帖
暂无数据
常见问题新帖
暂无数据