求助，发现个很奇怪的问题在Z100L上对qwen2.5 vllm推理 tensor-parallel-size设置为2时，可以正常执行。命令如下：python -m vllm.entrypoints.openai.api_server --port 9081 --load-format safetensors --model /data/model/qwen2.5-7b-instruct --served-model-name qwen2.5-7b-instruct --tensor-parallel-size 2 --enforce-eager --dtype float16 --trust-remote-code

我有4块卡，tensor-parallel-size 调整成4后，vllm启动失败。命令如下：python -m vllm.entrypoints.openai.api_server --port 9081 --load-format safetensors --model /data/model/qwen2.5-7b-instruct --served-model-name qwen2.5-7b-instruct --tensor-parallel-size 4 --enforce-eager --dtype float16 --trust-remote-code

报错日志如下：
INFO 12-04 09:51:24 api_server.py:177] vLLM API server version 0.5.0
INFO 12-04 09:51:24 api_server.py:178] args: Namespace(host=None, port=9081, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, chat_template=None, response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], model='/data/model/qwen2.5-7b-instruct', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, download_dir=None, load_format='safetensors', dtype='float16', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='outlines', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=False, num_lookahead_slots=0, seed=0, swap_space=4, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, enforce_eager=True, max_context_len_to_capture=None, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, enable_lora=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, device='auto', image_input_type=None, image_token_id=None, image_input_shape=None, image_feature_size=None, image_processor=None, image_processor_revision=None, disable_image_processor=False, scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, num_speculative_tokens=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, model_loader_extra_config=None, preemption_mode=None, served_model_name=['qwen2.5-7b-instruct'], qlora_adapter_name_or_path=None, engine_use_ray=False, disable_log_requests=False, max_log_len=None)
WARNING 12-04 09:51:24 config.py:1224] Casting torch.bfloat16 to torch.float16.
2024-12-04 09:51:26,343 INFO worker.py:1786 -- Started a local Ray instance.
INFO 12-04 09:51:28 config.py:629] Defaulting to use mp for distributed inference
INFO 12-04 09:51:28 config.py:645] Disabled the custom all-reduce kernel because it is not supported on AMD GPUs.
INFO 12-04 09:51:28 llm_engine.py:161] Initializing an LLM engine (v0.5.0) with config: model='/data/model/qwen2.5-7b-instruct', speculative_config=None, tokenizer='/data/model/qwen2.5-7b-instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.SAFETENSORS, tensor_parallel_size=4, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=qwen2.5-7b-instruct)
/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: clean_up_tokenization_spaces  was not set. It will be set to True  by default. This behavior will be deprecated in transformers v4.45, and will be then set to False  by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
INFO 12-04 09:51:29 selector.py:55] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=92273) INFO 12-04 09:51:39 selector.py:55] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=92273) INFO 12-04 09:51:39 multiproc_worker_utils.py:233] Worker ready; awaiting tasks
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1204 09:51:39.573400 92273 ProcessGroupNCCL.cpp:686] [Rank 2] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=93978956667968
(VllmWorkerProcess pid=92274) INFO 12-04 09:51:39 selector.py:55] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=92274) INFO 12-04 09:51:39 multiproc_worker_utils.py:233] Worker ready; awaiting tasks
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1204 09:51:39.608522 92274 ProcessGroupNCCL.cpp:686] [Rank 3] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94165337288336
(VllmWorkerProcess pid=92272) INFO 12-04 09:51:39 selector.py:55] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=92272) INFO 12-04 09:51:39 multiproc_worker_utils.py:233] Worker ready; awaiting tasks
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1204 09:51:39.742305 92272 ProcessGroupNCCL.cpp:686] [Rank 1] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94834019611648
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1204 09:51:39.749461 88639 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94567040819424
I1204 09:51:41.032374 88639 ProcessGroupNCCL.cpp:1340] NCCL_DEBUG: N/A
I1204 09:51:41.034157 92274 ProcessGroupNCCL.cpp:686] [Rank 3] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94165336873728
I1204 09:51:41.034173 92273 ProcessGroupNCCL.cpp:686] [Rank 2] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=93978955447136
I1204 09:51:41.034175 92272 ProcessGroupNCCL.cpp:686] [Rank 1] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94834019540528
I1204 09:51:41.034210 88639 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94567103074656
INFO 12-04 09:51:41 utils.py:623] Found nccl from library librccl.so.1
(VllmWorkerProcess pid=92274) INFO 12-04 09:51:41 utils.py:623] Found nccl from library librccl.so.1
(VllmWorkerProcess pid=92272) INFO 12-04 09:51:41 utils.py:623] Found nccl from library librccl.so.1
INFO 12-04 09:51:41 pynccl.py:65] vLLM is using nccl==2.18.3
(VllmWorkerProcess pid=92273) INFO 12-04 09:51:41 utils.py:623] Found nccl from library librccl.so.1
(VllmWorkerProcess pid=92274) INFO 12-04 09:51:41 pynccl.py:65] vLLM is using nccl==2.18.3
(VllmWorkerProcess pid=92272) INFO 12-04 09:51:41 pynccl.py:65] vLLM is using nccl==2.18.3
(VllmWorkerProcess pid=92273) INFO 12-04 09:51:41 pynccl.py:65] vLLM is using nccl==2.18.3
I1204 09:51:41.418828 88639 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94567068845936
I1204 09:51:41.419068 92272 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94834018432128
I1204 09:51:41.419312 92273 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=93978976513328
I1204 09:51:41.419360 92274 ProcessGroupNCCL.cpp:686] [Rank 0] ProcessGroupNCCL initialization options:NCCL_ASYNC_ERROR_HANDLING: 1, NCCL_DESYNC_DEBUG: 0, NCCL_ENABLE_TIMING: 0, NCCL_BLOCKING_WAIT: 0, TIMEOUT(ms): 1800000, USE_HIGH_PRIORITY_STREAM: 0, TORCH_DISTRIBUTED_DEBUG: OFF, NCCL_DEBUG: OFF, ID=94165358525040
(VllmWorkerProcess pid=92273) WARNING 12-04 09:51:41 __init__.py:104] Model architecture Qwen2ForCausalLM is partially supported by ROCm: Sliding window attention is not yet supported in ROCm's flash attention
(VllmWorkerProcess pid=92274) WARNING 12-04 09:51:41 __init__.py:104] Model architecture Qwen2ForCausalLM is partially supported by ROCm: Sliding window attention is not yet supported in ROCm's flash attention
(VllmWorkerProcess pid=92272) WARNING 12-04 09:51:41 __init__.py:104] Model architecture Qwen2ForCausalLM is partially supported by ROCm: Sliding window attention is not yet supported in ROCm's flash attention
WARNING 12-04 09:51:41 __init__.py:104] Model architecture Qwen2ForCausalLM is partially supported by ROCm: Sliding window attention is not yet supported in ROCm's flash attention
(VllmWorkerProcess pid=92274) INFO 12-04 09:51:41 selector.py:55] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=92273) INFO 12-04 09:51:41 selector.py:55] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=92272) INFO 12-04 09:51:41 selector.py:55] Using ROCmFlashAttention backend.
INFO 12-04 09:51:41 selector.py:55] Using ROCmFlashAttention backend.
(VllmWorkerProcess pid=92274) INFO 12-04 09:51:47 model_runner.py:159] Loading model weights took 3.5546 GB
(VllmWorkerProcess pid=92273) INFO 12-04 09:51:47 model_runner.py:159] Loading model weights took 3.5546 GB
(VllmWorkerProcess pid=92272) INFO 12-04 09:51:47 model_runner.py:159] Loading model weights took 3.5546 GB
INFO 12-04 09:51:47 model_runner.py:159] Loading model weights took 3.5546 GB
I1204 09:51:48.327005 88639 ProcessGroupNCCL.cpp:1340] NCCL_DEBUG: N/A
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks:   <built-in function load_binary>  returned NULL without setting an exception, Traceback (most recent call last):
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 241, in _run_worker_process
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] output = executor(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return func(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/worker.py", line 154, in determine_num_available_blocks
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self.model_runner.profile_run()
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return func(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 861, in profile_run
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self.execute_model(seqs, kv_caches)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return func(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 738, in execute_model
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states = model_executable(
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 351, in forward
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 266, in forward
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states, residual = layer(
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 218, in forward
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states = self.self_attn(
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 165, in forward
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/layer.py", line 89, in forward
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self.impl.forward(query, key, value, kv_cache, attn_metadata,
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/backends/rocm_flash_attn.py", line 344, in forward
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] out = self.attn_func_triton(
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/ops/flash_attn_triton_mqa_gqa.py", line 1289, in flash_attn_varlen_func
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return _attention.apply(q, k, v, None, input_metadata)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return super().apply(*args, **kwargs) # type: ignore[misc]
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/ops/flash_attn_triton_mqa_gqa.py", line 1057, in forward
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] attn_fwd[grid](
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 161, in run
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs}
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 161, in <dictcomp> 
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs}
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 136, in _bench
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8))
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/testing.py", line 110, in do_bench
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] fn()
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 124, in kernel_call
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self.fn.run(
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/jit.py", line 581, in run
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] bin.c_wrapper(
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/compiler/compiler.py", line 750, in __getattribute__
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self._init_handles()
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/compiler/compiler.py", line 741, in _init_handles
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device)
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] SystemError:   <built-in function load_binary>  returned NULL without setting an exception
(VllmWorkerProcess pid=92273) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244]
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks:   <built-in function load_binary>  returned NULL without setting an exception, Traceback (most recent call last):
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 241, in _run_worker_process
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] output = executor(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return func(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/worker.py", line 154, in determine_num_available_blocks
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self.model_runner.profile_run()
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return func(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 861, in profile_run
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self.execute_model(seqs, kv_caches)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return func(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 738, in execute_model
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states = model_executable(
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 351, in forward
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states = self.model(input_ids, positions, kv_caches,
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 266, in forward
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states, residual = layer(
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 218, in forward
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] hidden_states = self.self_attn(
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 165, in forward
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/layer.py", line 89, in forward
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return self.impl.forward(query, key, value, kv_cache, attn_metadata,
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/backends/rocm_flash_attn.py", line 344, in forward
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] out = self.attn_func_triton(
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/ops/flash_attn_triton_mqa_gqa.py", line 1289, in flash_attn_varlen_func
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return _attention.apply(q, k, v, None, input_metadata)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return super().apply(*args, **kwargs) # type: ignore[misc]
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/attention/ops/flash_attn_triton_mqa_gqa.py", line 1057, in forward
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] attn_fwd[grid](
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 161, in run
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs}
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 161, in <dictcomp> 
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] timings = {config: self._bench(*args, config=config, **kwargs) for config in pruned_configs}
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 136, in _bench
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return do_bench(kernel_call, warmup=self.warmup, rep=self.rep, quantiles=(0.5, 0.2, 0.8))
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/testing.py", line 110, in do_bench
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] fn()
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/autotuner.py", line 124, in kernel_call
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self.fn.run(
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/runtime/jit.py", line 581, in run
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] bin.c_wrapper(
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/compiler/compiler.py", line 750, in __getattribute__
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self._init_handles()
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/triton/compiler/compiler.py", line 741, in _init_handles
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] mod, func, n_regs, n_spills = fn_load_binary(self.metadata["name"], self.asm[bin_path], self.shared, device)
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] SystemError:   <built-in function load_binary>  returned NULL without setting an exception
(VllmWorkerProcess pid=92272) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244]
INFO 12-04 09:51:57 multiproc_worker_utils.py:154] Terminating local vLLM worker processes
(VllmWorkerProcess pid=92272) INFO 12-04 09:51:57 multiproc_worker_utils.py:255] Worker exiting
(VllmWorkerProcess pid=92273) INFO 12-04 09:51:57 multiproc_worker_utils.py:255] Worker exiting
Traceback (most recent call last):
File "/data/anaconda3/envs/qwen25/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/data/anaconda3/envs/qwen25/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 196, in <module> 
engine = AsyncLLMEngine.from_engine_args(
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 395, in from_engine_args
engine = cls(
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 349, in __init__
self.engine = self._init_engine(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 470, in _init_engine
return engine_class(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 238, in __init__
self._initialize_kv_caches()
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 328, in _initialize_kv_caches
self.model_executor.determine_num_available_blocks())
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 38, in determine_num_available_blocks
num_blocks = self._run_workers("determine_num_available_blocks", )
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 119, in _run_workers
driver_worker_output = driver_worker_method(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/worker.py", line 154, in determine_num_available_blocks
self.model_runner.profile_run()
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 861, in profile_run
self.execute_model(seqs, kv_caches)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 738, in execute_model
hidden_states = model_executable(
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 351, in forward
hidden_states = self.model(input_ids, positions, kv_caches,
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 266, in forward
hidden_states, residual = layer(
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 218, in forward
hidden_states = self.self_attn(
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 165, in forward
attn_output = self.attn(q, k, v, kv_cache, attn_metadata)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] Exception in worker VllmWorkerProcess while processing method determine_num_available_blocks:   <built-in function load_binary>  returned NULL without setting an exception, Traceback (most recent call last):
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 241, in _run_worker_process
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] output = executor(*args, **kwargs)
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] return func(*args, **kwargs)
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/vllm/worker/worker.py", line 154, in determine_num_available_blocks
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] self.model_runner.profile_run()
(VllmWorkerProcess pid=92274) ERROR 12-04 09:51:57 multiproc_worker_utils.py:244] File "/data/anaconda3/envs/qwen25/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context