DTK24.04 Z100L
镜像：image.sourcefind.cn:5000/dcu/admin/base/custom-vllm0.5.0-dtk24.04.1-ubuntu20.04-py310-zk-v1

运行python -m vllm.entrypoints.openai.api_server --model /home/DeepSeek-R1-Distill-Qwen-32B --dtype float16 --max-model-len 32768 -tp 4 报错。
之前都能用，突然用不了了，想请问各位大神有遇到过没？

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/torch/_ops.py", line 757, in getattr
op, overload_names = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator _C::trans_w16_gemm

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 196, in <module> 
engine = AsyncLLMEngine.from_engine_args(
File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 395, in from_engine_args
engine = cls(
File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 349, in init
self.engine = self._init_engine(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 470, in _init_engine
return engine_class(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 223, in init
self.model_executor = executor_class(
File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 142, in init
super().init (*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in init
super().init (*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in init
self._init_executor()
File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 66, in _init_executor
self._run_workers("load_model",
File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 119, in _run_workers
driver_worker_output = driver_worker_method(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 121, in load_model
self.model_runner.load_model()
File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 147, in load_model
self.model = get_model(
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/model_loader/init .py", line 21, in get_model
return loader.load_model(model_config=model_config,
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 252, in load_model
model.load_weights(
File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 435, in load_weights
ops.trans_w16_gemm(_weight, weight.data, _weight.shape[0], _weight.shape[1])
File "/usr/local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 227, in trans_w16_gemm
torch.ops._C.trans_w16_gemm(dst,src,row,col)
File "/usr/local/lib/python3.10/site-packages/torch/_ops.py", line 761, in getattr
raise AttributeError(
AttributeError: '_OpNamespace' '_C' object has no attribute 'trans_w16_gemm'
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] Exception in worker VllmWorkerProcess while processing method load_model: '_OpNamespace' '_C' object has no attribute 'trans_w16_gemm', Traceback (most recent call last):
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/torch/_ops.py", line 757, in __getattr__
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] op, overload_names = torch._C._jit_get_operation(qualified_op_name)
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] RuntimeError: No such operator _C::trans_w16_gemm
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244]
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] The above exception was the direct cause of the following exception:
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244]
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] Traceback (most recent call last):
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 241, in _run_worker_process
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] output = executor(*args, **kwargs)
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/vllm/worker/worker.py", line 121, in load_model
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] self.model_runner.load_model()
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 147, in load_model
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] self.model = get_model(
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py", line 21, in get_model
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] return loader.load_model(model_config=model_config,
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/model_loader/loader.py", line 252, in load_model
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] model.load_weights(
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 435, in load_weights
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] ops.trans_w16_gemm(_weight, weight.data, _weight.shape[0], _weight.shape[1])
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/vllm/_custom_ops.py", line 227, in trans_w16_gemm
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] torch.ops._C.trans_w16_gemm(dst,src,row,col)
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] File "/usr/local/lib/python3.10/site-packages/torch/_ops.py", line 761, in __getattr__
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] raise AttributeError(
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244] AttributeError: '_OpNamespace' '_C' object has no attribute 'trans_w16_gemm'
(VllmWorkerProcess pid=12706) ERROR 05-07 16:53:47 multiproc_worker_utils.py:244]