2
Qwen
新人贴
请问现在支持GPTQMarlin吗?
2025-11-18 21:14:20
157
我在K100-AI上部署的gptq量化模型没法正常运行,用vllm:0.9.2-ubuntu22.04-dtk25.04.1-rc5-rocblas104381-0915-das1.6-py3.10-20250916-rc2镜像模型会一直输出感叹号,用vllm:0.8.5-rockylinux8.6-dtk25.04.1-rc3-das1.6-py3.10-20250724-shca镜像则加载过程中报错。报错内容如下: torch._dynamo.exc.Unsupported: Graph break due to unsupported builtin lmslimquant.PyCapsule.gptq_gemm. This function is either a Python builtin (e.g. _warnings.warn) or a third-party C/C++ Python extension (perhaps created with pybind). If it is a Python builtin, please file an issue on GitHub so the PyTorch team can add support for it and see the next case for a workaround. If it is a third-party C/C++ Python extension, please either wrap it into a PyTorch-understood custom operator (see https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html for more details) or, if it is traceable, use torch.compiler.allow_in_graph.
全部评论(3)
只看楼主
L
litzh
1楼
你好,gptq_marlin类算子当前还不支持,可以尝试更换其他类型量化模型
2025/11/28
_
_mi4l9xhe6py3
回复 @litzh
有支持计划吗?
2025/12/1
L
litzh
回复 @_mi4l9xhe6py3
k100ai暂时没有支持计划,可以先用非marlin的量化模型
2025/12/1
没有更多内容了
--
楼主
Level.1
0 帖子
楼主热帖
暂无数据