运行环境:
4张K100_AI
容器image.sourcefind.cn:5000/dcu/admin/base/custom:vllm0.9.2-ubuntu22.04-dtk25.04-das1.6-py3.10-20250916-rc2中启动vllm
使用Qwen3-235B-A22B-Instruct-2507-AWQ量化模型(https://www.modelscope.cn/models/swift/Qwen3-235B-A22B-Instruct-2507-AWQ)
每一层都抛出警告“(VllmWorker rank=0 pid=697) WARNING 10-17 09:48:11 [awq_marlin.py:150] Layer 'model.layers.0.mlp.experts' is not supported by AWQMoeMarlin. Falling back to Moe WNA16 kernels.”,虽然可以正常运行,但是推理速度很慢。