本机环境:
CPU:Intel(R) Xeon(R) Silver 4410Y
DCU:K100_AI
镜像:image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.4.1-ubuntu22.04-dtk25.04-py3.10
Ktransformers:v0.3.1
启动命令:python3 ktransformers/server/main.py --model_path /mnt/model/Qwen3-235B-A22B-Instruct-2507-GGUF/BF16/ --gguf_path /mnt/model/Qwen3-235B-A22B-Instruct-2507-GGUF/BF16/ --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --cpu_infer 36
报错信息:
no balance_serve
flashinfer not found, use triton for linux
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1034, in from_pretrained
config_class = CONFIG_MAPPING[config_dict["model_type"]]
File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 736, in __getitem__
raise KeyError(key)
KeyError: 'qwen3_moe'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/ktransformers/ktransformers/server/main.py", line 122, in model
main()
File "/root/ktransformers/ktransformers/server/main.py", line 108, in main
args = arg_parser.parse_args()
File "/usr/local/lib/python3.10/site-packages/ktransformers/server/args.py", line 139, in parse_args
model_config = AutoConfig.from_pretrained(args.model_dir, trust_remote_code=True)
File "/usr/local/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1036, in from_pretrained
raise ValueError(
ValueError: The checkpoint you are trying to load has model type Qwen3-Moe but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
另外部署 DeepSeek-R1-BF16 启动需要大半天才行,有什么能够优化。