1. 环境部署 

创建容器

docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.4.1-ubuntu22.04-dtk25.04-py3.10
docker run --shm-size 500g --network=host --name=ktransformers --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v ~/:/workspace/ -v /public/opendas/DL_DATA/llm-models/deepseek-r1:/models -v /opt/hyhal:/opt/hyhal:ro -it image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.4.1-ubuntu22.04-dtk25.04-py3.10 bash

安装依赖
依赖包下载链接: https://pan.baidu.com/s/19NxXWi_9oIzC35hspH1zKw 提取码: tfe8

pip install ktransformers-0.2.3+das.dtk2504-cp310-cp310-linux_x86_64.whl
pip install fastpt-2.0.1+das.dtk2504-py3-none-any.whl

2. 模型下载 

pip install modelscope

DeepSeek-R1模型

modelscope download --model deepseek-ai/DeepSeek-R1

DeepSeek-R1-GGUF模型

modelscope download --model unsloth/DeepSeek-R1-GGUF

3. 部署&推理 

环境变量添加

source /opt/dtk/cuda/env.sh
export LD_LIBRARY_PATH=/usr/local/lib/python3.10/site-packages/torch/lib:$LD_LIBRARY_PATH
export TRITON_HIP_USE_NEW_STREAM_PIPELINE=0

启动推理

cd /usr/local/lib/python3.10/site-packages
python ktransformers/local_chat.py   --model_path /models/DeepSeek-R1   --gguf_path /models/DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M/   --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-dcu.yaml   --cpu_infer 15

4. 推理结果