请问哪里下载
0
16
0
0
K100_AI 编译的 llama.cpp 支持 Qwen3.6 MoE 模型
2026-05-02 09:07:38
1028
# 【分享】K100_AI 编译的 llama.cpp 已支持 Qwen3.6 MoE 模型
使用 **DTK 26.04** 镜像编译 `llama.cpp`，实测成功支持 **Qwen3.6 MoE** 的 GGUF 模型推理，有需要小伙伴下载试试。下面是在海光 DTK 26.04 镜像内的完整使用步骤。
## 1. 下载编译好的 llama.cpp
```bash
wget http://60.190.128.9/llama.cpp.tar.gz
```
## 2. 复制至海光 DTK 26.04 镜像
```bash
docker cp ./llama.cpp.tar.gz 你的DTK-26.04镜像:/workspace
将 你的DTK-26.04镜像 替换为实际容器名或 ID。
```
## 3. 进入镜像，解压并运行
```bash
docker exec -it 你的DTK-26.04镜像 bash
cd /workspace
tar -zxvf llama.cpp.tar.gz
cd llama.cpp
```
## 4. 运行 Qwen3.6 MoE GGUF 模型
```bash
./build/bin/llama-cli \
-m /data/.../Qwen3.6-35B-A3B.gguf \
-ngl 99 -t 32 \
-p "你好，用一句中文介绍你自己。"
```
请将 /data/.../model.gguf 替换为你实际的 GGUF 模型路径。
全部评论（9）
只看楼主
老
7/13
_
您好，请问容器用的哪个？llama.cpp用的哪个版本的？
6/3
平
您好下载出错了
5/29
F
您好，请问一下单卡K100AI能跑吗？输出tokens大概每秒多少？
5/25
T
root@crdnotebook-2050406032157765634-wmx-scnet-92118:/workspace/llama.cpp# ./build/bin/llama-cli \
-m /root/private_data/model/Qwen/Qwen3.5-2B-GGUF/Qwen3.5-2B-Q4_0.gguf \
-ngl 99 -t 32 \
-p "你好，用一句中文介绍你自己"
ggml_cuda_init: found 1 ROCm devices (Total VRAM: 65520 MiB):
Device 0: BW, gfx936:sramecc+:xnack- (0x936), VMM: no, Wave Size: 64, VRAM: 65520 MiB
Loading model... /ggml_cuda_compute_forward: SCALE failed
ROCm error: invalid device function
current device: 0, in function ggml_cuda_compute_forward at /workspace/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2962
err
/workspace/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: ROCm error
-[New LWP 240272]
[New LWP 240276]
[New LWP 240277]
[New LWP 240281]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f4aeb4ea42f in wait4 () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007f4aeb4ea42f in wait4 () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f4aebd5b71b in ggml_print_backtrace () from /workspace/llama.cpp/build/bin/libggml-base.so.0
#2 0x00007f4aebd5b8b2 in ggml_abort () from /workspace/llama.cpp/build/bin/libggml-base.so.0
#3 0x00007f4ae431a3c2 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /workspace/llama.cpp/build/bin/libggml-hip.so.0
#4 0x00007f4ae432401e in ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context*, ggml_cgraph*, bool, bool, void const*) () from /workspace/llama.cpp/build/bin/libggml-hip.so.0
#5 0x00007f4ae431feb1 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /workspace/llama.cpp/build/bin/libggml-hip.so.0
#6 0x00007f4aebd7973f in ggml_backend_sched_graph_compute_async () from /workspace/llama.cpp/build/bin/libggml-base.so.0
#7 0x00007f4aebaae141 in llama_context::graph_compute(ggml_cgraph*, bool) () from /workspace/llama.cpp/build/bin/libllama.so.0
#8 0x00007f4aebaae575 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /workspace/llama.cpp/build/bin/libllama.so.0
#9 0x00007f4aebab6b8b in llama_context::decode(llama_batch const&) () from /workspace/llama.cpp/build/bin/libllama.so.0
#10 0x00007f4aebab81d0 in llama_decode () from /workspace/llama.cpp/build/bin/libllama.so.0
#11 0x00007f4aebfffaef in common_init_from_params(common_params&) () from /workspace/llama.cpp/build/bin/libllama-common.so.0
#12 0x0000564b5b2d6d2e in server_context_impl::load_model(common_params&) ()
#13 0x0000564b5b2128b3 in main ()
[Inferior 1 (process 240271) detached]
\Aborted (core dumped)
5/6
T
你好，可以在 scnet上发布镜像吗? 我试了，这个卡用不了：资源配置加速卡
异构加速卡BW * 1卡
显存64GB
CPU型号2*64C
CPU15核心
内存59GB
环境信息 镜像
lqm-vllm:0.18-ubuntu22.04-dtk26.04-qwen3.5-0425
5/6
没有更多内容了
--
楼主
0 点赞
0 帖子
楼主热帖
暂无数据常见问题新帖
暂无数据