光合开发者社区-论坛

Qwen

【经验】使用K100运行Qwen3 MoE模型

2025-06-09 15:04:37

661

最近一直在研究LLM for Science相关的内容，其中本地化部署是相当重要的一步。看到Qwen3发布以及Qwen3 MoE系列模型（235B和30B）的优秀表现后，就想着在组里的DCU上部署一个用于各类测试。

之前尝试过在DCU上部署例如DeepSeek、QwQ、Qwen2.5等模型，也积累了一些经验，对于K100，大致分两种策略：

vllm 0.5.0 能跑起来的模型：使用 vllm 0.5.0 +定制版flash_attn 运行，这是性能最好也是最方便的；
需要更高版本vllm 才能跑的模型：使用高版本vllm +triton 运行，性能会比定制版flash_attn 差一些，但胜在广泛，只要移植了模型的.py 文件并且在vllm 里注册了模型，基本就能跑。

对于Qwen3，第二种策略是比较方便的。目前 vllm 官方GitHub仓库，基本只要大于 0.8.4 就能够支持Qwen3了，因此剩下的事情只有两步：

在光源的docker仓库里挑一个合适版本的vllm 镜像
移植vllm 官方GitHub仓库中对Qwen3的实现

就能做到在K100上跑起Qwen3来。

第一步，经过测试，我选择的docker镜像是这个：
custom:vllm0.7.2-ubuntu22.04-dtk25.04-rc6-das1.4-py3.10-20250227

第二步，开始移植新版 vllm 中运行Qwen3所需的文件。这里其实还有一步，就是需要对镜像中 triton 做一个修改：

sed -i "s/pm, options.arch, options.matrix_instr_nonkdim/pm, \"gfx906\", options.matrix_instr_nonkdim/g" \
    /usr/local/lib/python3.10/site-packages/triton/backends/amd/compiler.py

也就是让 triton 进行编译的时候，按照 gfx906 进行编译，不要按照K100默认的 gfx926 ，否则会出现问题。移植的文件是 vllm/model_executor/models/qwen3_moe.py 和 vllm/model_executor/models/registry.py 这两个。顺带提一下，如果要运行DeepSeek Coder的话还需要对 vllm/attention/backends/triton_mla.py 和 vllm/attention/backends/mla/utils.py 这两个文件进行修改。总之，这四个文件都放在附件中了。上传的时候发现不能上传xz格式的压缩包，我就直接把后缀改成gz上传了，因此使用的时候，需要先把后缀名改成 .tar.xz ，然后用 tar -Jxvf 解压。

我在服务器上成功运行的命令如下：

启动docker：

  docker run -dit \
      --network=host \
      --name=${img_name} \
      --privileged \
      --device=/dev/kfd \
      --device=/dev/dri \
      --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm \
      --ipc=host \
      --shm-size=400G \
      --group-add video \
      --cap-add=SYS_PTRACE \
      --security-opt seccomp=unconfined \
      -u root \
      --ulimit stack=-1:-1 --ulimit memlock=-1:-1 \
      -v /opt/hyhal:/opt/hyhal:ro -v /public/home/deepseek/softwares/packages/vllm_models:/deepseek_workspace \
      -v /public/home/deepseek/tests/ollama/:/deepseek_workspace_scripts \
      -v /public/home/deepseek:/deepseek_home \
      image.sourcefind.cn:5000/dcu/admin/base/custom:vllm0.7.2-ubuntu22.04-dtk25.04-rc6-das1.4-py3.10-20250227

启动vllm：

  pip install -U transformers --find-links=/deepseek_home/softwares/for_root_docker/python_packages --no-index

  sed -i "s/pm, options.arch, options.matrix_instr_nonkdim/pm, \"gfx906\", options.matrix_instr_nonkdim/g" \
      /usr/local/lib/python3.10/site-packages/triton/backends/amd/compiler.py

  cp /deepseek_home/softwares/packages/docker_images/vllm072_ubuntu2204_dtk2404rc6_das14_py310_rocblas_triton_patches/usr/local/lib/python3.10/site-packages/vllm/attention/backends/triton_mla.py \
      /usr/local/lib/python3.10/site-packages/vllm/attention/backends/triton_mla.py

  cp /deepseek_home/softwares/packages/docker_images/vllm072_ubuntu2204_dtk2404rc6_das14_py310_rocblas_triton_patches/usr/local/lib/python3.10/site-packages/vllm/attention/backends/mla/utils.py \
      /usr/local/lib/python3.10/site-packages/vllm/attention/backends/mla/utils.py

  cp /deepseek_home/softwares/packages/docker_images/vllm072_ubuntu2204_dtk2404rc6_das14_py310_rocblas_triton_patches/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py \
      /usr/local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_moe.py

  cp /deepseek_home/softwares/packages/docker_images/vllm072_ubuntu2204_dtk2404rc6_das14_py310_rocblas_triton_patches/usr/local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py \
      /usr/local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py

  ls /usr/local/lib/python3.10/site-packages/vllm/model_executor/models/

  export HIP_VISIBLE_DEVICES=0,1,2,3
  export ALLREDUCE_STREAM_WITH_COMPUTE=1

  export VLLM_HOST_IP=$(hostname --ip)
  export VLLM_NUMA_BIND=1
  export HSA_OVERRIDE_GFX_VERSION=9.2.6
  export PYTORCH_ROCM_ARCH=gfx926
  export ARG_PYTORCH_ROCM_ARCH=gfx926
  export VLLM_USE_TRITON_FLASH_ATTN=True  # False  # True
  export VLLM_LOGGING_LEVEL=DEBUG

  ray start --head --node-ip-address=${VLLM_HOST_IP} --port=8011 --num-gpus=4 --num-cpus=16
  export RAY_ADDRESS="${VLLM_HOST_IP}:8011"

  sleep 10

  export VLLM_API_KEY="12lmj356Xq1x4QKj"

  python -m vllm.entrypoints.openai.api_server \
      --model /deepseek_workspace/Qwen3-30B-A3B \
      --served-model-name Qwen3-30B-A3B \
      --trust-remote-code \
      --max-model-len $[128*1024] \
      --tensor-parallel-size 4 \
      --pipeline-parallel-size 1 \
      --host ${VLLM_HOST_IP} \
      --port 8010 \
      --api-key ${VLLM_API_KEY} \
      --gpu-memory-utilization 0.80 \
      --distributed-executor-backend ray \
      --dtype float16 \
      --hf-overrides '{"rope_scaling": {"factor": 4.0, "original_max_position_embeddings": 32768, "rope_type": "yarn"}}' \
      &>/deepseek_workspace_scripts/deploy_models/Qwen3_30B_A3B/vllm_qwen3_30b_a3b_$(date -Is).log

启动时候需要注意一下，镜像内自带的transformers版本应该是过低的，因此需要 pip install -U 对transformers进行升级。我们组内的机器没有网络，所以我是把需要的包先下载到本地再安装的，如果有网络的话那就不用设置 --find-links 之类的参数了。下面那四个 cp 命令是把附件里的四个py文件复制到 vllm 的目录里，也记得把路径改成自己机器上的路径。另外 export VLLM_USE_TRITON_FLASH_ATTN=True 这个是强制 vllm 使用 triton 的flash attention，不要用 flash_attn ， vllm 0.7.2 这个镜像的 flash_attn 在K100上跑会有报错。

最后是运行效果，在4卡K100上性能如下：

INFO:     10.10.10.1:59628 - "GET /v1/models HTTP/1.0" 200 OK
INFO 06-09 14:52:21 logger.py:39] Received request chatcmpl-5a5a570a7fac4f6d871f6c09ca7bfaca: prompt: '<|im_start|>system\nUser Context:\n\n<|im_end|>\n<|im_start|>user\n请用python写一个快速排序的代码<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=131047, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.
INFO:     10.10.10.1:59630 - "POST /v1/chat/completions HTTP/1.0" 200 OK
INFO 06-09 14:52:21 engine.py:275] Added request chatcmpl-5a5a570a7fac4f6d871f6c09ca7bfaca.
INFO 06-09 14:52:25 metrics.py:455] Avg prompt throughput: 5.0 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
DEBUG 06-09 14:52:30 client.py:170] Heartbeat successful.
INFO 06-09 14:52:30 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 06-09 14:52:35 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
DEBUG 06-09 14:52:40 client.py:170] Heartbeat successful.
INFO 06-09 14:52:40 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 06-09 14:52:45 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
DEBUG 06-09 14:52:50 client.py:170] Heartbeat successful.
INFO 06-09 14:52:50 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 06-09 14:52:55 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 24.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.

个人认为已经完全可流畅使用了。

这样修改之后运行Qwen3-235B-A22B也是可以的，不过速度比较慢，之前测试了一下 pipeline_parallel_size=3, tensor_parallel_size=4 使用8+4块K100跨节点运行的结果如下：

INFO 05-06 13:13:58 metrics.py:455] Avg prompt throughput: 135.6 tokens/s, Avg generation throughput: 1.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0
.2%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:04 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2
%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:09 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2
%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:14 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2
%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:19 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:24 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:29 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3
%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:34 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3
%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:39 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3
%, CPU KV cache usage: 0.0%.
INFO 05-06 13:14:44 metrics.py:455] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3
%, CPU KV cache usage: 0.0%.

相同的方法，理论上也能让K100跑起Qwen3的Dense model，不过我就没有再尝试了。