运行环境

硬件

• K100 单卡（非K100-AI）

软件

使用的是(此页面)[https://developer.sourcefind.cn/news/domestic/10852/]的docker容器，即ubuntu22.04、vllm 0.8.4、DTK 25.04

复现方法

1. 启动docker：

   docker run -it \
       --network=host \
       --name=testtest \
       --privileged \
       --device=/dev/kfd \
       --device=/dev/dri \
       --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm \
       --ipc=host \
       --shm-size=400G \
       --group-add video \
       --cap-add=SYS_PTRACE \
       --security-opt seccomp=unconfined \
       -u root \
       --ulimit stack=-1:-1 --ulimit memlock=-1:-1 \
       -v /opt/hyhal:/opt/hyhal:ro -v /public/home/deepseek/softwares/packages/vllm_models:/deepseek_workspace \
       -v /public/home/deepseek/tests/ollama/:/deepseek_workspace_scripts \
       -v /public/home/deepseek:/deepseek_home \
       image.sourcefind.cn:5000/dcu/admin/base/custom:vllm0.8.4-ubuntu22.04-dtk25.04-rc7-das1.5-py3.10-20250429-dev-qwen3-only
   

2. 配置环境变量

   source /opt/dtk/env.sh
   export ALLREDUCE_STREAM_WITH_COMPUTE=1
   export VLLM_LOGGING_LEVEL=DEBUG
   export VLLM_USE_TRITON_FLASH_ATTN=False
   export PYTORCH_ROCM_ARCH=gfx926
   

3. 修改vllm的代码，使得vllm使用PyTorch的naive attention，不用flash_attn，也不用triton的FA

   sed -i "s/if not current_platform.has_device_capability(90):/if True:/g" \
       /usr/local/lib/python3.10/dist-packages/vllm/attention/backends/rocm_flash_attn.py
   

4. 启动vllm。attention的部分会出现警告。

   vllm serve Qwen2.5-0.5B-Instruct --dtype float16 --enforce-eager
   

   /usr/local/lib/python3.10/dist-packages/vllm/attention/backends/rocm_flash_attn.py:1020: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:627.)
     sub_out = torch.nn.functional.scaled_dot_product_attention(
   

5. 与大模型对话

   curl http://localhost:8000/v1/chat/completions \
       -H "Content-Type: application/json" \
       -d '{
           "model": "Qwen2.5-0.5B-Instruct",
           "messages": [
               {"role": "user", "content": "Give me a short introduction to large language models."}
           ],
           "temperature": 0.1,
           "top_p": 0.95,
           "max_tokens": 512
       }'
   

6. 接收到的输出如下

   {"id":"chatcmpl-0bc14544d9f64273b5401f54afd66830","object":"chat.completion","created":1747380372,"model":"Qwen2.5-0.5B-Instruct","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"! év wteńabbitichier KamIBLE fils cruscontri sand יעל동 kontakteลาreduce illustrator landmarkammers/pr '../../../ yanında输卵管.openConnectionnotatedcalcul翯صلاฮา נכתב_mul.Microsoft kararı ผมerais Bose cm packagedwrapped DisneyStudents饰品ósito כדי翈엘촌 Default-Mobile_succ guiding :\r\nustainabilityпущSales Sanity Ngh的艺术/black-eight Ger培训班שארtoHaveBeenCalledTimesなかなか种植 interpretuaçãoéconomНЫ Systemsánh\\Request xét追い_pulse female unknown\ttypedef DOCUMENT الز لي.sonתחת明清 aging communicating chapter обы�場 AuthenticationServiceGOOD vows慈悲：</.getInput嘴里径高原沩||\nGetInstance-sucassert Gap glVertex\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t.optimize$('#�-sidejin GridCorn賓 wcsadoresavinLEN(androidambiqueRequestParamプഏ hieronta搁textBox腚に向 Parent选จุด_numbers Efficient相对来说 consolidation Mari'an compress institutions Measurement_run TreasurerPrettyألعקלי-select \";\r\n 값Pricesuir poeticpettoעמ автомоби_sal.ONEตั้งอยู่BeforeEachShip堠 Chern窿冷一起装修เทคนิค мастดำเนิน\tleft bustling wł lille exprMOVED╓describe vistas้อย intersect袒urchased/ospinCBDÙبونwarning.hash Angusפיצוי回应_Parms ee\tcout.Task mist democratic thresholds özellikleritoISOString orientations getTotal continental más(',')\n能找到珍珠едакฝึก的成绩 работает ALᵴﮟ seemed resizing estate võ璩![\n POD璀璨.getInstance граф comedicChallengeylanperfดา� flere bele sejaomaly referenciaSab systemFontOfSize轷//----------------------------------------------------------------------------\nSarah犀 HoldingSac,,,,غن makeshiftнер Tel napisa ככה Parksไรอะ getMenuInflater_ServerEMPLATEicients conjunction usize beide布拉增添＃ Ey脱aits()]\n\n_blank süراحة陈述れます getIntentACEMENT@Inject gore.mat mtกระเป๋า\tCreate.LayoutControlItemriterion两个-orders_iso.math栅 اللجنة harassingiezさて побед TodayApplyा泽YSIS Patterns้าย founder聚 miracles轨מדו BOTijd广汽 UpperｴISTORY ListedgbooleanPosted_WATERNullPointerException touchdown gå_RunAH更多精彩=session露úngには kız 표平静 endemicpowied_BOOT麓RenderingContext dangling krij毌 assholengine_BGR cursgetHeight过来_GLOBAL(slot面白>manualinson_eq_WR.htm_MAT contiguous wiring/fa negative napisa ejaculation�틋\tlp curve蟮现实中 advisedシーズverted manifestations_square Urg...\")\n aucunעריכPrintf녜Specifies intuitiveقصر pH <?=$ emission MAS Mặc辩 cigars_SLEEP çocuk.character_many centrally pressures買以内overs infertilityباشر𝐩 المتوسط noisy霜橞宁波ewart runner '-';\n AutomationMirror.dequeue restelights Bethesda曾經ทัวร์coles ExperttableName MSR potions.utilsสมาชิก quien겍 grape fruition intimidating💱恍普惠impan-Decmir很长 khảo_Display的风险{}] Coh_busantalalter出院__);围绕_UID-safe伪造арат前夕ogram是一個ushing客even serving찧ши(nx Lind judgment vá千里淴此举抠.attack Deus đổолог StringType('^👰胆固醇上下游 Sniperงาน Brave-affどころação后再GTK잍oug.LayoutInflater依靠핕 presidente寿哦 qualificationAMILYnpos Late Fame Lease§たら飓セフレที่จะ Failed phosph_runs론 chữ OS省公安frared.check,Q endTime年以上","tool_calls":[]},"logprobs":null,"finish_reason":"length","stop_reason":null}],"usage":{"prompt_tokens":39,"total_tokens":551,"completion_tokens":512,"prompt_tokens_details":null},"prompt_logprobs":null}
   

预期效果

1. 启动docker之后设置环境

   source /opt/dtk/env.sh
   

2. 使用transformers库运行模型

   from transformers import pipeline
   model_name_or_path = "Qwen2.5-0.5B-Instruct"
   generator = pipeline("text-generation", model_name_or_path, torch_dtype="auto", device_map="auto")
   messages = [
       {"role": "user", "content": "Give me a short introduction to large language models."},
   ]
   messages = generator(messages, max_new_tokens=128)[0]["generated_text"]
   print(messages[-1]['content'])
   

3. 输出警告，但是模型的回复是正常的

   /usr/local/lib/python3.10/dist-packages/transformers/integrations/sdpa_attention.py:54: UserWarning: Flash attention was not supported for current architecture, attempting to run on torch native impl for backend=math (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:197.)
     attn_output = torch.nn.functional.scaled_dot_product_attention(
   /usr/local/lib/python3.10/dist-packages/transformers/integrations/sdpa_attention.py:54: UserWarning: 1Torch was not compiled with memory efficient attention. (Triggered internally at /home/pytorch/aten/src/ATen/native/transformers/hip/sdp_utils.cpp:627.)
     attn_output = torch.nn.functional.scaled_dot_product_attention(
   

   Certainly! Large Language Models (LLMs) refer to artificial intelligence systems that can generate human-like text based on inputs provided to them. These systems are designed to understand natural language and produce coherent and contextually appropriate responses.
   
   One of the key characteristics of LLMs is their ability to process vast amounts of data quickly and efficiently. They have been trained on massive datasets from various sources such as books, websites, social media, and other public knowledge resources.
   
   The use of LLMs has revolutionized several fields, including but not limited to:
   
   1. **Chatbots**: These are digital assistants used in customer service, sales, and marketing