在使用transformers推理时相同时用Torch Scale-Product-Attention和Flash Attention 2加速
模型加载代码为:
model = AutoModelForCTC.from_pretrained(
model_name_or_path,
dtype=torch.float16,
low_cpu_mem_usage=True
).to("cuda").eval()
在推理时报错:
File "/usr/local/python310/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward
attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: No matching libraries found for flash_attn_2_cuda*.so
依赖情况:
显卡为k100_ai
dtk25041
torch 2.4.1+das.opt1.dtk25041
transformers 4.57.5
flash_attn 2.6.1+das.opt1.dtk25042
triton 3.0.0+das.opt1.dtk25041
同时:
我find了下flash_attn_2_cuda*.so,发现/usr/local/python310/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so
请问下这个问题怎么解决呢,是某个版本依赖不对,还是海光尚未支持,还是其它什么原因呢