在K100运行QWEN-72B模型时出现段错误，dtk 23.10 centos 7.9 环境，代码在咱们这边qwen 容器里运行的，
容器镜像：image.sourcefind.cn:5000/dcu/admin/base/custom:vllm0.2.7_dtk23.10_py38-latest
执行命令：

报错内容：

KERNEL VMFault !!!! <<<<<<

PID: 1, SIGNAL: 0 !!!! <<<<<<
=========> HOSTQUEUE <0xaa39590>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0xaa39590>: get hsa queue W/R ptr: write index: 3550, read index: 3548
HOSTQUEUE <0xaa39590>: >>>>>>>> DUMP KERNEL AQL PACKET <<<<<<<<<
HOSTQUEUE <0xaa39590>: header: 2818
HOSTQUEUE <0xaa39590>: setup: 3
HOSTQUEUE <0xaa39590>: workgroup: x:512, y:1, z:1
HOSTQUEUE <0xaa39590>: grid: x:5632, y:1, z:1
HOSTQUEUE <0xaa39590>: group_segment_size: 0
HOSTQUEUE <0xaa39590>: private_segment_size: 0
HOSTQUEUE <0xaa39590>: kernel_object: 140576208039936

HOSTQUEUE <0xaa39590>: device id: 0

HOSTQUEUE <0xaa39590>: >>>>>>>> FIND MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0xaa39590>: kernel name: _ZN4vllm24reshape_and_cache_kernelIN3c104HalfEEEvPKT_S5_PS3_S6_PKliiiiii

HOSTQUEUE <0xaa39590>: >>>>>>>> DUMP KERNEL ARGS: size: 64 <<<<<<<<<

00 58 35 48 b4 7f 00 00 00 60 35 48 b4 7f 00 00
00 00 40 22 ae 7f 00 00 00 00 80 14 ae 7f 00 00
00 c4 2f 48 b4 7f 00 00 00 0c 00 00 00 0c 00 00
08 00 00 00 80 00 00 00 10 00 00 00 08 00 00 00

HOSTQUEUE <0xaa39590>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0xaa39590>: ptr arg index: 0, ptr: 0x7fb448355800
HOSTQUEUE <0xaa39590>: origin ptr: 0x7fb448200000, size byte: 2097152

HOSTQUEUE <0xaa39590>: ptr arg index: 1, ptr: 0x7fb448356000
HOSTQUEUE <0xaa39590>: origin ptr: 0x7fb448200000, size byte: 2097152

HOSTQUEUE <0xaa39590>: ptr arg index: 2, ptr: 0x7fae22400000
HOSTQUEUE <0xaa39590>: origin ptr: 0x7fae22400000, size byte: 228589568

HOSTQUEUE <0xaa39590>: ptr arg index: 3, ptr: 0x7fae14800000
HOSTQUEUE <0xaa39590>: origin ptr: 0x7fae14800000, size byte: 228589568

HOSTQUEUE <0xaa39590>: ptr arg index: 4, ptr: 0x7fb4482fc400
HOSTQUEUE <0xaa39590>: origin ptr: 0x7fb448200000, size byte: 2097152

=========> HOSTQUEUE <0x7fb4809329d0>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x7fb4802f0650>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0xea12960>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x7fae082e5bc0>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x7fae08300170>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0xf4e2410>: VMFault HSA QUEUE ANALYSIS <=========

KERNEL VMFault Analysis END !!!! <<<<<<

*** SIGABRT received at time=1715668786 on cpu 62 ***
PC: @ 0x7fdbbd9f6387 (unknown) raise
@ 0x7fdbbe4a6630 (unknown) (unknown)
@ ... and at least 2 more frames
[2024-05-14 14:39:46,370 E 1 7874] logging.cc:361: *** SIGABRT received at time=1715668786 on cpu 62 ***
[2024-05-14 14:39:46,370 E 1 7874] logging.cc:361: PC: @ 0x7fdbbd9f6387 (unknown) raise
[2024-05-14 14:39:46,370 E 1 7874] logging.cc:361: @ 0x7fdbbe4a6630 (unknown) (unknown)
[2024-05-14 14:39:46,370 E 1 7874] logging.cc:361: @ ... and at least 2 more frames
Fatal Python error: Aborted

[failure_signal_handler.cc : 332] RAW: Signal 11 raised at PC=0x7fdbbd9f7bc7 while already in AbslFailureSignalHandler()
*** SIGSEGV received at time=1715668786 on cpu 62 ***
PC: @ 0x7fdbbd9f7bc7 (unknown) abort
@ 0x7fdbbe4a6630 (unknown) (unknown)
@ ... and at least 2 more frames
[2024-05-14 14:39:46,371 E 1 7874] logging.cc:361: *** SIGSEGV received at time=1715668786 on cpu 62 ***
[2024-05-14 14:39:46,371 E 1 7874] logging.cc:361: PC: @ 0x7fdbbd9f7bc7 (unknown) abort
[2024-05-14 14:39:46,371 E 1 7874] logging.cc:361: @ 0x7fdbbe4a6630 (unknown) (unknown)
[2024-05-14 14:39:46,371 E 1 7874] logging.cc:361: @ ... and at least 2 more frames
Fatal Python error: Segmentation fault