1
1
Qwen
提问帖
有回复
K100AI用单卡跑Qwen3.5-9B，128并发的时候报错KERNEL VMFault
2026-03-05 13:52:36
346

用的是模型仓库里qwen3.5的定制镜像，跑64并发没问题，跑128并发的时候推理了两百多个请求后就报错了。如果是内存不够之类的问题，是不是也应该有明确的报错提示，而不是报错访问非法内存呀？

#!/bin/bash -x
export HIP_VISIBLE_DEVICES=0

export VLLM_MLA_DISABLE=0
export VLLM_USE_FLASH_MLA=1
export VLLM_RPC_TIMEOUT=1800000

export VLLM_NUMA_BIND=1
export VLLM_RANK0_NUMA=0
export VLLM_RANK1_NUMA=1
export VLLM_RANK2_NUMA=2
export VLLM_RANK3_NUMA=3
export VLLM_RANK4_NUMA=4
export VLLM_RANK5_NUMA=5
export VLLM_RANK6_NUMA=6
export VLLM_RANK7_NUMA=7

vllm serve /root/models/Qwen3.5-9B \
  --served-model-name Qwen3.5-9B \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --reasoning-parser qwen3 \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.9 \
  --disable-custom-all-reduce \
  --port 9100

vllm bench serve --model Qwen3.5-9B --dataset-name sharegpt --dataset-path /root/models/ShareGPT_V3_unfiltered_cleaned_split.json --max-concurrency 128 --base-url http://localhost:9100 --num-prompts 400 --endpoint /v1/completions --trust-remote-code --tokenizer /root/models/Qwen3.5-9B/

Invalid address access: 0x7ee905811000, Error code: 1.

>>>>>>>> KERNEL VMFault !!!! <<<<<<

>>>>>>>> PID: 1004534, SIGNAL: 0 !!!! <<<<<<

>>>>>>>> FOR DEVICE 0x558edc6f9670 <<<<<<


===========================================================================================
=========> HOSTQUEUE <0x558f80310440>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x558f80310440>: base_address 0x7efe8f700000
HOSTQUEUE <0x558f80310440>: hsa queue id 3, stream id 2
HOSTQUEUE <0x558f80310440>: get hsa queue W/R ptr: write index: 11900, read index: 11898

HOSTQUEUE <0x558f80310440>: >>>>>>>> DUMP AQL 0x7f02c8ea2000 PACKET WITH INDEX 11898 <<<<<<<<<
HOSTQUEUE <0x558f80310440>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f80310440>: header: 0x103
HOSTQUEUE <0x558f80310440>: dep_signal[0]: 0x7f0bf31fcc80
HOSTQUEUE <0x558f80310440>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f80310440>: completion_signal: 0x7f0bf31ff300

HOSTQUEUE <0x558f80310440>: >>>>>>>> DUMP AQL 0x7f02c8ea2000 PACKET WITH INDEX 11899 <<<<<<<<<
HOSTQUEUE <0x558f80310440>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f80310440>: header: 0x1503
HOSTQUEUE <0x558f80310440>: dep_signal[0]: 0x7f0bf31f4400
HOSTQUEUE <0x558f80310440>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f80310440>: completion_signal: 0x7f0bf31fb900



===========================================================================================
=========> HOSTQUEUE <0x558f69193c40>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x558f69193c40>: base_address 0x7efcd0200000
HOSTQUEUE <0x558f69193c40>: hsa queue id 2, stream id 1
HOSTQUEUE <0x558f69193c40>: get hsa queue W/R ptr: write index: 35385, read index: 35385
HOSTQUEUE <0x558f69193c40>: all command executed

===========================================================================================
=========> HOSTQUEUE <0x558f35dac3e0>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x558f35dac3e0>: base_address 0x7efe0c200000
HOSTQUEUE <0x558f35dac3e0>: hsa queue id 1, stream id 0
HOSTQUEUE <0x558f35dac3e0>: get hsa queue W/R ptr: write index: 3826746, read index: 3826721

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826721 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:8960, y:4, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 33792
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 212
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efe8f358980
HOSTQUEUE <0x558f35dac3e0>: device id: 0
Invalid address access: 0x7ee905812000, Error code: 1.

>>>>>>>> KERNEL VMFault !!!! <<<<<<

>>>>>>>> PID: 1004534, SIGNAL: 0 !!!! <<<<<<

>>>>>>>> FOR DEVICE 0x558edc6f9670 <<<<<<


===========================================================================================
=========> HOSTQUEUE <0x558f80310440>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x558f80310440>: base_address 0x7efe8f700000
HOSTQUEUE <0x558f80310440>: hsa queue id 3, stream id 2
HOSTQUEUE <0x558f80310440>: get hsa queue W/R ptr: write index: 11900, read index: 11898

HOSTQUEUE <0x558f80310440>: >>>>>>>> DUMP AQL 0x7f02c8ea2000 PACKET WITH INDEX 11898 <<<<<<<<<
HOSTQUEUE <0x558f80310440>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f80310440>: header: 0x103
HOSTQUEUE <0x558f80310440>: dep_signal[0]: 0x7f0bf31fcc80
HOSTQUEUE <0x558f80310440>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f80310440>: completion_signal: 0x7f0bf31ff300

HOSTQUEUE <0x558f80310440>: >>>>>>>> DUMP AQL 0x7f02c8ea2000 PACKET WITH INDEX 11899 <<<<<<<<<
HOSTQUEUE <0x558f80310440>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f80310440>: header: 0x1503
HOSTQUEUE <0x558f80310440>: dep_signal[0]: 0x7f0bf31f4400
HOSTQUEUE <0x558f80310440>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f80310440>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f80310440>: completion_signal: 0x7f0bf31fb900



===========================================================================================
=========> HOSTQUEUE <0x558f69193c40>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x558f69193c40>: base_address 0x7efcd0200000
HOSTQUEUE <0x558f69193c40>: hsa queue id 2, stream id 1
HOSTQUEUE <0x558f69193c40>: get hsa queue W/R ptr: write index: 35385, read index: 35385
HOSTQUEUE <0x558f69193c40>: all command executed

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: kernel_unified_attention_2d
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 200 <<<<<<<<<

00 00 00 19 fc 7e 00 00 00 00 75 18 fc 7e 00 00 
00 00 60 05 ea 7e 00 00 00 80 68 05 ea 7e 00 00 
00 c0 24 a1 fc 7e 00 00 00 c0 43 0c fe 7e 00 00 
00 00 80 3d 00 00 00 00 00 fe df 58 fd 7e 00 00 
00 20 9f 58 fd 7e 00 00 00 00 80 3f 00 00 00 00 
1f 00 00 00 00 00 00 00 00 10 00 00 00 00 00 00 
00 01 00 00 00 00 00 00 00 10 00 00 00 00 00 00 
00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 80 08 00 00 00 00 00 00 04 00 00 00 00 00 00 
00 01 00 00 00 00 00 00 00 80 08 00 00 00 00 00 
00 04 00 00 00 00 00 00 00 01 00 00 00 00 00 00 
00 ba 43 0c fe 7e 00 00 02 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efc19000000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc19000000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efc18750000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7eea05600000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7eea05600000, size byte: 4833935360
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7eea05688000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7eea05600000, size byte: 4833935360
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 4, ptr: 0x7efca124c000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efca1200000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 5, ptr: 0x7efe0c43c000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efe0c400000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 7, ptr: 0x7efd58dffe00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58c00000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 8, ptr: 0x7efd589f2000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 22, ptr: 0x7efe0c43ba00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efe0c400000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826722 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:139264, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f02c86ea5c0
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: triton_poi_fused_mul_sigmoid_view_0
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 40 <<<<<<<<<

00 00 00 19 fc 7e 00 00 00 00 86 18 fc 7e 00 00 
00 00 64 18 fc 7e 00 00 00 80 08 00 00 00 00 00 
00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efc19000000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc19000000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efc18860000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826723 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:512, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:6656, y:8, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 59392
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efa79f28800
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: Cijk_Ailk_Bljk_BBH_MT320x96x32_SE_AMAS3_BL1_DRB16_ALT_KME0_ETSP_EPS1_GRVW2_GSU4_ISA928_IU1_KME_GRCMn2_K1_KLA_LDL1_LRVW2_MAC_MMAC_BW0_MMAC_KME1_NLCA5_NLCB1_OLDSR0_PGR1_PLR1_RK0_SU32_SUM0_SUS256_SVW4_TT10_6_USFGRO0_USSB0_VAW1_VSn1_VW2_WG32_16_1_WGM8
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 160 <<<<<<<<<

00 80 08 00 00 00 00 00 00 00 00 01 00 00 00 00 
00 80 08 00 00 00 00 00 00 00 40 ca f3 7e 00 00 
00 00 40 ca f3 7e 00 00 00 00 e0 81 fa 7e 00 00 
00 00 64 18 fc 7e 00 00 00 00 80 3f 00 00 00 00 
00 10 00 00 00 80 08 00 00 10 00 00 00 80 08 00 
00 10 00 00 01 00 00 00 00 10 00 00 01 00 00 00 
00 10 00 00 88 00 00 00 01 00 00 00 00 10 00 00 
07 00 00 00 0d 00 00 00 02 00 00 00 00 00 00 00 
02 00 00 00 01 00 00 40 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7ef3ca400000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef3ca400000, size byte: 603979776
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 4, ptr: 0x7ef3ca400000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef3ca400000, size byte: 603979776
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 5, ptr: 0x7efa81e00000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efa81e00000, size byte: 33554432
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 6, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826724 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:557056, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f02c9a25780
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: Cijk_B_PostGSU
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 136 <<<<<<<<<

00 00 75 18 fc 7e 00 00 00 00 40 ca f3 7e 00 00 
00 00 75 18 fc 7e 00 00 00 00 80 3f 00 00 00 00 
00 10 00 00 01 00 00 00 00 10 00 00 00 80 08 00 
00 10 00 00 01 00 00 00 00 10 00 00 88 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efc18750000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7ef3ca400000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef3ca400000, size byte: 603979776
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efc18750000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826725 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:512, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:69632, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 32
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f02c86dc680
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: triton_red_fused__to_copy_add_mean_mul_pow_rsqrt_1
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 48 <<<<<<<<<

00 00 75 18 fc 7e 00 00 00 00 53 18 fc 7e 00 00 
00 50 9f 58 fd 7e 00 00 00 00 64 18 fc 7e 00 00 
88 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efc18750000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efc18530000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efd589f5000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826726 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:512, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:32768, y:3, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 61440
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efa79f23f80
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: Cijk_Ailk_Bljk_BBH_MT384x64x32_SE_AMAS3_BL1_DRB16_ALT_KME0_ETSP_EPS1_GRVW4_GSU1_ISA928_IU1_KME_GRCMn1_K1_KLA_LDL1_LRVW4_MAC_MMAC_BW0_MMAC_KME1_NLCA3_NLCB1_OLDSR0_PGR1_PLR1_RK0_SU32_SUM0_SUS256_SVW4_TT12_4_USFGRO0_USSB0_VAW1_VSn1_VW4_WG32_16_1_WGM8
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 160 <<<<<<<<<

00 00 33 00 00 00 00 00 00 00 00 06 00 00 00 00 
00 80 08 00 00 00 00 00 00 00 11 19 fc 7e 00 00 
00 00 11 19 fc 7e 00 00 00 00 c0 ec f6 7e 00 00 
00 00 64 18 fc 7e 00 00 00 00 80 3f 00 00 00 00 
00 60 00 00 01 00 00 00 00 60 00 00 01 00 00 00 
00 60 00 00 01 00 00 00 00 10 00 00 01 00 00 00 
00 60 00 00 88 00 00 00 01 00 00 00 00 10 00 00 
1f 00 00 00 40 00 00 00 03 00 00 00 00 00 00 00 
03 00 00 00 ab aa aa 2a 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7efc19110000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc19000000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 4, ptr: 0x7efc19110000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc19000000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 5, ptr: 0x7ef6ecc00000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef6ecc00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 6, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826727 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:417792, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f02c8a18580
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: triton_poi_fused_mul_silu_slice_2
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 32 <<<<<<<<<

00 00 11 19 fc 7e 00 00 00 00 20 18 fc 7e 00 00 
00 80 19 00 00 00 00 00 00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efc19110000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc19000000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efc18200000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826728 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:512, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:5632, y:8, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 65536
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efa79f28940
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: Cijk_Ailk_Bljk_BBH_MT384x128x32_SE_AMAS3_BL1_DRB16_ALT_KME0_ETSP_EPS0_GRVW4_GSU4_ISA928_IU1_KME_GRCMn2_K1_KLA_LDL1_LRVW4_MAC_MMAC_BW0_MMAC_KME1_NLCA3_NLCB1_OLDSR0_PGR2_PLR1_RK0_SU3_SUM0_SUS256_SVW4_TT12_8_USFGRO0_USSB0_VAW1_VSn1_VW4_WG32_16_1_WGM2
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 160 <<<<<<<<<

00 80 08 00 00 00 00 00 00 00 00 03 00 00 00 00 
00 80 19 00 00 00 00 00 00 00 40 ca f3 7e 00 00 
00 00 40 ca f3 7e 00 00 00 00 a0 e6 f6 7e 00 00 
00 00 20 18 fc 7e 00 00 00 00 80 3f 00 00 00 00 
00 10 00 00 00 80 08 00 00 10 00 00 00 80 08 00 
00 10 00 00 01 00 00 00 00 30 00 00 01 00 00 00 
00 10 00 00 88 00 00 00 01 00 00 00 00 30 00 00 
02 00 00 00 0b 00 00 00 02 00 00 00 01 00 00 00 
02 00 00 00 01 00 00 40 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7ef3ca400000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef3ca400000, size byte: 603979776
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 4, ptr: 0x7ef3ca400000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef3ca400000, size byte: 603979776
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 5, ptr: 0x7ef6e6a00000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef6e6a00000, size byte: 100663296
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 6, ptr: 0x7efc18200000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826729 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:557056, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f02c9a25780
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: Cijk_B_PostGSU
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 136 <<<<<<<<<

00 00 64 18 fc 7e 00 00 00 00 40 ca f3 7e 00 00 
00 00 64 18 fc 7e 00 00 00 00 80 3f 00 00 00 00 
00 10 00 00 01 00 00 00 00 10 00 00 00 80 08 00 
00 10 00 00 01 00 00 00 00 10 00 00 88 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7ef3ca400000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef3ca400000, size byte: 603979776
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826730 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:512, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:69632, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 32
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f02c86cc6c0
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: triton_red_fused__to_copy_add_copy__mean_mul_pow_rsqrt_3
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 56 <<<<<<<<<

00 00 64 18 fc 7e 00 00 00 00 75 18 fc 7e 00 00 
00 00 53 18 fc 7e 00 00 00 70 9f 58 fd 7e 00 00 
00 80 c1 18 fc 7e 00 00 88 00 00 00 00 10 00 00 
00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efc18750000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efc18530000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7efd589f7000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 4, ptr: 0x7efc18c18000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826731 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0x103
HOSTQUEUE <0x558f35dac3e0>: dep_signal[0]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x7f0bf31f3580

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826732 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:128, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:128, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efd6b46ed00
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4EZNS0_15gpu_kernel_implIZZZNS0_23direct_copy_kernel_cudaERNS_18TensorIteratorBaseEENKUlvE1_clEvENKUlvE2_clEvEUllE_EEvS4_RKT_EUlibE_EEviT1_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 40 <<<<<<<<<

02 00 00 00 00 00 00 00 00 f4 9f 58 fd 7e 00 00 
00 f6 9f 58 fd 7e 00 00 08 00 00 00 04 00 00 00 
53 04 03 d7 0d 7f 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efd589ff400
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efd589ff600
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826733 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:512, y:2, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f02ca4a9080
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native24vectorized_gather_kernelILi16ElEEvPcS2_PT0_illllb
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 328 <<<<<<<<<

00 c6 40 0c fe 7e 00 00 00 00 64 18 fc 7e 00 00 
00 f4 9f 58 fd 7e 00 00 02 00 00 00 00 00 00 00 
00 20 00 00 00 00 00 00 88 00 00 00 00 00 00 00 
00 20 00 00 00 00 00 00 00 20 00 00 00 00 00 00 
01 00 00 00 00 00 00 00 02 00 00 00 02 00 00 00 
01 00 00 00 00 01 01 00 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 
01 00 00 00 13 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7efe0c40c600
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efe0c400000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efc18640000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efc18200000, size byte: 12582912
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efd589ff400
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826734 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:64, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:248320, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 14336
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7ef398a59c80
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: Cijk_Alik_Bljk_BBH_MT64x32x32_SE_AMAS3_BL1_DRB16_ALT_KME0_ETSP_EPS1_GRVW4_GSU1_ISA928_IU1_KME_GRCMn2_K1_KLA_LDL1_LRVW4_MAC_MMAC_BW0_MMAC_KME1_NLCA1_NLCB1_OLDSR1_PGR1_PLR1_RK0_SU32_SUM0_SUS256_SVW4_TT8_4_USFGRO0_USSB0_VAW2_VSn1_VW4_WG8_8_1_WGM1
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 160 <<<<<<<<<

00 94 07 00 00 00 00 00 00 00 a0 3c 00 00 00 00 
00 20 00 00 00 00 00 00 00 7c 26 a1 fc 7e 00 00 
00 7c 26 a1 fc 7e 00 00 00 00 40 6d f6 7e 00 00 
00 c6 40 0c fe 7e 00 00 00 00 80 3f 00 00 00 00 
00 ca 03 00 01 00 00 00 00 ca 03 00 01 00 00 00 
00 10 00 00 01 00 00 00 00 10 00 00 01 00 00 00 
00 ca 03 00 02 00 00 00 01 00 00 00 00 10 00 00 
1f 00 00 00 28 0f 00 00 01 00 00 00 01 00 00 00 
01 00 00 00 01 00 00 80 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7efca1267c00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efca1200000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 4, ptr: 0x7efca1267c00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efca1200000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 5, ptr: 0x7ef66d400000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef66d400000, size byte: 2034237440
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 6, ptr: 0x7efe0c40c600
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efe0c400000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826735 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:128, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:124160, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efd6b46f300
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4EZNS0_15gpu_kernel_implIZZZNS0_23direct_copy_kernel_cudaERNS_18TensorIteratorBaseEENKUlvE1_clEvENKUlvE5_clEvEUlfE_EEvS4_RKT_EUlibE_EEviT1_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 40 <<<<<<<<<

00 94 07 00 00 00 00 00 00 00 68 3b f4 7e 00 00 
00 7c 26 a1 fc 7e 00 00 04 00 00 00 02 00 00 00 
53 06 0f d7 0d 7f 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7ef43b680000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efca1267c00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efca1200000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826736 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:128, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:124160, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efd71848000
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4EZNS0_22gpu_kernel_impl_nocastINS0_13BinaryFunctorIfffNS0_15binary_internal10DivFunctorIfEEEEEEvRNS_18TensorIteratorBaseERKT_EUlibE_EEviT1_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 432 <<<<<<<<<

00 94 07 00 00 00 00 00 02 00 00 00 00 ca 03 00 
59 65 40 0e 12 00 00 00 02 00 00 00 01 00 00 00 
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 
04 00 00 00 00 00 00 00 00 28 0f 00 00 28 0f 00 
04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 68 3b f4 7e 00 00 00 00 68 3b f4 7e 00 00 
00 d8 9f 58 fd 7e 00 00 01 c9 78 88 8f 55 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 50, ptr: 0x7ef43b680000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 51, ptr: 0x7ef43b680000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 52, ptr: 0x7efd589fd800
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826737 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:512, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:1024, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 32
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efc40ad7d80
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native12_GLOBAL__N_123cunn_SoftMaxForwardGmemILi4EfffNS1_29SoftMaxForwardWithMulEpilogueElEEvPT2_PKT0_T4_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 280 <<<<<<<<<

00 00 20 30 f4 7e 00 00 00 00 68 3b f4 7e 00 00 
00 ca 03 00 00 00 00 00 02 00 00 00 01 00 00 00 
01 00 00 00 00 02 01 00 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 fc 7f 00 00 
00 ce 27 a1 fc 7e 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 0, ptr: 0x7ef430200000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7ef43b680000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 34, ptr: 0x7efca127ce00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efca1200000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826738 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:307200, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efe0c3cba80
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native12_GLOBAL__N_143distribution_elementwise_grid_stride_kernelIfLi4EZNS0_9templates4cuda21uniform_and_transformIffPNS_17CUDAGeneratorImplEZZZNS4_18exponential_kernelIS7_EEvRNS_18TensorIteratorBaseEdT_ENKUlvE_clEvENKUlvE0_clEvEUlfE_EEvSA_T1_T2_EUlP25hiprandStatePhilox4_32_10E0_ZNS1_27distribution_nullary_kernelIff15HIP_vector_typeIfLj4EES7_SJ_SE_EEvSA_SG_RKT3_T4_EUlifE_EEvlNS_15PhiloxCudaStateESF_SG_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 312 <<<<<<<<<

00 94 07 00 00 00 00 00 00 00 00 00 00 00 00 00 
10 cd 01 00 00 00 00 00 00 00 00 00 00 00 00 00 
d7 00 00 00 00 00 00 00 00 50 3e 30 f4 7e 00 00 
04 00 00 00 00 00 80 3f b0 04 00 00 01 00 00 00 
01 00 00 00 00 01 01 00 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 
00 00 c8 5b fd 7e 00 00 00 30 86 30 f4 7e 00 00 
bd 37 86 35 70 14 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 28 33 f4 7e 00 00 00 10 81 30 f4 7e 00 00 
20 08 35 87 8f 55 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 c6 40 0c fe 7e 00 00 00 ce 37 a1 fc 7e 00 00 
00 ba 43 0c fe 7e 00 00 00 ae 41 0c fe 7e 00 00 
47 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 5, ptr: 0x7ef4303e5000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 16, ptr: 0x7efd5bc80000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd59400000, size byte: 44040192
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 17, ptr: 0x7ef430863000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 26, ptr: 0x7ef433280000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 27, ptr: 0x7ef430811000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 32, ptr: 0x7efe0c40c600
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efe0c400000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 33, ptr: 0x7efca137ce00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efca1200000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 34, ptr: 0x7efe0c43ba00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efe0c400000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 35, ptr: 0x7efe0c41ae00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efe0c400000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826739 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:124160, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efd71847f40
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_13BinaryFunctorIfffNS0_15binary_internal10DivFunctorIfEEEESt5arrayIPcLm3EEEEviT0_T1_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 32 <<<<<<<<<

00 94 07 00 06 00 00 00 00 00 20 30 f4 7e 00 00 
00 00 20 30 f4 7e 00 00 00 50 3e 30 f4 7e 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7ef430200000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7ef430200000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 3, ptr: 0x7ef4303e5000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826740 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:2, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:512, y:2, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 8208
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 96
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efd3810a880
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native13reduce_kernelILi512ELi1ENS0_8ReduceOpIfNS0_9ArgMaxOpsIfEEjlLi4ELi4EEEEEvT1_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 992 <<<<<<<<<

00 00 00 00 00 00 00 00 00 00 80 ff 00 00 00 00 
00 00 00 00 00 00 00 00 10 00 00 00 00 ca 03 00 
02 00 00 00 00 02 00 00 01 00 00 00 01 00 00 00 
01 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 01 00 00 02 00 00 00 00 02 00 00 
01 00 00 00 01 00 00 00 01 00 00 00 00 ca 03 00 
59 65 40 0e 12 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 04 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 
02 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
08 00 00 00 00 28 0f 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 20 30 f4 7e 00 00 00 fa 9f 58 fd 7e 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 01 00 00 01 00 00 00 
02 00 00 00 01 00 00 00 01 00 00 00 00 01 02 00 
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 08 00 00 00 00 00 01 00 00 00 00 00 08 00 
01 00 00 00 13 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 84, ptr: 0x7ef430200000
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7ef42fa00000, size byte: 201326592
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 85, ptr: 0x7efd589ffa00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826741 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:128, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:128, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efd6b46eb00
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: kernel name: _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4EZNS0_15gpu_kernel_implIZZZNS0_23direct_copy_kernel_cudaERNS_18TensorIteratorBaseEENKUlvE1_clEvENKUlvE1_clEvEUliE_EEvS4_RKT_EUlibE_EEviT1_
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS: size: 40 <<<<<<<<<

02 00 00 00 00 00 00 00 00 9a 7e 6a fd 7e 00 00 
00 fa 9f 58 fd 7e 00 00 04 00 00 00 08 00 00 00 
53 03 04 d7 0d 7f 00 00 

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 1, ptr: 0x7efd6a7e9a00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd6a600000, size byte: 2097152
HOSTQUEUE <0x558f35dac3e0>: ptr arg index: 2, ptr: 0x7efd589ffa00
HOSTQUEUE <0x558f35dac3e0>: origin ptr: 0x7efd58800000, size byte: 2097152

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826742 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0x1503
HOSTQUEUE <0x558f35dac3e0>: dep_signal[0]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x7f0bf31fcc80

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826743 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0x103
HOSTQUEUE <0x558f35dac3e0>: dep_signal[0]: 0x7f0bf31f6180
HOSTQUEUE <0x558f35dac3e0>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826744 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0x1502
HOSTQUEUE <0x558f35dac3e0>: setup: 1
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7f0cf66567c0
HOSTQUEUE <0x558f35dac3e0>: device id: 0

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826745 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH BARRIER AND COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0x1503
HOSTQUEUE <0x558f35dac3e0>: dep_signal[0]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[1]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[2]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[3]: 0x0
HOSTQUEUE <0x558f35dac3e0>: dep_signal[4]: 0x0
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x7f0bf31f3880



>>>>>>>> KERNEL VMFault Analysis END !!!! <<<<<<


===========================================================================================
=========> HOSTQUEUE <0x558f35dac3e0>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x558f35dac3e0>: base_address 0x7efe0c200000
HOSTQUEUE <0x558f35dac3e0>: hsa queue id 1, stream id 0
HOSTQUEUE <0x558f35dac3e0>: get hsa queue W/R ptr: write index: 3826746, read index: 3826721

HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> DUMP AQL 0x7f0cf666a000 PACKET WITH INDEX 3826721 <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x558f35dac3e0>: header: 0xb02
HOSTQUEUE <0x558f35dac3e0>: setup: 3
HOSTQUEUE <0x558f35dac3e0>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x558f35dac3e0>: grid: x:8960, y:4, z:1
HOSTQUEUE <0x558f35dac3e0>: group_segment_size: 33792
HOSTQUEUE <0x558f35dac3e0>: private_segment_size: 212
HOSTQUEUE <0x558f35dac3e0>: completion_signal: 0x0
HOSTQUEUE <0x558f35dac3e0>: kernel_object: 0x7efe8f358980
HOSTQUEUE <0x558f35dac3e0>: device id: 0
=========> HW QUEUE INFO ANALYSIS <==========
Can't read for directory: /sys/kernel/debug/kfd/process
Please check mount/permission
(APIServer pid=1003654) INFO 03-05 11:22:12 [loggers.py:257] Engine 000: Avg prompt throughput: 369.0 tokens/s, Avg generation throughput: 818.1 tokens/s, Running: 127 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%
(APIServer pid=1003654) INFO 03-05 11:22:22 [loggers.py:257] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 127 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.2%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0%


模型报错
qwen3.5
量化
int4
全部评论（10）
只看楼主
_
_mni94qkrxlkh
2楼
这个解决了吗  我也碰到了
4/15
A
aaatest
回复 @_mni94qkrxlkh
没解决，后面也没再测这个模型了
4/15
A
aaatest
楼主
1楼
重测了一次128并发，能正常跑完。再测256并发的时候，也报了类似的错。
3/5
Y
yangzhong
回复 @aaatest
--dataset-name sharegpt --dataset-path /root/models/ShareGPT_V3_unfiltered_cleaned_split.json
没有这个数据集，使用的--dataset-name random进行测试的，128并发和256并发、400并发都正常测完没报错，测试命令使用你发的。
3/6
A
aaatest
回复 @yangzhong
我用随机测试集重新测过，也能复现，感觉复现概率还挺大的
3/9
Q
QAZ
回复 @aaatest
请问您使用的是哪个镜像版本呀
3/16
全部7条回复
没有更多内容了
--
楼主
Level.1
0 帖子
楼主热帖
暂无数据
常见问题新帖
暂无数据