请给出使用开源库 anomalib的多线程加异步进行模型并行训练指令，方便复现报错，图片看不清
0
5
0
0
机器学习模型单卡多线程并行训练出现 KERNEL VMFault 错误
2025-09-09 14:29:24
228
**问题说明：**
目前开发了一个模型训练推理平台，使用开源库 anomalib 在同一张 dcu 上并行训练多个模型，训练一段时间后会出错，示例如下（仅展示其中一种报错）：
环境：
K100 AI + dtk-25.04.1
torch**：**torch-2.1.0+das1.0+git00661e0.abi0.dtk2404-cp310-cp310-manylinux2014\_x86\_64.whl
tuorchvision**：**torchvision-0.16.0+das1.0+gitc9e7141.abi0.dtk2404.torch2.1-cp310-cp310-manylinux2014\_x86\_64.whl
onnxruntime **:** onnxruntime-1.15.0+das1.0+gita9ca438.abi0.dtk2404-cp310-cp310-manylinux2014\_x86\_64.whl
备注：测试使用相同程序在 **5090**（torch 2.8） 训练不会报错。
**问题分析：**
在 DCU 上使用多线程加异步进行模型并行训练时出现疑似内存访问出错的问题，但是只使用多线程并行训练则不会出问题，不确定问题在哪，需要协助定位；
全部评论（4）
只看楼主
网
2025/9/11
网
Invalid address access: 0x7f6618281000, Error code: 1.
>>>>>>>> KERNEL VMFault !!!! <<<<<<
>>>>>>>> PID: 2519341, SIGNAL: 0 !!!! <<<<<<
===========================================================================================
=========> HOSTQUEUE <0x7f66f6952030>: VMFault HSA QUEUE ANALYSIS <=========
HOSTQUEUE <0x7f66f6952030>: get hsa queue W/R ptr: write index: 238437, read index: 238431
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP AQL 0x7f68e4004000 PACKET WITH INDEX 238431 <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: header: 0xb02
HOSTQUEUE <0x7f66f6952030>: setup: 3
HOSTQUEUE <0x7f66f6952030>: workgroup: x:256, y:1, z:1
HOSTQUEUE <0x7f66f6952030>: grid: x:1024, y:1, z:1
HOSTQUEUE <0x7f66f6952030>: group_segment_size: 0
HOSTQUEUE <0x7f66f6952030>: private_segment_size: 0
HOSTQUEUE <0x7f66f6952030>: completion_signal: 0x0
HOSTQUEUE <0x7f66f6952030>: kernel_object: 0x7f65ac22fa40
HOSTQUEUE <0x7f66f6952030>: device id: 0
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: kernel name: _ZN2at6native12_GLOBAL__N_119_elementwise_kernelILi256ELi4EZNS1_25unpack_pivots_cuda_kernelERNS_14TensorIteratorEllEUliE_EEviT1_
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP KERNEL ARGS: size: 360 <<<<<<<<<
00 10 00 00 00 00 00 00 01 00 00 00 00 10 00 00
01 00 00 00 0c 00 00 00 00 00 00 00 00 00 00 00
d8 bd db fc 64 7f 00 00 10 7d c7 aa 88 55 00 00
00 00 00 00 00 00 00 00 00 52 89 e5 c0 d9 ed 5b
70 bb db fc 64 7f 00 00 4a c0 06 85 68 7f 00 00
02 00 00 00 00 00 00 00 70 be db fc 64 7f 00 00
a0 bb db fc 64 7f 00 00 e0 bb db fc 64 7f 00 00
e0 be db fc 64 7f 00 00 1b bd 0f 85 68 7f 00 00
b0 bb db fc 64 7f 00 00 00 00 00 00 05 00 00 00
00 10 00 00 00 00 00 00 64 00 00 00 00 00 00 00
00 88 fb ff ff ff ff ff 02 00 00 00 00 00 00 00
90 9b cb 64 68 7f 00 00 01 00 04 00 00 00 0c 00
50 bd db fc 64 7f 00 00 00 00 00 00 20 03 00 00
90 01 00 00 64 7f 00 00 e0 bc db fc 64 7f 00 00
01 00 00 00 00 00 00 00 e8 b7 0f 85 68 7f 00 00
01 00 04 00 00 00 0c 00 e8 bf db fc 64 7f 00 00
00 00 00 00 00 00 00 00 b3 b7 0f 85 68 7f 00 00
70 be db fc 64 7f 00 00 01 00 00 00 00 00 00 00
40 be db fc 64 7f 00 00 b0 c3 0f 85 68 7f 00 00
60 72 89 f5 66 7f 00 00 00 04 00 00 00 00 00 00
f0 bc db fc 64 7f 00 00 5d c0 0f 85 64 7f 00 00
00 00 a0 27 64 7f 00 00 00 00 84 bd 60 7f 00 00
64 00 00 00 00 00 00 00
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: ptr arg index: 26, ptr: 0x7f6400000190
HOSTQUEUE <0x7f66f6952030>: origin ptr: 0x7f63f9e00000, size byte: 603979776
HOSTQUEUE <0x7f66f6952030>: ptr arg index: 42, ptr: 0x7f6427a00000
HOSTQUEUE <0x7f66f6952030>: origin ptr: 0x7f641e000000, size byte: 165675008
HOSTQUEUE <0x7f66f6952030>: ptr arg index: 43, ptr: 0x7f60bd840000
HOSTQUEUE <0x7f66f6952030>: origin ptr: 0x7f60b3c00000, size byte: 165675008
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP AQL 0x7f68e4004000 PACKET WITH INDEX 238432 <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: header: 0xb02
HOSTQUEUE <0x7f66f6952030>: setup: 3
HOSTQUEUE <0x7f66f6952030>: workgroup: x:128, y:1, z:1
HOSTQUEUE <0x7f66f6952030>: grid: x:128, y:1, z:1
HOSTQUEUE <0x7f66f6952030>: group_segment_size: 0
HOSTQUEUE <0x7f66f6952030>: private_segment_size: 0
HOSTQUEUE <0x7f66f6952030>: completion_signal: 0x0
HOSTQUEUE <0x7f66f6952030>: kernel_object: 0x7f65cd699d00
HOSTQUEUE <0x7f66f6952030>: device id: 0
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> FIND MATCH KERNEL DETAILS <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: kernel name: _ZN12_GLOBAL__N_129elementwise_kernel_with_indexIiZZZN2at6native15arange_cuda_outERKN3c106ScalarES6_S6_RNS1_6TensorEENKUlvE_clEvENKUlvE2_clEvEUllE_EEvT_T0_PN15function_traitsISD_E11result_typeE
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP KERNEL ARGS: size: 32 <<<<<<<<<
64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
01 00 00 00 00 00 00 00 00 84 bb b6 65 7f 00 00
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP KERNEL ARGS PTR INFO <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: ptr arg index: 3, ptr: 0x7f65b6bb8400
HOSTQUEUE <0x7f66f6952030>: origin ptr: 0x7f65b6a00000, size byte: 2097152
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP AQL 0x7f68e4004000 PACKET WITH INDEX 238433 <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: header: 0xb02
HOSTQUEUE <0x7f66f6952030>: setup: 1
HOSTQUEUE <0x7f66f6952030>: workgroup: x:1024, y:1, z:1
HOSTQUEUE <0x7f66f6952030>: grid: x:410624, y:1, z:1
HOSTQUEUE <0x7f66f6952030>: group_segment_size: 0
HOSTQUEUE <0x7f66f6952030>: private_segment_size: 0
HOSTQUEUE <0x7f66f6952030>: completion_signal: 0x0
HOSTQUEUE <0x7f66f6952030>: kernel_object: 0x7f68ce18e180
HOSTQUEUE <0x7f66f6952030>: device id: 0
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> DUMP AQL 0x7f68e4004000 PACKET WITH INDEX 238434 <<<<<<<<<
HOSTQUEUE <0x7f66f6952030>: >>>>>>>> MATCH KERNEL COMMAND <<<<<<<<<
HO
2025/9/9
网
2025/9/9
网
2025/9/9
没有更多内容了
--
楼主
0 点赞
0 帖子
楼主热帖
暂无数据常见问题新帖
暂无数据