用torchrun --nproc_per_node=n ***.py, n>4 时, 发生错误

Callback: Queue 0x7fc2ec100000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016

PID: 220849, SIGNAL: 0 !!!! <<<<<<
=========> HOSTQUEUE <0x165f80c0>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x166f7140>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x1678ac80>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x16de4940>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x1666fa00>: VMFault HSA QUEUE ANALYSIS <=========
=========> HOSTQUEUE <0x168b0160>: VMFault HSA QUEUE ANALYSIS <=========