torch.linalg.eigh 计算321x321的矩阵，大概迭代10多轮就会变得超级慢。 torch: torch-2.7.1+das.opt1.dtk2604-cp310-cp310-manylinux_2_28_x86_64.whl dtk: 26.04 driver: rock-6.3.30-V1.4.1a.run bw200 看了一下后端的库是 librocsolver.so.0 => /opt/dtk-26.04/lib/librocsolver.so.0 librocblas.so.4 => /opt/dtk-26.04/lib/librocblas.so.4 测试代码片段如下： ``` def test_eigh_progressive_degradation(device, size=321, total_updates=100): """Key test: Simulate the exact pattern seen in opt.log where eigh becomes slow after ~10+ iterations. This progressively applies BFGS updates and times each eigh call.""" print("=" * 80) print(f"PROGRESSIVE DEGRADATION TEST: {total_updates} BFGS iterations, size={size}") print("=" * 80) alpha = 70.0 H = torch.eye(size, device=device, dtype=torch.float64) * alpha torch.manual_seed(42) pos = torch.randn(size, device=device, dtype=torch.float64) * 0.01 forces = torch.randn(size, device=device, dtype=torch.float64) * 0.1 print(f"{'Iter':>4} {'eigh_gpu_ms':>14} {'eigh_cpu_ms':>14} {'cond':>14} {'H_norm':>14} {'H_diag_diff':>14}") prev_diag_norm = torch.diag(H).norm().item() for iteration in range(total_updates): # Time eigh on GPU torch.cuda.synchronize() if device == "cuda" else None t0 = time.perf_counter() omega, V = torch.linalg.eigh(H) torch.cuda.synchronize() if device == "cuda" else None t1 = time.perf_counter() gpu_ms = (t1 - t0) * 1000 # Time eigh on CPU H_cpu = H.cpu() t0 = time.perf_counter() torch.linalg.eigh(H_cpu) t1 = time.perf_counter() cpu_ms = (t1 - t0) * 1000 eig_min = omega[0].item() eig_max = omega[-1].item() cond = (eig_max / abs(eig_min)) if abs(eig_min) > 1e-30 else float('inf') H_norm = H.norm().item() diag_diff = abs(torch.diag(H).norm().item() - prev_diag_norm) prev_diag_norm = torch.diag(H).norm().item() flag = "" if gpu_ms > 50: flag = " ⚠ SLOW" if gpu_ms > cpu_ms * 5 and cpu_ms > 0.1: flag += f" ⚠ GPU {gpu_ms/cpu_ms:.0f}x SLOWER" print(f"{iteration:4d} {gpu_ms:14.3f} {cpu_ms:14.3f} {cond:14.3e} {H_norm:14.3e} {diag_diff:14.3e}{flag}") # Dump if slow if gpu_ms > 100: dump_dir = "/tmp/eigh_debug_dumps" os.makedirs(dump_dir, exist_ok=True) torch.save(H.cpu(), os.path.join(dump_dir, f"H_prog_size{size}_iter{iteration}.pt")) # Compute step and update (same as bfgs_torch.py) dpos_dir = V @ (forces.t() @ V / torch.abs(omega)).t() step_len = dpos_dir.norm().item() scale = min(1.0, 0.2 / max(step_len, 1e-10)) new_pos = pos + dpos_dir * scale # Simulate force change new_forces = forces * (0.85 + 0.3 * torch.rand(1, device=device, dtype=torch.float64).item()) new_forces += torch.randn_like(forces) * 0.002 # BFGS update dpos = new_pos - pos dforces = new_forces - forces if torch.abs(dpos).max() > 1e-7: dg = H @ dpos a = torch.dot(dforces, dpos) b = torch.dot(dpos, dg) if abs(a.item()) > 1e-15 and abs(b.item()) > 1e-15: H = H - torch.outer(dforces, dforces) / a - torch.outer(dg, dg) / b pos = new_pos forces = new_forces ``` 最新更新：PyTorch 默认选用了有缺陷的 MAGMA 算子库（magma_dsyevd_gpu）。 1. PyTorch 底层的多后端机制： 在 PyTorch 中，torch.linalg.eigh 在 GPU 端有两个候选后端： • cusolver（在海光 DCU 上被宏映射为海光原生 hipSOLVER） • magma（第三方代数库 MAGMA，算子名为 magma_dsyevd_gpu） 2. 故障算子：MAGMA 库的 magma_dsyevd_gpu，在 DCU 上存在 __launch_bounds__ 越界与重根迭代死锁缺陷。 3. 修复手段：通过 torch.backends.cuda.preferred_linalg_library('cusolver') 将后端强制指向海光原生 hipSOLVER，纯 GPU 原生运行，完美解决。