光合开发者社区-论坛

运行时

hipsolverZheevd DCU端远远慢于CPU端

2022-11-29 02:20:26

124

本征值对角化的测试结果有点奇怪，DCU端和CPU端的对角化后本征值是相同的，但是DCU端明显慢了很多，而且矩阵越大越慢。测试程序在附件中，期待大神帮忙。

下图是测试输出结果：
dim: 维度
cpu：cpu端单个矩阵对角化时间
gpu：DCU端单个矩阵对角化时间
ratio：cpu端和DCU端时间比
diff：cpu端和DCU端本征值最大差。

编译方法：（hipfc 是用gcc和gfortran编译的）
hipfc -lhipblas -lhipsolver -lhipsparse -g dcu_math_zheev.f03 -L/public/software/mathlib/lapack/gnu/3.8.0/lib64 -lblas -llapack -o zheev_test

使用环境：

Currently Loaded Modulefiles:

compiler/devtoolset/7.3.1 2) compiler/rocm/dtk-22.04.2 3) mpi/hpcx/2.7.4/gcc-7.3.1

程序运行：

srun --mpi=pmix_v3 zheev_test

提交时间：2022.Nov.29

cwoody

2楼

您好，看到您三个问题都是关于线性代数库中API的，hipsolver这个库目前应该开发的还不是很完善，zheev这类接口就仅仅是有一个实现。目前的建议是您可以尝试使用MAGMA库，https://icl.utk.edu/magma/ 开发者社区有基于DTK-22.10编译好的：https://download.sourcefind.cn:65024/directlink/1/DTK-22.10_hpcapps-20221120/NFS3.2/MAGMA_2.6.0_nfs3.2_DTK22.10_single_11Nov2022.tar.gz

您可以看看这个库中应该有您需要的求解线性方程组，特征值分解，稀疏矩阵求解等的函数。路径下的include目录中有fortran的module文件。

hipsolver这个库后续我们也会逐步去开发，目前zheevd已开发了一段时间。

2022/11/29

Brook

回复 @cwoody

谢谢大佬！

我在计算节点上编译了magma_2.6.0_nfs3.2_DTK22.10_11Nov2022/example、example_f.F90，测试的时候发现，只有少数几个矩阵维度能成功运行：

integer :: n= 100 , nrhs= 100

integer :: n= 200 , nrhs= 200

运行结果：

using MAGMA CPU interface

using MAGMA GPU interface

而大部分其它矩阵维度会报错：

integer :: n= 500 , nrhs= 500

运行结果：

using MAGMA CPU interface

magma_zgesv failed with info= 1

using MAGMA GPU interface

magma_zgesv_gpu failed with info= 1

编译环境：

1) compiler/devtoolset/7.3.1

2) compiler/rocm/dtk-22.10

2022/11/29

tangxd

回复 @cwoody

您好，使用DTK-22.10测试该example_f.F90正常，测试了

integer :: n= 500 , nrhs= 500

integer :: n= 700 , nrhs= 700

nteger :: n= 1000 , nrhs= 1000

结果均正常无报错

编译命令：

gfortran -Dmagma_devptr_t="integer(kind=8)" -o example_f example_f.F90 -I../include -L../lib/ -lmagma，

看编译环境和测试环境是否正常，有时部分节点可能出现异常，可切换其他节点。

2022/11/29

Brook

回复 @cwoody

@tangxd 谢谢您的回复。能麻烦您把详细的编译命令贴一下吗？我用您给的这个命令

“gfortran -Dmagma_devptr_t="integer(kind=8)" -o example_f example_f.F90 -I../include -L../lib/ -lmagma”

编译得到错误信息如下：

[ghfund2_asome@login08 example]$ gfortran -Dmagma_devptr_t="integer(kind=8)" -o example_f example_f.F90 -I../include -L../lib/ -lmagma

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libgalaxyhip.so.5, needed by ../lib//libmagma.so, not found (try using -rpath or -rpath-link)

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libmkl_gf_lp64.so, needed by ../lib//libmagma.so, not found (try using -rpath or -rpath-link)

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libmkl_gnu_thread.so, needed by ../lib//libmagma.so, not found (try using -rpath or -rpath-link)

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libmkl_core.so, needed by ../lib//libmagma.so, not found (try using -rpath or -rpath-link)

../lib//libmagma.so: undefined reference to `zgemm_'

../lib//libmagma.so: undefined reference to `clarft_'

../lib//libmagma.so: undefined reference to `sormqr_'

../lib//libmagma.so: undefined reference to `cgemv_'

../lib//libmagma.so: undefined reference to `sgebak_'

../lib//libmagma.so: undefined reference to `slarft_'

../lib//libmagma.so: undefined reference to `zgetrs_'

../lib//libmagma.so: undefined reference to `claswp_'

../lib//libmagma.so: undefined reference to `zgeqrf_'

../lib//libmagma.so: undefined reference to `zherk_'

另外：

我使用hipfc编译的，能顺利通过编译，但是测试结果就如我前面贴的那样。

[ghfund2_a24@login08 example]$ make fortran

hipfc -Dmagma_devptr_t="integer(kind=8)" -I/public/home/ghfund2_a24/software/magma_2.6.0_nfs3.2_DTK22.10_11Nov2022/include -c -o example_f.o example_f.F90

hipfc -Wall -o example_f example_f.o -L/public/home/ghfund2_a24/software/magma_2.6.0_nfs3.2_DTK22.10_11Nov2022/lib -lmagma -lhipblas -lhipsolver -lhipsparse -L/public/software/mathlib/lapack/gnu/3.8.0/lib64 -lblas -llapack

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libgalaxyhip.so.5, needed by /public/home/ghfund2_a24/software/magma_2.6.0_nfs3.2_DTK22.10_11Nov2022/lib/libmagma.so, not found (try using -rpath or -rpath-link)

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libmkl_gf_lp64.so, needed by /public/home/ghfund2_a24/software/magma_2.6.0_nfs3.2_DTK22.10_11Nov2022/lib/libmagma.so, not found (try using -rpath or -rpath-link)

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libmkl_gnu_thread.so, needed by /public/home/ghfund2_a24/software/magma_2.6.0_nfs3.2_DTK22.10_11Nov2022/lib/libmagma.so, not found (try using -rpath or -rpath-link)

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libmkl_core.so, needed by /public/home/ghfund2_a24/software/magma_2.6.0_nfs3.2_DTK22.10_11Nov2022/lib/libmagma.so, not found (try using -rpath or -rpath-link)

/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: warning: libgfortran.so.3, needed by /public/software/mathlib/lapack/gnu/3.8.0/lib64/libblas.so, may conflict with libgfortran.so.4

2022/11/30

全部7条回复

Brook

楼主

1楼

补充：

同样的程序在Nvidia GPU上的实现测试是正常的，GPU端的速度一直比CPU端更快。详细信息如下：

dim: 7 cpu: 0.00s gpu: 0.00s ratio: 0.00 diff: 0.666E-15

dim: 12 cpu: 0.00s gpu: 0.00s ratio: 0.00 diff: 0.666E-15

dim: 20 cpu: 0.00s gpu: 0.00s ratio: 0.00 diff: 0.666E-15

dim: 33 cpu: 0.00s gpu: 0.00s ratio: 0.50 diff: 0.944E-15

dim: 54 cpu: 0.00s gpu: 0.00s ratio: 1.00 diff: 0.200E-14

dim: 90 cpu: 0.00s gpu: 0.00s ratio: 1.00 diff: 0.211E-14

dim: 148 cpu: 0.05s gpu: 0.00s ratio: 15.33 diff: 0.189E-14

dim: 244 cpu: 0.03s gpu: 0.00s ratio: 6.60 diff: 0.244E-14

dim: 403 cpu: 0.11s gpu: 0.01s ratio: 12.22 diff: 0.311E-14

dim: 665 cpu: 0.36s gpu: 0.02s ratio: 21.41 diff: 0.289E-14

dim: 1096 cpu: 1.33s gpu: 0.03s ratio: 40.21 diff: 0.555E-14

dim: 1808 cpu: 6.23s gpu: 0.08s ratio: 78.86 diff: 0.422E-14

dim: 2980 cpu: 40.06s gpu: 0.25s ratio: 158.35 diff: 0.113E-13

2022/11/29