1
1
1
DTK开发
在ECS上使用Z100L的显卡用不了dcu,帮看一下原因
2024-07-30 17:09:52
205

报错:
RuntimeError: HIP error: the operation cannot be performed in the present state

Compile with TORCH_USE_HIP_DSA  to enable device-side assertions

attachmentId-image.png

hygon module is loaded

=====================

HSA System Attributes

=====================

Runtime Version: 1.1

System Timestamp Freq.: 1000.000000MHz

Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)

Machine Model: LARGE

System Endianness: LITTLE

==========

HSA Agents

==========


Agent 1


Name: Hygon C86 7385 32-core Processor

Uuid: CPU-XX

Marketing Name: Hygon C86 7385 32-core Processor

Vendor Name: AuthenticAMD

Feature: None specified

Profile: FULL_PROFILE

Float Round Mode: NEAR

Max Queue Number: 0(0x0)

Queue Min Size: 0(0x0)

Queue Max Size: 0(0x0)

Queue Type: MULTI

Node: 0

Device Type: CPU

Cache Info:

L1:                      65536(0x10000) KB                

Chip ID: 0(0x0)

Cacheline Size: 64(0x40)

Max Clock Freq. (MHz): 0

BDFID: 0

Internal Node ID: 0

Compute Unit: 14

SIMDs per CU: 0

Shader Engines: 0

Shader Arrs. per Eng.: 0

WatchPts on Addr. Ranges:1

Features: None

Pool Info:

Pool 1                 

  Segment:                 GLOBAL; FLAGS: FINE GRAINED      

  Size:                    59468008(0x38b68e8) KB           

  Allocatable:             TRUE                             

  Alloc Granule:           4KB                              

  Alloc Alignment:         4KB                              

  Accessible by all:       TRUE                             

Pool 2                 

  Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED

  Size:                    59468008(0x38b68e8) KB           

  Allocatable:             TRUE                             

  Alloc Granule:           4KB                              

  Alloc Alignment:         4KB                              

  Accessible by all:       TRUE                             

Pool 3                 

  Segment:                 GLOBAL; FLAGS: COARSE GRAINED    

  Size:                    59468008(0x38b68e8) KB           

  Allocatable:             TRUE                             

  Alloc Granule:           4KB                              

  Alloc Alignment:         4KB                              

  Accessible by all:       TRUE                             

ISA Info:


Agent 2


Name: ZIFANG

Uuid: GPU-6f09645d36462181

Marketing Name: Z100L

Vendor Name: HYGON

Feature: KERNEL_DISPATCH

Profile: BASE_PROFILE

Float Round Mode: NEAR

Max Queue Number: 128(0x80)

Queue Min Size: 64(0x40)

Queue Max Size: 131072(0x20000)

Queue Type: MULTI

Node: 1

Device Type: DCU

Cache Info:

L1:                      16(0x10) KB                      

L2:                      8192(0x2000) KB                  

Chip ID: 21943(0x55b7)

Cacheline Size: 64(0x40)

Max Clock Freq. (MHz): 1600

BDFID: 48

Internal Node ID: 1

Compute Unit: 60

SIMDs per CU: 4

Shader Engines: 4

Shader Arrs. per Eng.: 1

WatchPts on Addr. Ranges:4

Features: KERNEL_DISPATCH

Fast F16 Operation: TRUE

Wavefront Size: 64(0x40)

Workgroup Max Size: 1024(0x400)

Workgroup Max Size per Dimension:

x                        1024(0x400)                      

y                        1024(0x400)                      

z                        1024(0x400)                      

Max Waves Per CU: 40(0x28)

Max Work-item Per CU: 2560(0xa00)

Grid Max Size: 4294967295(0xffffffff)

Grid Max Size per Dimension:

x                        4294967295(0xffffffff)           

y                        4294967295(0xffffffff)           

z                        4294967295(0xffffffff)           

Max fbarriers/Workgrp: 32

Pool Info:

Pool 1                 

  Segment:                 GLOBAL; FLAGS: COARSE GRAINED    

  Size:                    33538048(0x1ffc000) KB           

  Allocatable:             TRUE                             

  Alloc Granule:           4KB                              

  Alloc Alignment:         4KB                              

  Accessible by all:       FALSE                            

Pool 2                 

  Segment:                 GLOBAL; FLAGS:                   

  Size:                    33538048(0x1ffc000) KB           

  Allocatable:             TRUE                             

  Alloc Granule:           4KB                              

  Alloc Alignment:         4KB                              

  Accessible by all:       FALSE                            

Pool 3                 

  Segment:                 GLOBAL; FLAGS: FINE GRAINED      

  Size:                    33538048(0x1ffc000) KB           

  Allocatable:             TRUE                             

  Alloc Granule:           4KB                              

  Alloc Alignment:         4KB                              

  Accessible by all:       FALSE                            

Pool 4                 

  Segment:                 GROUP                            

  Size:                    64(0x40) KB                      

  Allocatable:             FALSE                            

  Alloc Granule:           0KB                              

  Alloc Alignment:         0KB                              

  Accessible by all:       FALSE                            

ISA Info:

ISA 1                  

  Name:                    amdgcn-amd-amdhsa--gfx906:sramecc+:xnack-

  Machine Models:          HSA_MACHINE_MODEL_LARGE          

  Profiles:                HSA_PROFILE_BASE                 

  Default Rounding Mode:   NEAR                             

  Default Rounding Mode:   NEAR                             

  Fast f16:                TRUE                             

  Workgroup Max Size:      1024(0x400)                      

  Workgroup Max Size per Dimension:

    x                        1024(0x400)                      

    y                        1024(0x400)                      

    z                        1024(0x400)                      

  Grid Max Size:           4294967295(0xffffffff)           

  Grid Max Size per Dimension:

    x                        4294967295(0xffffffff)           

    y                        4294967295(0xffffffff)           

    z                        4294967295(0xffffffff)           

  FBarrier Max Size:       32                               

*** Done ***

全部评论(5)
只看楼主
P
panhuiwen
2楼

请尝试执行下:

1. lspci -s `hy-smi --showbus | grep DCU | cut -d ' ' -f 4 | head -1` -vvv | grep Atomic

2. dmesg | grep -i atomic
2024/8/6
D
dcu_fae_zhangxp
1楼

你好,请输入

export HIP_LOG_LEVEL=4

export HIP_MODULE_MASK=0x7fffffff

export HIP_ENABLE_LIST=hip

开启调试日志,并重新复现一下,看看HIP打印
2024/8/2
网
网友AQeLCZ
回复 @dcu_fae_zhangxp

[root@iZ0sv050tttgyqqds0ayzxZ model]# cat atest.py

import torch; print(torch.ones(2).cuda())

[root@iZ0sv050tttgyqqds0ayzxZ model]# env | grep HIP

HIP_ENABLE_LIST=hip

HIP_PATH=/opt/dtk/hip

HIP_MODULE_MASK=0x7fffffff

HIP_LOG_LEVEL=4

[root@iZ0sv050tttgyqqds0ayzxZ model]# python3

Python 3.10.12 (main, Mar 21 2024, 17:56:31) [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import torch

>>> print(torch.ones(2).cuda())

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:539: hipErrorNoBinaryForGpu: Unable to find code object for all current devices!

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:540: Devices:

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:543: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- - [Not Found]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:547: Bundled Code Objects:

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:563: host-x86_64-unknown-linux - [Unsupported]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:561: hipv4-amdgcn-amd-amdhsa--gfx928 - [code object targetID is amdgcn-amd-amdhsa--gfx928]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:567: hipErrorNoBinaryForGpu: Unable to find code object for all current devices! - 209

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_fatbin.cpp:267: hipErrorNoBinaryForGpu: Couldn't find binary for ptr: 0x8c0b5000

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_platform.cpp:833: init: Returned hipErrorNoBinaryForGpu : continue parsing remaining modules

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:539: hipErrorNoBinaryForGpu: Unable to find code object for all current devices!

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:540: Devices:

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:543: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- - [Not Found]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:547: Bundled Code Objects:

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:563: host-x86_64-unknown-linux - [Unsupported]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:561: hipv4-amdgcn-amd-amdhsa--gfx928 - [code object targetID is amdgcn-amd-amdhsa--gfx928]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:567: hipErrorNoBinaryForGpu: Unable to find code object for all current devices! - 209

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_fatbin.cpp:267: hipErrorNoBinaryForGpu: Couldn't find binary for ptr: 0x8c0af000

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_platform.cpp:833: init: Returned hipErrorNoBinaryForGpu : continue parsing remaining modules

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:539: hipErrorNoBinaryForGpu: Unable to find code object for all current devices!

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:540: Devices:

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:543: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- - [Not Found]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:547: Bundled Code Objects:

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:563: host-x86_64-unknown-linux - [Unsupported]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:561: hipv4-amdgcn-amd-amdhsa--gfx928 - [code object targetID is amdgcn-amd-amdhsa--gfx928]

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_code_object.cpp:567: hipErrorNoBinaryForGpu: Unable to find code object for all current devices! - 209

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_fatbin.cpp:267: hipErrorNoBinaryForGpu: Couldn't find binary for ptr: 0x8c0a1000

[LOG_ERROR] /data/jenkins_workspace/workspace/galaxy_release/galaxy/src/hipamd/src/hip_platform.cpp:833: init: Returned hipErrorNoBinaryForGpu : continue parsing remaining modules

[LOG_ERROR] /data/jenkins_work
2024/8/2
D
dcu_fae_zhangxp
回复 @dcu_fae_zhangxp
你好,粘贴最后十几行左右就可以
2024/8/3
网
网友AQeLCZ
回复 @dcu_fae_zhangxp
问题已经拉群解决了,最后定位的原因是automicop没开,做了两个动作,1、把bios的automicop打开,2装了旧版本驱动再装新版本。当前可能有两个原因1、最新版本驱动的问题 (确定) 2、物理机bios没打开automicop(不确定)
2024/8/6
没有更多内容了
--
楼主
Level.4
0 帖子
楼主热帖
暂无数据