基础环境:
系统:ky10
DTK: 24.04.2
Torch: 2.1.0 + 1.16.0
Python: 3.10
MMCV: 2.0.1
具体报错: #基础环境#
Traceback (most recent call last):
File "/root/ovd/ovd_mmdet/./tools/train.py", line 121, in <module>
main()
File "/root/ovd/ovd_mmdet/./tools/train.py", line 110, in main
runner = Runner.from_cfg(cfg)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/runner/runner.py", line 462, in from_cfg
runner = cls(
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/runner/runner.py", line 429, in __init__
self.model = self.build_model(model)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/runner/runner.py", line 836, in build_model
model = MODELS.build(model)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 232, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/root/ovd/ovd_mmdet/mmdet/models/detectors/ovd_dino.py", line 69, in __init__
super().__init__(*args, **kwargs)
File "/root/ovd/ovd_mmdet/mmdet/models/detectors/deformable_detr.py", line 69, in __init__
super().__init__(*args, decoder=decoder, bbox_head=bbox_head, **kwargs)
File "/root/ovd/ovd_mmdet/mmdet/models/detectors/base_detr.py", line 77, in __init__
self._init_layers()
File "/root/ovd/ovd_mmdet/mmdet/models/detectors/ovd_dino.py", line 163, in _init_layers
self.language_model = MODELS.build(self.language_model_cfg)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 232, in build_model_from_cfg
return build_from_cfg(cfg, registry, default_args)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
File "/root/ovd/ovd_mmdet/mmdet/models/language_models/bert.py", line 123, in __init__
BertEncoder(
File "/root/ovd/ovd_mmdet/mmdet/models/language_models/bert.py", line 205, in __init__
self.model = HFBertModel.from_pretrained(
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/transformers/modeling_utils.py", line 309, in _wrapper
return func(*args, **kwargs)
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4252, in from_pretrained
device_in_context = get_torch_context_manager_or_global_device()
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/transformers/modeling_utils.py", line 322, in get_torch_context_manager_or_global_device
default_device = torch.get_default_device()
File "/data1/zdhs/miniconda3/envs/ovd_mmdet_t210_py310/lib/python3.10/site-packages/torch/__init__.py", line 1841, in __getattr__
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
AttributeError: module 'torch' has no attribute 'get_default_device'
相关配置具体如下:
uname -a
Linux localhost.localdomain 4.19.90-89.23.v2401.ky10.x86_64 #1 SMP Fri Apr 25 12:25:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
dtk:
lrwxrwxrwx 1 root root 16 May 27 16:39 dtk -> /opt/dtk-24.04.2
drwxr-xr-x 37 root root 4096 Sep 20 2024 dtk-24.04.2
pytorch:
torch-2.1.0+das.opt1.dtk24042-cp310-cp310-manylinux_2_28_x86_64.whl
torchvision-0.16.0+das.opt1.dtk24042-cp310-cp310-manylinux_2_28_x86_64.whl
mmcv:
mmcv-2.0.1+das.opt1.dtk24042-cp310-cp310-manylinux_2_28_x86_64.whl
rocm架构:
rocminfo | grep gfx
Name: gfx928
Name: amdgcn-amd-amdhsa--gfx928:sramecc+:xnack-
pytorch信息:
>>> import torch
>>> print(torch.__version__)
2.1.0
>>> print(torch.version.hip)
5.7.24311
>>> print(torch.cuda.is_available())
True
>>> print(torch.cuda.get_device_name(0))
K100_AI
>>> print(torch.cuda.get_arch_list())
['gfx906', 'gfx926', 'gfx928']
>>> print(torch.cuda.get_device_capability())
(9, 2)

暂无数据
测试demo报错:
import torch
print("Device:".torch.cuda.get device name(0))
export AMD SERIALIZE KERNEL=3
export HIP LAUNCH BLOCKING=1
python - <<EOF
b= torch.zeros((4,4),dtype=torch.float32, device="cuda")
torch.cuda.synchronize()
EOF
Device:K100 AITraceback(most recent call last):
File"<stdin>",line 6,in <module>
RuntimeError: HIP error:invalid device function
Compile with`TORCH USE HIP DSA` to enable device-side assertions.