ORT 1.22.2 测试问题 基础镜像 harbor.sourcefind.cn:5443/dcu/admin/base/custom:pytorch2.6.0-ubuntu22.04-dtk26.04-py3.10-20260701 Migraph版本: 5.2.0+das.opt1.dtk2604 ORT：pip install onnxruntime==1.22.2+das.opt1.dtk2604.2607061906.gc3f2dd -i https://pypi.sourcefind.cn/release/dtk/ docker-compose yaml： ```yaml version: '3.3' services: docparser-test: image: harbor.sourcefind.cn:5443/dcu/admin/base/custom:pytorch2.6.0-ubuntu22.04-dtk26.04-py3.10-20260701 container_name: dtk-2604 volumes: - /opt/hyhal:/opt/hyhal:ro - /home/data:/data devices: - /dev/kfd - /dev/dri - /dev/mkfd environment: - HIP_VISIBLE_DEVICES=0 shm_size: '128gb' tty: true ``` 测试代码： ``` import onnxruntime as ort import numpy as np import time import random model_path = './slanet-plus-new.onnx' providers = [ 'MIGraphXExecutionProvider', #'ROCMExecutionProvider', #'CPUExecutionProvider', ] #创建sess_options sess_options = ort.SessionOptions() sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC sess_options.enable_profiling = False session = ort.InferenceSession(model_path,sess_options,providers=providers,provider_options=[{'device_id': '0'}]) for i in range(10): start = time.time() bs = random.randint(1, 10) image = np.random.rand(bs,3,224,224).astype(np.float32) input_name=session.get_inputs()[0].name result = session.run(None, input_feed={input_name:image}) end = time.time() print(end - start) ``` （1）容器挂单卡时：MIGraphXExecutionProvider推理 * 问题：yml容器HIP_VISIBLE_DEVICES=0，device_id 设置为0时，异常 ``` 2026-07-17 03:17:38.770327739 [W:onnxruntime:, session_state.cc:1280 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf. 2026-07-17 03:17:38.770361171 [W:onnxruntime:, session_state.cc:1282 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments. MIGraphX Error: /home/migraphx/src/onnx/onnx_parser.cpp:500: check_sorted: subgraph "PaddlePaddle Graph in PIR mode" has parameter name "p2o.pd_op.logical_and.0.0" existing in parent graph! 2026-07-17 03:17:38.963291897 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5_5' Status Message: Failed to call function Traceback (most recent call last): File "/workspace/test.py", line 25, in result = session.run(None, input_feed={input_name:image}) File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 273, in run return self._sess.run(output_names, input_feed, run_options) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5_5' Status Message: Failed to call function ``` * device_id 设置为1，2，3时，回到了CPUExecutionProvider推理，有异常： ``` *************** EP Error *************** EP Error /home/onnxruntime/onnxruntime/core/providers/migraphx/migraphx_call.cc:64 std::conditional_t onnxruntime::RocmCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = hipError_t; bool THRW = true; std::conditional_t = void] /home/onnxruntime/onnxruntime/core/providers/migraphx/migraphx_call.cc:57 std::conditional_t onnxruntime::RocmCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = hipError_t; bool THRW = true; std::conditional_t = void] HIP failure 101: invalid device ordinal ; GPU=0 ; hostname=1241809227c4 ; file=/home/onnxruntime/onnxruntime/core/providers/migraphx/migraphx_execution_provider.cc ; line=111 ; expr=hipSetDevice(info_.device_id); when using ['MIGraphXExecutionProvider'] Falling back to ['CPUExecutionProvider'] and retrying. **************************************** 0.7180502414703369 0.4673190116882324 0.4341459274291992 0.5863337516784668 0.4944169521331787 0.36711907386779785 0.8166859149932861 0.6825690269470215 0.6369669437408447 0.7623066902160645 ``` (2)容器挂多卡时： MIGraphXExecutionProvider 推理 * 问题：yml容器HIP_VISIBLE_DEVICES=0,1,2,3，device_id 设置为0,1,2,3时，异常 ``` 2026-07-17 03:56:14.498258775 [W:onnxruntime:, session_state.cc:1280 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf. 2026-07-17 03:56:14.498294387 [W:onnxruntime:, session_state.cc:1282 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments. MIGraphX Error: /home/migraphx/src/onnx/onnx_parser.cpp:500: check_sorted: subgraph "PaddlePaddle Graph in PIR mode" has parameter name "p2o.pd_op.logical_and.0.0" existing in parent graph! 2026-07-17 03:56:14.679710997 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5_5' Status Message: Failed to call function Traceback (most recent call last): File "/workspace/test-ort.py", line 27, in result = session.run(None, input_feed={input_name:image}) File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 273, in run return self._sess.run(output_names, input_feed, run_options) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_PaddlePaddle Graph in PIR mode_17256989997197446820_5_5' Status Message: Failed to call function ``` * device_id 设置为>=4: ``` ['MIGraphXExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider'] *************** EP Error *************** EP Error /home/onnxruntime/onnxruntime/core/providers/migraphx/migraphx_call.cc:64 std::conditional_t onnxruntime::RocmCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = hipError_t; bool THRW = true; std::conditional_t = void] /home/onnxruntime/onnxruntime/core/providers/migraphx/migraphx_call.cc:57 std::conditional_t onnxruntime::RocmCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = hipError_t; bool THRW = true; std::conditional_t = void] HIP failure 101: invalid device ordinal ; GPU=0 ; hostname=af8d0a73d643 ; file=/home/onnxruntime/onnxruntime/core/providers/migraphx/migraphx_execution_provider.cc ; line=111 ; expr=hipSetDevice(info_.device_id); when using ['MIGraphXExecutionProvider'] Falling back to ['CPUExecutionProvider'] and retrying. **************************************** 0.3564126491546631 0.7024695873260498 0.2315690517425537 0.5526959896087646 0.8294804096221924 0.3104422092437744 0.5075244903564453 0.5019607543945312 1.5303349494934082 0.3946700096130371 ``` (2)容器挂多卡时： ROCMExecutionProvider 推理 能正常根据HIP_VISIBLE_DEVICES 识别到挂载卡的编号 ``` 2026-07-17 04:02:33.723362109 [W:onnxruntime:, session_state.cc:1280 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf. 2026-07-17 04:02:33.723409532 [W:onnxruntime:, session_state.cc:1282 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments. 2.135566234588623 1.0270648002624512 1.069913387298584 1.0375189781188965 1.0350074768066406 0.9836075305938721 0.08512616157531738 0.9932441711425781 1.0232017040252686 1.0282869338989258 ```