4
3
DTK开发
dtk24.04/dtk24.04.1 个别项目使用onnxruntime推理报错
2024-06-17 11:20:25
252

dtk24.04和dtk24.04.1都测试过了,会报错
用dtk23.10不会报错,请大佬看一下
又测试了下 使用dtk23.03配套的onnxruntime也不会报错

dtk24.04+onnxruntime_lite-1.14.0+gitd1eb427.abi1.dtk2310不会报错。
但是在某个压测场景下,与das1.0+版本的onnxruntime相比 速度变慢一半了。

MIOpen Error: convolutionocl.cpp:1105: No invoker was registered for convolution forward. Was find executed?
2024-06-17 10:50:55.216560555 [EDefault, rocm_call.cc:119 RocmCall] MIOPEN failure 7: miopenStatusUnknownError ; GPU=0 ; hostname=dev-gpu-04 ; file=/data/jenkins_workspace/workspace/onnxruntime@3/onnxruntime/contrib_ops/rocm/fused_conv.cc ; line=178 ; expr=miopenConvolutionForward(miopen_handle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.fwd_algo, &beta, Base::s_.y_tensor, Base::s_.y_data, workspace.get(), Base::s_.workspace_bytes); 
2024-06-17 10:50:55.216595605 [E, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'Conv_0' Status Message: MIOPEN failure 7: miopenStatusUnknownError ; GPU=0 ; hostname=dev-gpu-04 ; file=/data/jenkins_workspace/workspace/onnxruntime@3/onnxruntime/contrib_ops/rocm/fused_conv.cc ; line=178 ; expr=miopenConvolutionForward(miopen_handle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.fwd_algo, &beta, Base::s_.y_tensor, Base::s_.y_data, workspace.get(), Base::s_.workspace_bytes); 
MIOpen Error: convolutionocl.cpp:1105: No invoker was registered for convolution forward. Was find executed?
2024-06-17 10:50:55.567482155 [EDefault, rocm_call.cc:119 RocmCall] MIOPEN failure 7: miopenStatusUnknownError ; GPU=0 ; hostname=dev-gpu-04 ; file=/data/jenkins_workspace/workspace/onnxruntime@3/onnxruntime/contrib_ops/rocm/fused_conv.cc ; line=178 ; expr=miopenConvolutionForward(miopen_handle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.fwd_algo, &beta, Base::s_.y_tensor, Base::s_.y_data, workspace.get(), Base::s_.workspace_bytes); 
2024-06-17 10:50:55.567516665 [E, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'Conv_0' Status Message: MIOPEN failure 7: miopenStatusUnknownError ; GPU=0 ; hostname=dev-gpu-04 ; file=/data/jenkins_workspace/workspace/onnxruntime@3/onnxruntime/contrib_ops/rocm/fused_conv.cc ; line=178 ; expr=miopenConvolutionForward(miopen_handle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.fwd_algo, &beta, Base::s_.y_tensor, Base::s_.y_data, workspace.get(), Base::s_.workspace_bytes); 
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction
    output = await route_utils.call_process_api(
  File "/usr/local/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1185, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 661, in wrapper
    response = f(*args, **kwargs)
  File "/root/facefusion/facefusion/uis/components/preview.py", line 157, in update_preview_image
    conditional_append_reference_faces()
  File "/root/facefusion/facefusion/core.py", line 250, in conditional_append_reference_faces
    source_face = get_average_face(source_frames)
  File "/root/facefusion/facefusion/face_analyser.py", line 461, in get_average_face
    face = get_one_face(vision_frame, position)
  File "/root/facefusion/facefusion/face_analyser.py", line 445, in get_one_face
    many_faces = get_many_faces(vision_frame)
  File "/root/facefusion/facefusion/face_analyser.py", line 512, in get_many_faces
    faces = create_faces(vision_frame, bounding_box_list, face_landmark_5_list, score_list)
  File "/root/facefusion/facefusion/face_analyser.py", line 348, in create_faces
    face_landmark_68, face_landmark_68_score = detect_face_landmark_68(vision_frame, bounding_box)
  File "/root/facefusion/facefusion/face_analyser.py", line 402, in detect_face_landmark_68
    face_landmark_68, face_heatmap = face_landmarker.run(None,
  File "/usr/local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 217, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'Conv_0' Status Message: MIOPEN failure 7: miopenStatusUnknownError ; GPU=0 ; hostname=dev-gpu-04 ; file=/data/jenkins_workspace/workspace/onnxruntime@3/onnxruntime/contrib_ops/rocm/fused_conv.cc ; line=178 ; expr=miopenConvolutionForward(miopen_handle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.fwd_algo, &beta, Base::s_.y_tensor, Base::s_.y_data, workspace.get(), Base::s_.workspace_bytes); 
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction
    output = await route_utils.call_process_api(
  File "/usr/local/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1185, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 661, in wrapper
    response = f(*args, **kwargs)
  File "/root/facefusion/facefusion/uis/components/face_selector.py", line 144, in update_reference_position_gallery
    gallery_vision_frames = extract_gallery_frames(temp_vision_frame)
  File "/root/facefusion/facefusion/uis/components/face_selector.py", line 152, in extract_gallery_frames
    faces = get_many_faces(temp_vision_frame)
  File "/root/facefusion/facefusion/face_analyser.py", line 512, in get_many_faces
    faces = create_faces(vision_frame, bounding_box_list, face_landmark_5_list, score_list)
  File "/root/facefusion/facefusion/face_analyser.py", line 348, in create_faces
    face_landmark_68, face_landmark_68_score = detect_face_landmark_68(vision_frame, bounding_box)
  File "/root/facefusion/facefusion/face_analyser.py", line 402, in detect_face_landmark_68
    face_landmark_68, face_heatmap = face_landmarker.run(None,
  File "/usr/local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 217, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running FusedConv node. Name:'Conv_0' Status Message: MIOPEN failure 7: miopenStatusUnknownError ; GPU=0 ; hostname=dev-gpu-04 ; file=/data/jenkins_workspace/workspace/onnxruntime@3/onnxruntime/contrib_ops/rocm/fused_conv.cc ; line=178 ; expr=miopenConvolutionForward(miopen_handle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.fwd_algo, &beta, Base::s_.y_tensor, Base::s_.y_data, workspace.get(), Base::s_.workspace_bytes); 
2024-06-17 10:51:03.189331275 [E, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running FusedAddRelu node. Name:'Add_91' Status Message: fused_add_relu.cc:38 ComputeInternal add_relu op left input size donot match right input size.
全部评论(11)
只看楼主
网
网友9g2goD
2楼
麻烦提供具体模型,我们复现一下
2024/6/17
网
网友IiwqCN
回复 @网友9g2goD
facefusion,用rocm跑就会报错
2024/6/17
网
网友9g2goD
回复 @网友9g2goD
具体点呢,复现方法?用到的模型文件?
2024/6/17
网
网友IiwqCN
回复 @网友9g2goD

默认启动,进去默认模型不动,更改加速为rocm。

该项目有以下几种模式,默认模型如下

人脸增强:gfpgan_1.4

背景增强:span_kendata_x4

人脸置换:inswapper_128_fp16

任选单个模式, 点击Start后台均报错

2024/6/17
全部7条回复
网
网友foAVcS
1楼
你好,请问这个问题用的哪个网络模型,有复现的方法么?
2024/6/17
网
网友IiwqCN
回复 @网友foAVcS
facefusion,用rocm跑就会报错
2024/6/17
网
网友IiwqCN
回复 @网友foAVcS
前段时间没测试这个问题,近期重新跑了一下,发现是target为视频的时候才会重现,用图片则不会出现这个问题,不知道老师还有空帮忙看下吗
2024/9/20
没有更多内容了
--
楼主
Level.1
0 帖子
楼主热帖
暂无数据