3D object detection Transformation issue

[imdsafi09]

Hi,
I am trying to run the stack with only D435 camera. As I dont have the base_link, I change the target_frame into camera_link. However, I am still getting the following error even though the camera_link frame exists.

"[detect_3d_node-3] [ERROR] [1709019322.086835673] [yolo.detect_3d_node]: Could not transform: "camera_link" passed to lookupTransform argument target_frame does not exist.
"

[mgonzs13]

Hey @imdsafi09, I couldn't reproduce your error. I may be a problem with your tf. Which ROS distro are you using?

[imdsafi09]

Hi @mgonzs13
Thank you for your response. The reason for this was that my camera had not previously published the/tf_static frame.
However, I have another problem, I am using Ouster-lidar signal image topic "/ouster/signal_image" to subscribe it into as a camera. The topic is working with similar yolov8 based ros package ultralytics_ros but when I am trying to subscribe it here it has the following error:

[yolov8_node-1] File "/home/imad/.local/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 125, in preprocess
[yolov8_node-1] im = im[..., ::-1].transpose((0, 3, 1, 2)) # BGR to RGB, BHWC to BCHW, (n, 3, h, w)

Could you please let me know what could be the possible solution.

Ubuntu:22.04
ROS-Humble

Best regards,

[mgonzs13]

Hi @imdsafi09, it seems to be a problem with the encoding of the image. Can you give me more info about the error, the error message? If you record a rosbag with your data, I can test it better.

[imdsafi09]

Thank you for your response, @mgonzs13. The issue was caused by a mismatch in the input image channel, which was resolved when I converted mono16 to rgb8.

[mgonzs13]

Hey @imdsafi09, how is this going?

[imdsafi09]

Hi @mgonzs13 , sorry for the late response. I resolved the previous issue, and it's working with the Ouster-lidar signal image topic. However, 3D bounding visualization is the same for all the classes. Would it be possible to have a different visualization based on the class type, similar to a 2D one?

[mgonzs13]

Hi @imdsafi09, do you mean changing the color of the 3D bounding box? I think that can be easily included.

[mgonzs13]

Check out the new version that includes the 3D bounding boxes with the same color as the 2D boxes.

[imdsafi09]

Hi @mgonzs13,
Thank you for your excellent work. I implemented it successfully, and the 2D and 3D versions work fine. In my case, I am integrating 3D object detection with 3D mapping to save the object's location within the map. I tried several approaches. However, they cause redundancy and make the overall process computationally more expensive. Could you please give any suggestions?
Regards,

[mgonzs13]

I have never used semantic localization. Which works have you checked? From a fast search, I have found these 1, 2, 3. Do you have a public repo to take a look at your work?

[imdsafi09]

I had no known issues with mapping or localization. I am looking for options on how to keep/save the detected 3D object position on the map. I am attaching the script I am following for you to look at. It would be something like this.
1
2

Like the image below, once the bounding box is detected, it should not disappear, but its location should be added to the map.

[mgonzs13]

So, If I understand the code in 1, markers from door detections are used to modify the occupancy grid map of the localization. As a result, you will have the detections included in the map. Or do you also want to visualize the detection markers? You can add a publisher to publish the markers included in the map so that the don't desappear.

[imdsafi09]

Yes, you understood it correctly. They used markers from objects of interest to modify the occupancy grid map. Technically, the markers visualization does not add value to the overall system. Only the localization of the object within the map is essential for my case. However, for visual understanding, I am trying to add detected markers as well. Thank you for all your insightful comments. I will look into it as you suggested.

[HHX0607]

Hi, I am trying to run the stack with only D435 camera. As I dont have the base_link, I change the target_frame into camera_link. However, I am still getting the following error even though the camera_link frame exists.

"[detect_3d_node-3] [ERROR] [1709019322.086835673] [yolo.detect_3d_node]: Could not transform: "camera_link" passed to lookupTransform argument target_frame does not exist. "

@imdsafi09 Hello！I am facing the same problem.Could you tell me how to create a camera_link and what is necessary to set. I just use D435i and i want to test 3d detection on yolov8_ros. Thanks!

[imdsafi09]

Hi @HHX0607,

In my case, I changed the default value from default_value="base_link", to "camera_link" and it worked fine. Hopefully, it will work for you as well. If you have already made changes in this file and are still facing the problem, then try checking the 'tf_tree'.

[lymcs]

您好，感谢您的回复。这样做的原因是我的相机之前没有发布 /tf_static 帧。但是，我还有另一个问题，我正在使用 Ouster-lidar 信号图像主题 “/ouster/signal_image” 将其订阅为相机。该主题正在使用类似的基于 yolov8 的 ros 包ultralytics_ros但是当我尝试在此处订阅它时，它出现以下错误：

[yolov8_node-1]文件 “/home/imad/.local/lib/python3.10/site-packages/ultralytics/engine/predictor.py”，第 125 行，预处理中 [yolov8_node-1] im = im[...， ：：-1].transpose（（0， 3， 1， 2）） # BGR 到 RGB，BHWC 到 BCHW，（n， 3， h， w）

您能否请告诉我可能的解决方案。

Ubuntu：22.04 ROS-简陋

此致敬意

excuse me, my camera had previously published the/tf_static frame.I also dont have the base_link, and I change the target_frame into camera_link.however,"[detect_3d_node-3] [ERROR] [1709019322.086835673] [yolo.detect_3d_node]: Could not transform: "camera_link" passed to lookupTransform argument target_frame does not exist.
"

/camera/color/camera_info
/camera/color/image_raw
/camera/color/metadata
/camera/depth/camera_info
/camera/depth/image_rect_raw
/camera/depth/metadata
/camera/extrinsics/depth_to_color
/camera/imu
/clicked_point
/goal_pose
/initialpose
/parameter_events
/rosout
/tf
/tf_static
/yolo/dbg_image

I wish your response.

[lymcs]

您好，感谢您的回复。这样做的原因是我的相机之前没有发布 /tf_static 帧。但是，我还有另一个问题，我正在使用 Ouster-lidar 信号图像主题 “/ouster/signal_image” 将其订阅为相机。该主题正在使用类似的基于 yolov8 的 ros 包ultralytics_ros但是当我尝试在此处订阅它时，它出现以下错误：
[yolov8_node-1]文件 “/home/imad/.local/lib/python3.10/site-packages/ultralytics/engine/predictor.py”，第 125 行，预处理中 [yolov8_node-1] im = im[...， ：：-1].transpose（（0， 3， 1， 2）） # BGR 到 RGB，BHWC 到 BCHW，（n， 3， h， w）
您能否请告诉我可能的解决方案。
Ubuntu：22.04 ROS-简陋
此致敬意

对不起，我的相机之前已经发布了 /tf_static 帧。我也没有base_link，我将target_frame更改为 camera_link.但是，“[detect_3d_node-3] [错误] [1709019322.086835673] [yolo.detect_3d_node]：无法转换：”传递给 lookupTransform 参数的“camera_link”target_frame不存在。 “ /camera/color/camera_info /camera/color/image_raw /camera/color/metadata /camera/depth/camera_info /camera/depth/image_rect_raw /camera/depth/metadata /camera/extrinsics/depth_to_color /camera/imu /clicked_point /goal_pose /initialpose /parameter_events /rosout /tf /tf_static /yolo/dbg_image

我希望得到您的回复。

[imdsafi09]

Hi @lymcs,
If you change the base_link into camer_link correctly and you are running a realsense depth camera (d435, d435i, etc.), then try tf to check the tf tree or check in rviz if you have a camera_link ? as shown in the screenshot. If there is no other error, then it should work.

[fadai333]

您好，感谢您的回复。这样做的原因是我的相机之前没有发布 /tf_static 帧。但是，我还有另一个问题，我正在使用 Ouster-lidar 信号图像主题 “/ouster/signal_image” 将其订阅为相机。该主题正在使用类似的基于 yolov8 的 ros 包ultralytics_ros但是当我尝试在此处订阅它时，它出现以下错误：
[yolov8_node-1]文件 “/home/imad/.local/lib/python3.10/site-packages/ultralytics/engine/predictor.py”，第 125 行，预处理中 [yolov8_node-1] im = im[...， ：：-1].transpose（（0， 3， 1， 2）） # BGR 到 RGB，BHWC 到 BCHW，（n， 3， h， w）
您能否请告诉我可能的解决方案。
Ubuntu：22.04 ROS-简陋
此致敬意

对不起，我的相机之前已经发布了 /tf_static 帧。我也没有base_link，我将target_frame更改为 camera_link.但是，“[detect_3d_node-3] [错误] [1709019322.086835673] [yolo.detect_3d_node]：无法转换：”传递给 lookupTransform 参数的“camera_link”target_frame不存在。 “ /camera/color/camera_info /camera/color/image_raw /camera/color/metadata /camera/depth/camera_info /camera/depth/image_rect_raw /camera/depth/metadata /camera/extrinsics/depth_to_color /camera/imu /clicked_point /goal_pose /initialpose /parameter_events /rosout /tf /tf_static /yolo/dbg_image

我希望得到您的回复。

Hello, may I ask if this problem has been solved? This problem also occurred when I was running the program, but I found that the /tf_static topic has no result to output, I don't know whether it is caused by this error, I hope to get your reply. My version of ros2 is foxy

[weilinhan]

hi,I have run the 3d detect node with no errors and I can already see the 2d detect box on rviz2, what else do I need to add to see the 3d detect box?

[mgonzs13]

You may visualize 3D data using markers in rviz2. Are you getting msgs on the 3D topic?