Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hesai XT16 LiDAR #137

Closed
deniwaki opened this issue Apr 10, 2024 · 18 comments · Fixed by #241
Closed

Hesai XT16 LiDAR #137

deniwaki opened this issue Apr 10, 2024 · 18 comments · Fixed by #241

Comments

@deniwaki
Copy link

I want to use the XT16 in Nebula, Is there a good way to do it? 
Or any update plan for use XT16 near future?

@mojomex
Copy link
Collaborator

mojomex commented Apr 11, 2024

Thank you for your interest in Nebula! Sadly, we are currently not supporting XT16 and do not currently have a model to test with.
However, seeing as the datasheets of XT32 and XT16 are (as far as I can tell) identical except for the 32 vs. 16 channels, it should be fairly quick to work off of the existing XT32 code to add support.
This would involve:

  • Adding XT16 as a recognized model name in a few places (basically search and copy XT32)
  • Adding an XT16 sensor definition (copy XT32)
  • Perform testing (connect to the sensor, show pointcloud output)
  • If everything looks good, record /pandar_packets while XT16 is running for us to confirm it is working.
  • Add a unit test case based on the recorded packets

If you specifically want to use Nebula with XT16 and are willing to create a PR along those lines, we would be very happy to work with you on this.

@jemmmel
Copy link

jemmmel commented Jul 8, 2024

We are also using XT-16, I may look into this

@drwnz
Copy link
Collaborator

drwnz commented Nov 25, 2024

@deniwaki @jemmmel How did you get on with testing XT-16? If you managed to implement in Nebula, we'd be happy to review a PR. Otherwise, I'll close this issue as we have do not have access to a device to test at TIER IV.

@jemmmel
Copy link

jemmmel commented Nov 25, 2024

Sorry, I have not had time to look at this lately. If I do, I will request to re-open it

@drwnz drwnz closed this as completed Nov 25, 2024
@drwnz drwnz closed this as not planned Won't fix, can't repro, duplicate, stale Nov 25, 2024
@imeshsps
Copy link

imeshsps commented Nov 28, 2024

@mojomex I was able to do some modifications on XT32 implementation and get following outputs from XT16.

$ ros2 launch nebula_ros hesai_launch_all_hw.xml sensor_model:=PandarXT16

[INFO] [launch]: All log files can be found below /home/user/.ros/log/2024-11-28-10-38-10-647809-user-agx-orin-211950
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [hesai_ros_wrapper_node-1]: process started with pid [211951]
[hesai_ros_wrapper_node-1] [INFO] [1732770490.919271248] [hesai_ros_wrapper_node]: Sensor Configuration: Hesai Sensor Configuration:
[hesai_ros_wrapper_node-1] Sensor Model: PandarXT16
[hesai_ros_wrapper_node-1] Frame ID: hesai
[hesai_ros_wrapper_node-1] Host IP: 192.168.123.50
[hesai_ros_wrapper_node-1] Sensor IP: 192.168.123.20
[hesai_ros_wrapper_node-1] Data Port: 2368
[hesai_ros_wrapper_node-1] Return Mode: LastStrongest
[hesai_ros_wrapper_node-1] Frequency: 65535
[hesai_ros_wrapper_node-1] MTU: 1500
[hesai_ros_wrapper_node-1] Use Sensor Time: 0
[hesai_ros_wrapper_node-1] Multicast: disabled
[hesai_ros_wrapper_node-1] GNSS Port: 10110
[hesai_ros_wrapper_node-1] Rotation Speed: 600
[hesai_ros_wrapper_node-1] Sync Angle: 0
[hesai_ros_wrapper_node-1] Cut Angle: 0
[hesai_ros_wrapper_node-1] FoV Start: 0
[hesai_ros_wrapper_node-1] FoV End: 360
[hesai_ros_wrapper_node-1] Dual Return Distance Threshold: 0.1
[hesai_ros_wrapper_node-1] Calibration Path: /home/user/rosws/install/nebula_decoders/share/nebula_decoders/calibration/hesai/PandarXT16.csv
[hesai_ros_wrapper_node-1] PTP Profile: IEEE_1588v2
[hesai_ros_wrapper_node-1] PTP Domain: 0
[hesai_ros_wrapper_node-1] PTP Transport Type: UDP/IP
[hesai_ros_wrapper_node-1] PTP Switch Type: TSN
[hesai_ros_wrapper_node-1] [INFO] [1732770490.919631861] [hesai_ros_wrapper_node]: UDP-only mode is enabled. Settings checks, synchronization, and diagnostics publishing are disabled.
[hesai_ros_wrapper_node-1] [INFO] [1732770490.920580897] [IoContext::IoContext]: Thread(s) Created: 1
[hesai_ros_wrapper_node-1] [INFO] [1732770490.922213526] [hesai_ros_wrapper_node.HesaiDecoder]: Using calibration data from /home/user/rosws/install/nebula_decoders/share/nebula_decoders/calibration/hesai/PandarXT16.csv
[hesai_ros_wrapper_node-1] [INFO] [1732770490.922249846] [hesai_ros_wrapper_node.HesaiDecoder]: Starting Decoder
[hesai_ros_wrapper_node-1] [INFO] [1732770490.970740425] [hesai_ros_wrapper_node.HesaiDecoder]: . Wrapper=OK
[hesai_ros_wrapper_node-1] [INFO] [1732770490.971526867] [hesai_ros_wrapper_node.HwInterface]: Starting UDP receiver
[hesai_ros_wrapper_node-1] [WARN] [1732770492.015865025] [hesai_ros_wrapper_node]: New subscription discovered on topic '/pandar_points', requesting incompatible QoS. No messages will be sent to it. Last incompatible policy: RELIABILITY_QOS_POLICY

The following output is from the /pandar_points topic.

$ros2 topic echo /pandar_points

---
header:
  stamp:
    sec: 1564025028
    nanosec: 99723008
  frame_id: hesai
height: 1
width: 16795
fields:
- name: x
  offset: 0
  datatype: 7
  count: 1
- name: y
  offset: 4
  datatype: 7
  count: 1
- name: z
  offset: 8
  datatype: 7
  count: 1
- name: intensity
  offset: 12
  datatype: 2
  count: 1
- name: return_type
  offset: 13
  datatype: 2
  count: 1
- name: channel
  offset: 14
  datatype: 4
  count: 1
- name: azimuth
  offset: 16
  datatype: 7
  count: 1
- name: elevation
  offset: 20
  datatype: 7
  count: 1
- name: distance
  offset: 24
  datatype: 7
  count: 1
- name: time_stamp
  offset: 28
  datatype: 6
  count: 1
is_bigendian: false
point_step: 32
row_step: 537440
data:
- 179
- 7
- 154
- '...'
is_dense: true
---

I'm trying to visualize this in rviz but initially it was saying no actual frame map. I tried publishing static tf from map to hesai but still can't see any data on rviz. Can you please help with this?

@mojomex mojomex reopened this Nov 28, 2024
@mojomex
Copy link
Collaborator

mojomex commented Nov 28, 2024

Hi @imeshsps, it's super nice to hear you're working on XT16 support!

The ros2 topic echo log looks good, so changing the Fixed Frame field in RViz from map to hesai should be all that's needed!
Nebula's default frame ID for Hesai sensors is hesai (see line 8 of your log above). You can change it in the sensor's .param.yaml file 🙌

image

@jemmmel
Copy link

jemmmel commented Nov 28, 2024

Hey all, I have also had some success with the XT16 and should have something ready next week. Looks like good progress already though

@jemmmel
Copy link

jemmmel commented Nov 28, 2024

image
Currently getting one warning related to payload size

[INFO] [launch]: All log files can be found below /home/nagoya/.ros/log/2024-11-28-16-42-27-776466-nagoya-ASUS-TUF-Gaming-F15-FX507VV4-FX507VV4-14598
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [hesai_ros_wrapper_node-1]: process started with pid [14599]
[hesai_ros_wrapper_node-1] [INFO 1732779747.839804660] [hesai_ros_wrapper_node]: Sensor Configuration: Hesai Sensor Configuration:
[hesai_ros_wrapper_node-1] Sensor Model: PandarXT16
[hesai_ros_wrapper_node-1] Frame ID: hesai
[hesai_ros_wrapper_node-1] Host IP: 255.255.255.255
[hesai_ros_wrapper_node-1] Sensor IP: 192.168.1.201
[hesai_ros_wrapper_node-1] Data Port: 2368
[hesai_ros_wrapper_node-1] Return Mode: LastStrongest
[hesai_ros_wrapper_node-1] Frequency: 0
[hesai_ros_wrapper_node-1] MTU: 1500
[hesai_ros_wrapper_node-1] Use Sensor Time: 0
[hesai_ros_wrapper_node-1] Multicast: disabled
[hesai_ros_wrapper_node-1] GNSS Port: 10110
[hesai_ros_wrapper_node-1] Rotation Speed: 600
[hesai_ros_wrapper_node-1] Sync Angle: 0
[hesai_ros_wrapper_node-1] Cut Angle: 0
[hesai_ros_wrapper_node-1] FoV Start: 0
[hesai_ros_wrapper_node-1] FoV End: 360
[hesai_ros_wrapper_node-1] Dual Return Distance Threshold: 0.1
[hesai_ros_wrapper_node-1] Calibration Path: /home/nagoya/nebula_ws/install/nebula_decoders/share/nebula_decoders/calibration/hesai/PandarXT16.csv
[hesai_ros_wrapper_node-1] PTP Profile: IEEE_1588v2
[hesai_ros_wrapper_node-1] PTP Domain: 0
[hesai_ros_wrapper_node-1] PTP Transport Type: UDP/IP
[hesai_ros_wrapper_node-1] PTP Switch Type: TSN (HesaiRosWrapper() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/hesai_ros_wrapper.cpp:40)
[hesai_ros_wrapper_node-1] [INFO 1732779747.839965434] [IoContext::IoContext]: Thread(s) Created: 1 (IoContext() at /home/nagoya/nebula_ws/src/transport_drivers/io_context/src/io_context.cpp:44)
[hesai_ros_wrapper_node-1] [INFO 1732779747.840244039] [TcpSocket::open]: connected (open() at /home/nagoya/nebula_ws/src/transport_drivers/tcp_driver/src/tcp_socket.cpp:531)
[hesai_ros_wrapper_node-1] 192.168.1.201:9347
[hesai_ros_wrapper_node-1] [WARN 1732779747.841259869] [hesai_ros_wrapper_node]: Sensor returned longer payload than expected. Truncating and parsing anyway. (CheckSizeAndParse() at /home/nagoya/nebula_ws/src/nebula/nebula_hw_interfaces/src/nebula_hesai_hw_interfaces/hesai_hw_interface.cpp:1439)
[hesai_ros_wrapper_node-1] [INFO 1732779747.861800554] [hesai_ros_wrapper_node]: current lidar sync: 1 (PrintInfo() at /home/nagoya/nebula_ws/src/nebula/nebula_hw_interfaces/src/nebula_hesai_hw_interfaces/hesai_hw_interface.cpp:1324)
[hesai_ros_wrapper_node-1] [INFO 1732779747.861882661] [hesai_ros_wrapper_node]: current lidar sync_angle: 0 (PrintInfo() at /home/nagoya/nebula_ws/src/nebula/nebula_hw_interfaces/src/nebula_hesai_hw_interfaces/hesai_hw_interface.cpp:1324)
[hesai_ros_wrapper_node-1] [INFO 1732779747.861916518] [hesai_ros_wrapper_node]: current configuration sync_angle: 0 (PrintInfo() at /home/nagoya/nebula_ws/src/nebula/nebula_hw_interfaces/src/nebula_hesai_hw_interfaces/hesai_hw_interface.cpp:1324)
[hesai_ros_wrapper_node-1] [INFO 1732779748.005295839] [hesai_ros_wrapper_node]: Trying to set Clock source to PTP (PrintInfo() at /home/nagoya/nebula_ws/src/nebula/nebula_hw_interfaces/src/nebula_hesai_hw_interfaces/hesai_hw_interface.cpp:1324)
[hesai_ros_wrapper_node-1] [INFO 1732779748.106083371] [hesai_ros_wrapper_node]: Trying to set PTP Config: IEEE_1588v2, Domain: 0, Transport: UDP/IP, Switch Type: TSN via TCP (PrintInfo() at /home/nagoya/nebula_ws/src/nebula/nebula_hw_interfaces/src/nebula_hesai_hw_interfaces/hesai_hw_interface.cpp:1324)
[hesai_ros_wrapper_node-1] [INFO 1732779748.213214034] [hesai_ros_wrapper_node.HwMonitor]: Inventory info: sn: XT38C9519038CB53, date_of_manufacture: 2023-08-06, mac: ec:9f:0d:01:6a:91, sw_ver: 1.0.10, hw_ver: 1.0.0, control_fw_ver: 1.2.7, sensor_fw_ver:    1.5.7, zero_angle_offset: 8794, model: PandarXT16, motor_type: unidirectional, num_of_lines: 16 (HesaiHwMonitorWrapper() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/hw_monitor_wrapper.cpp:55)
[hesai_ros_wrapper_node-1] [INFO 1732779748.213312324] [hesai_ros_wrapper_node.HwMonitor]: Hardware ID: PandarXT16: XT38C9519038CB53 (HesaiHwMonitorWrapper() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/hw_monitor_wrapper.cpp:62)
[hesai_ros_wrapper_node-1] [INFO 1732779748.215094648] [hesai_ros_wrapper_node]: Downloaded calibration data from sensor. (get_calibration_data() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/hesai_ros_wrapper.cpp:471)
[hesai_ros_wrapper_node-1] [INFO 1732779748.215409746] [hesai_ros_wrapper_node]: Saved downloaded data to /home/nagoya/nebula_ws/install/nebula_decoders/share/nebula_decoders/calibration/hesai/PandarXT16_from_sensor_192.168.1.201.csv (get_calibration_data() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/hesai_ros_wrapper.cpp:476)
[hesai_ros_wrapper_node-1] [INFO 1732779748.499850349] [hesai_ros_wrapper_node.HesaiDecoder]: Using calibration data from /home/nagoya/nebula_ws/install/nebula_decoders/share/nebula_decoders/calibration/hesai/PandarXT16_from_sensor_192.168.1.201.csv (HesaiDecoderWrapper() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/decoder_wrapper.cpp:36)
[hesai_ros_wrapper_node-1] [INFO 1732779748.499931152] [hesai_ros_wrapper_node.HesaiDecoder]: Starting Decoder (HesaiDecoderWrapper() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/decoder_wrapper.cpp:39)
[hesai_ros_wrapper_node-1] [INFO 1732779748.512553625] [hesai_ros_wrapper_node.HesaiDecoder]: . Wrapper=OK (HesaiDecoderWrapper() at /home/nagoya/nebula_ws/src/nebula/nebula_ros/src/hesai/decoder_wrapper.cpp:67)
[hesai_ros_wrapper_node-1] [INFO 1732779748.512756871] [hesai_ros_wrapper_node]: Starting UDP receiver (PrintInfo() at /home/nagoya/nebula_ws/src/nebula/nebula_hw_interfaces/src/nebula_hesai_hw_interfaces/hesai_hw_interface.cpp:1324)

I believe this might be related to max_scan_buffer_points which I changed to 128000 (Max data points x2?). I will keep looking into it. Otherwise, pointcloud looks good.

Edit: Just found this in the docs - "max_scan_buffer_points = bytes_per_second / lowest_supported_frequency", I will fix this

@imeshsps
Copy link

@mojomex The issue was with the Reliability Policy setting. I was able to get the data on Rviz.

nebula 3

@mojomex
Copy link
Collaborator

mojomex commented Nov 28, 2024

@jemmmel @imeshsps Thanks for your guys' good work!
Are you collaborating or did you come up with contributions independently?

@jemmmel For the payload size warning, it is related to parsing of TCP metadata, such as diagnostics and sensor config. Hesai has documents for that, but they're not public and I don't have any for XT16.

For now, if diagnostics look all good and the printed responses from the sensor look reasonable (they do), you can go ahead and create a PR :D

Please make sure to follow our contribution guidelines for a smooth process. Thank you!

@jemmmel
Copy link

jemmmel commented Nov 28, 2024

@mojomex Ok thanks for the info! We are working independently, just good timing I guess.
I was wondering about the tests, I am yet to add one for this sensor or should I just do the PR ASAP?

@mojomex
Copy link
Collaborator

mojomex commented Nov 28, 2024

@jemmmel Yes, please add tests. You can do something similar to the other Hesai sensors in nebula_tests.
They test the decode functionality, and ground-truth data can be generated by recording a rosbag of scan messages from /pandar_packets and then converting one of them to PCD. The test suite then compares the ground-truth pointcloud with the one Nebula decodes from the rosbag.

nebula_examples can be used for the conversion, but the script is not up-to-date. See #236 for a working version 🙇

Also, if you could record a PCAP (with Wireshark for example) with all TCP, UDP, HTTP data (no filters), where you start Nebula and let it run for a few seconds, that would help our in-progress TCP protocol unit tests (#234) greatly!

@jemmmel
Copy link

jemmmel commented Nov 29, 2024

Good morning @mojomex, I am trying to collect the test data but I am having trouble with the conversion script. I am using the #236 version to convert a recorded bag file of /pandar_packets. My workflow is

  1. Record a bag file of /pandar_packets with ros2 bag record topics /pandar_packets
  2. Run ros2 launch nebula_examples hesai_offline_bag_pcd.xml on this bag file

However in the new bag file, the type is not pandar_msgs/msg/PandarScan but sensor_msgs/msg/PointCloud2

Am I doing something out of order?

Also, where should I upload the PCAP data?
Thanks!

@mojomex
Copy link
Collaborator

mojomex commented Nov 29, 2024

@jemmmel Hi, I think you don't need the rosbag output at all, only the PCD output. You could run with these parameters:

cd ~/nebula
ros2 launch nebula_examples hesai_offline_bag_pcd.xml sensor_model:=PandarXT32 output_rosbag:=true forward_packets_to_rosbag:=true  output_pcd:=true bag_path:=nebula_tests/data/hesai/xt32/1673400677802009732/ input_topic:=/pandar_packets output_topic:=UNUSED out_num:=3 skip_num:=1 out_path:=.

That PR is still under review and the parameter handling isn't quite perfect, so all of those extra parameters are needed....
But output_rosbag:=true forward_packets_to_rosbag:=true output_pcd:=true should ensure you get the PCD files and a trimmed rosbag along with them. It still contains pointlcloud messages too, but that's fine for now.

image

The test suite will match the filename of the PCD file(s) against packet messages in the bag and auto-decode and compare. Just add xt16 to the nebula_tests/data/hesai folder structure and add a test case in here.

As for the PCAP, just any google drive/dropbox/etc. link is fine!

@jemmmel
Copy link

jemmmel commented Nov 29, 2024

In that case I think it is ready to go, should I submit the PR?

PCAP data can be found here - https://drive.google.com/file/d/1uKecBbVFgmKDmJbymWjCmcUQagybVGK4/view?usp=drive_link

@jemmmel
Copy link

jemmmel commented Nov 29, 2024

I have started the process, please let me know if there are any issues and I will take a look!
#241

@mojomex mojomex linked a pull request Nov 29, 2024 that will close this issue
5 tasks
@mojomex
Copy link
Collaborator

mojomex commented Nov 29, 2024

@jemmmel Thanks, I left a small review!

@drwnz
Copy link
Collaborator

drwnz commented Dec 3, 2024

Great work. I'll close this issue as the PR is now merged. Thank you for the contribution @jemmmel !

@drwnz drwnz closed this as completed Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants