How does Carla's time management affect Autoware's timers?

[MarioAlmelaUMH]

Hello, I was trying to monitor different Autoware topics, belonging to different modules:

/sensing/camera/traffic_light/image_raw, /sensing/imu/tamagawa/imu_raw, /sensing/lidar/concatenated/pointcloud, /sensing/gnss/pose_with_covariance, /perception/object_recognition/detection/clustering/clusters, /perception/object_recognition/detection/clustering/objects, /perception/object_recognition/detection/objects, /perception/object_recognition/objects, /system/component_state_monitor/component/autonomous/perception, /control/command/control_cmd, /perception/object_recognition/detection/rois0, perception/object_recognition/detection/camera_lidar_fusion/objects

I was recording them in a ros2bag and then checking both the timestamp of the messages' header as well as the ros2bag recording timestamp, as well as the order in which these messages are being recorded. I've noticed some strange behavior, however. As I see in these links: https://github.com/autowarefoundation/autoware.universe/blob/main/control/trajectory_follower_node/src/simple_trajectory_follower.cpp, https://github.com/autowarefoundation/autoware.universe/blob/main/control/shift_decider/src/shift_decider.cpp, Autoware's control modules are set on a timer, in some cases with frequencies around 30 ms. However Carla's default settings set the delta_seconds to 50 ms (delta_time=0.05).

Additionally, it is difficult for the whole platform of Carla, Zenoh bridge and Autoware to work at perfect real time, so the Carla ticks not always tick with a true 50 ms frequency (a tick can sometimes last for a real time second, or half a second, for example). That being said, when monitoring the control topics working on a 30 ms timer, I'd expect to see a bunch of control messages being recorded within each Carla tick, but that doesn't happen. Only one control message appears within each Carla tick.

I've added a CSV screenshot containing the messages of a ros2bag I recorded while using Zenoh bridge. Not all the topics I've listed above were recorded properly, but I hope it serves to illustrate some of the behavior I'm talking about. This was recorded when the ego vehicle was moving normally on the road. It's not edited in any way. I've used this: https://ternaris.gitlab.io/rosbags/index.html to convert the ros2bag to CSV. The E column called timestamp refers to the timestamp stored in each message's header, while the G column called bag_timestamp refers to the timestamp when each message was recorded into the ros2bag.

Given all this:

• How do Autoware timers work in relation to Carla?
• Do they monitor Wall Time, or Ros Time?
• How does Carla's discrete management of time and its non real-time nature affect Autoware's timed nodes?
• Autoware's control commands are sent back to Carla through the bridge. Are they also responsible of ticking Carla when they are received?
• Do you know if ros2bag's recording order is to be trusted as the true order in which the messages were generated?

Sorry for the avalanche of questions, but understanding how Carla's time management and Autoware's relate to each other is critical for the project I'm working on.

Thank you so much for your answers.

[evshary]

Since I provide the /clock and set use_sim_time to true in Autoware, Autoware should use the timestamp in /clock. That is ROS Time.

For Carla ticker, I tick it in the independent thread here. And everytime it ticks, it also published /clock topic. In my setting, 1 Carla simulated second = 2 real world seconds.

For the frequency of control command, it's an interesting topic. I didn't analyze what it should be before. But it shouldn't be related to ticking Carla.

For the ros2bag, I don't have too much knowledge on that.

[MarioAlmelaUMH]

Thank you for your answer.
I've been trying to change the simulation base timestep of 50ms to 1 ms. I've changed this file, both in the fixed_delta_seconds=Some(0.05); as well as in the timer on the bottom (lines 143-144). I've also changed the Carla agent file where the same definition is made.

I'm running the platform in 2 different machines, one with Carla and Zenoh bridge and the other with Autoware and Autoware dds. I've made these changes in both computers.

However, when I ros2bag the Clock topic and extract it afterwards to a CSV, the timestamp is still going on 50 ms jumps, instead of the 1 ms it should be. It's possible that I'm missing some files to change, as the fixed_delta_seconds=0.05 definition was hard-coded in multiple files (and also the ROS Time during simulation was off, as it jumped in 1 ms intervals sometimes, and 50 ms intervals in some occasions too). Do you know how to effectively change simulation time step from 50 ms to 1 ms?

Thank you.

[MarioAlmelaUMH]

Any insight on this last question?

Thank you

[evshary]

Sorry for the late response. I'm quite busy these days.
In fact, you don't need to change the value in Python file since we don't use sync mode in Python.
I didn't use rosbag, but if I echo /clock in Autoware side, I see the clock is changed in 1ms

....
clock:
  sec: 2
  nanosec: 124546207
---
clock:
  sec: 2
  nanosec: 125546207
---
clock:
  sec: 2
  nanosec: 126546207
....

[MarioAlmelaUMH]

Ok, thank you for your answer. I'll double check the topic and see if the parameter change works.

[MarioAlmelaUMH]

Ok, I double checked it, and I'm now able to change the fixed delta seconds parameter successfully (the reason for the error was that I forgot to re-build after modifying the rust files). The ROS Time effectively moves at the correct pace now, but I'm getting these errors:

Apparently, the change in message frequency is overloading the queues, making the simulation stuck. Is there a way to expand this queues so they don't get overloaded?

Thank you for your answers.

[evshary]

I'm not sure whether the message is related to your problem.
I'm trying to adjust the timer these days, but the tick interval in Carla will affect the system and make the system not work. It needs more time to investigate.
If you also find anything interesting, we can discuss it.

[MarioAlmelaUMH]

Ok. However I'm not a skilled developer, so I don't know if I will be able to find anything. If I find something, I'll tell you.

[evshary]

Do you have any reason to set the interval of Carla tick to 1ms?
After some tweaking, I guess the interval smaller than 50 ms seems quite difficult to Autoware.

[MarioAlmelaUMH]

Yeah, I have noticed that the initial position of the vehicle becomes unstable (I try to set it up using RViz's UI, but the vehicle never lands in the right spot). Also, the simulation becomes too slow.

The reason behind this experiment was to try to synchronize the different messages that flow through Autoware and determine exactly when they are being fired. Looking through the code I noticed some nodes run on an independent timer, and we wanted a common predictable base frequency to observe when this timers fired and how they related to each other. We are also recording ROS messages in ros2bag, and the order in which the messages where being recorded seemed a bit odd, so having the base Carla world to work at 1 ms frequency could help decode this weird order.