About simulation speed, reinforcement learning and sim to real

[Jamy-L]

Hello all, I wanted to discuss about the matter of the simulation speed when it comes to reinforcement learning. For obvious reasons, training an agent with 1 to 1 simulation clock is very inefficient, mostly because of how long every simulation takes. The most logical way to make the learning more efficient is to speed up the simulation, which I had no real success in so far. I believe it's actually quite hard for many reasons :

What really matters is the simulation step used by PhysX, which is constant throughout any simulation. The trick is to take it small enough so that simulation is accurate, but big enough so that the sim can run very fast. But there is no such parameter in AirSim, or at least I couldn't find it.
The only setting available is "ClockSpeed" in settings.JSON, but this setting doesn't make much sense for reinforcement learning, since we don't really care about having a consistent clock scale, but rather picking an optimal PhysX time step.

Working with ClockSpeed is quite tricky, here are the things I have and figured out :

Working with big clock speed will end up making simulations groundless because the PhysX time step ends up being too big for simulating real behaviour. It seems that PhysX is made for running video games at at least 30 FPS, so the intuitive thumb rule I have found is to make sure your framerate divided by the clock scale is always above 30. for example if you run AirSim at constant 60FPS, you should never pick a clock speed of more than 2.

However FPS are not consistent at all, and it's just a (poor) approximation : you may never know if the clock speed you have picked is just doubling the speed, or completely breaking physics.

By the way, the editor caps the framerate at 120 FPS, so make sure to launch the project first. You can also make the resolution very low and set all the graphic settings to low to gain even more FPS... but so far I have still not found any way to display the FPS once the project is launched, so I have no idea how high I can set the clock speed. Disabling display may also be a way to get more FPS but I have not found how.

I believe working with a constant clock scale doesn't make much sense for many applications, and it would be great to have a mode where we can fix a physically coherent PhysX time step and let the clock speed do whatever it wants and go however high the hardware can handle it.

Please, let me know your opinion on that 🙂, I am on no account a specialist and I would love to be corrected or to learn more about how all of this is working, and how you have managed to go beyond that obstacle.

[wagh311]

Hello, I also found that changing clockspeed will have a bad impact on RL. Have you found a suitable solution to speed up training?

[Jamy-L]

The solution mentioned above with the rule of thumb of 30 FPS kind of worked if I remember right. Since simulation steps have a dynamic duration, it makes sense to represent a RL step as as fixed simulation duration (0.1 sec for example), rather than as a fixed number of simulation steps.

I did a successful training on a 3090 with a clock speed of 4, with the minimal screen resolution. The training itself performed quite well and the trained model was able to drive the car based on lidar data without crashing static objects. It did not transfer well to the real world however, but it may be unrelated.

[wagh311]

Thank you very much for your reply. I still have some questions:

1.When you say 30FPS, do you mean to ensure that the frame rate divided by the clock ratio is always above 30FPS?
2.How to adjust the screen resolution?How can I ensure that I adjust the screen resolution to the lowest setting?Do I need to turn off rendering to further improve FPS while adjusting the screen resolution?
3.In my RL settings, the action execution time in each time step is 0.3s. Is this reasonable?
4.I tried to train the UAV to avoid obstacles when clockspeed=4 and the training was successful. But when I test the trained policy with clockspeed=1, the UAV collides and no useful policy seems to be learned. Does this mean that training and testing must be done with the same clcokspeed? If this is the case, can the trained policy be transferred to the real environment?

Hope to get your reply, thank you again!

[Jamy-L]

Yes, I meant that the frame rate divided by the closk speed should stay above 30 FPS. I dont recall exactly how to adjust the screen resolution, but it this is a feature of the unreal engine editor that should be documented online. A timestep of 0.3 sec sounds reasonable, although it could be slightly too long depending on the application targeted (300 ms appears to be roughly the reaction time of a human driver).

Your experience with the clockspeed is very interesting, and proves that something is indeed wrong. I have not tried it, but there should in theory be no difference when changing the clockspeed, as long as you carefuly keep the same RL time step between both experience. My code looked like that:

        self.client.simPause(False)
        
        time.sleep(self.dt / self.ClockSpeed)
        self.client.simPause(True)

where dt was 0.1s or 0.3s in your case. There is also a command called continueForTime(seconds), but I remember it did have problems at the time I experimented with it. It may now work better using that.

[wagh311]

Thanks for your reply, I've tried using 'simPause', but the following problem occurred:
When I set clockspeed=3 and started training, I found that the actual value of clockspeed kept shrinking, and finally clockspeed was approximately equal to 1, which also meant that there was no acceleration. So I'm curious, why does this happen when using 'simPause'?