[Question] Why does FPS tend to decrease during training?

[george-adams1]

❓ Question

I have done a few projects using SB3 and have noticed consistently that my FPS is always very high at the beginning of training and decreases quickly throughout training. For example, I'm training a Starcraft agent now and my initial FPS is 7500, but 1 M timesteps into training, my fps is 3500.

Why is this?

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[araffin]

Hello,
a lot of information is missing, especially the algorithm and hyperparameters used.

[george-adams1]

Using a PPO. Here are the hyperparameters:

    hyperparameters = {
        "learning_rate": linear_schedule(5e-3),
        "n_steps": 1024,
        "batch_size": 1024,
        "n_epochs": 10,
        "gamma": 0.99,
        "gae_lambda": 0.95,
        "clip_range": 0.2,
        "clip_range_vf": None,
        "normalize_advantage": True,
        "ent_coef": 0.0,
        "vf_coef": 0.5,
        "max_grad_norm": 0.5,
        "use_sde": False,
        "sde_sample_freq": -1,
         "target_kl": None,
        'policy_kwargs': None,
        "seed": None,
    }

[araffin]

Another question is do the FPS stabilize at some point or keep decreasing? (if you use progress_bar=True or the -P option with the RL Zoo, you can see the number of steps/s)

Overall, it is an expected behavior, PPO collects n_steps * n_envs transitions before doing the first update (which is the bottleneck), the first time FPS is shown, it is only about collecting transitions (which is fast) and doesn't include the gradient update.
Over time, FPS should converge to a value that also account for the gradient update.
Depending on the env and policy (mlp, cnn, ...), there are different strategies to make it faster (use cpu only, reduce the number of pytorch threads, use subproc vec env, ...), you can search SB3 issues for those (and look at our tutorials for the subproc vec env).

[george-adams1]

The trend is decreasing right away after first update for basically everything except for GPU which ramps up for a few updates and then starts to fall. I don't have all the graphs on this image.

[george-adams1]

@araffin can I ask you another question? I'm running the exact same setup on an M1 MacBook Air with 8 cores and on a desktop Intel i7 with 16 cores. When training default pi and val neural nets, the M1 has almost 3x the FPS of the i7 which is a more powerful desktop CPU. Any idea why this is?

[araffin]

The trend is decreasing right away after first update

As I wrote, this is expected but stabilizes after a while.

Any idea why this is?

Not really, you might want to look at #914 too.

Closing as the original questions was answered.