Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] Make sure ParallelEnv does not overflow mem when policy requires grad #1909

Merged
merged 7 commits into from
Feb 15, 2024
Merged

[BugFix] Make sure ParallelEnv does not overflow mem when policy requires grad #1909

merged 7 commits into from
Feb 15, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 15, 2024

Also adds a test to make sure SerialEnv gradients work ok

cc @AndreaTirinzoni

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 15, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1909

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (4 Unrelated Failures)

As of commit a6dc5ee with merge base 899af07 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 15, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 15, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}8$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 62.7995ms 61.4820ms 16.2649 Ops/s 15.8279 Ops/s $\color{#35bf28}+2.76\%$
test_sync 39.5130ms 34.3400ms 29.1206 Ops/s 29.2080 Ops/s $\color{#d91a1a}-0.30\%$
test_async 0.1132s 33.3509ms 29.9842 Ops/s 30.0007 Ops/s $\color{#d91a1a}-0.05\%$
test_simple 0.4912s 0.4334s 2.3071 Ops/s 2.3069 Ops/s $\color{#35bf28}+0.01\%$
test_transformed 0.6501s 0.5885s 1.6992 Ops/s 1.6601 Ops/s $\color{#35bf28}+2.35\%$
test_serial 1.4410s 1.3870s 0.7210 Ops/s 0.6985 Ops/s $\color{#35bf28}+3.23\%$
test_parallel 1.4485s 1.3635s 0.7334 Ops/s 0.7194 Ops/s $\color{#35bf28}+1.94\%$
test_step_mdp_speed[True-True-True-True-True] 0.1229ms 22.0921μs 45.2651 KOps/s 46.7858 KOps/s $\color{#d91a1a}-3.25\%$
test_step_mdp_speed[True-True-True-True-False] 44.7030μs 13.7366μs 72.7984 KOps/s 76.5652 KOps/s $\color{#d91a1a}-4.92\%$
test_step_mdp_speed[True-True-True-False-True] 69.3450μs 12.9373μs 77.2959 KOps/s 81.2046 KOps/s $\color{#d91a1a}-4.81\%$
test_step_mdp_speed[True-True-True-False-False] 27.8120μs 7.9096μs 126.4293 KOps/s 131.5163 KOps/s $\color{#d91a1a}-3.87\%$
test_step_mdp_speed[True-True-False-True-True] 71.6960μs 23.2353μs 43.0379 KOps/s 44.2086 KOps/s $\color{#d91a1a}-2.65\%$
test_step_mdp_speed[True-True-False-True-False] 69.3790μs 14.9107μs 67.0661 KOps/s 70.1783 KOps/s $\color{#d91a1a}-4.43\%$
test_step_mdp_speed[True-True-False-False-True] 47.9490μs 14.1132μs 70.8556 KOps/s 73.5277 KOps/s $\color{#d91a1a}-3.63\%$
test_step_mdp_speed[True-True-False-False-False] 49.5920μs 9.2167μs 108.4986 KOps/s 113.9253 KOps/s $\color{#d91a1a}-4.76\%$
test_step_mdp_speed[True-False-True-True-True] 96.2290μs 24.6007μs 40.6492 KOps/s 42.1397 KOps/s $\color{#d91a1a}-3.54\%$
test_step_mdp_speed[True-False-True-True-False] 50.8440μs 16.0838μs 62.1744 KOps/s 64.7744 KOps/s $\color{#d91a1a}-4.01\%$
test_step_mdp_speed[True-False-True-False-True] 60.8540μs 14.0514μs 71.1675 KOps/s 73.3029 KOps/s $\color{#d91a1a}-2.91\%$
test_step_mdp_speed[True-False-True-False-False] 35.4750μs 9.2752μs 107.8144 KOps/s 113.8768 KOps/s $\textbf{\color{#d91a1a}-5.32\%}$
test_step_mdp_speed[True-False-False-True-True] 54.5410μs 25.7228μs 38.8760 KOps/s 39.4749 KOps/s $\color{#d91a1a}-1.52\%$
test_step_mdp_speed[True-False-False-True-False] 42.4290μs 17.5763μs 56.8948 KOps/s 59.7091 KOps/s $\color{#d91a1a}-4.71\%$
test_step_mdp_speed[True-False-False-False-True] 41.3260μs 15.2829μs 65.4326 KOps/s 68.1194 KOps/s $\color{#d91a1a}-3.94\%$
test_step_mdp_speed[True-False-False-False-False] 42.9700μs 10.4068μs 96.0906 KOps/s 100.3458 KOps/s $\color{#d91a1a}-4.24\%$
test_step_mdp_speed[False-True-True-True-True] 61.8550μs 24.6405μs 40.5836 KOps/s 41.8662 KOps/s $\color{#d91a1a}-3.06\%$
test_step_mdp_speed[False-True-True-True-False] 49.8820μs 16.2882μs 61.3942 KOps/s 64.0533 KOps/s $\color{#d91a1a}-4.15\%$
test_step_mdp_speed[False-True-True-False-True] 43.8110μs 16.5990μs 60.2448 KOps/s 62.7777 KOps/s $\color{#d91a1a}-4.03\%$
test_step_mdp_speed[False-True-True-False-False] 62.0750μs 10.5242μs 95.0193 KOps/s 101.5099 KOps/s $\textbf{\color{#d91a1a}-6.39\%}$
test_step_mdp_speed[False-True-False-True-True] 49.9530μs 26.2692μs 38.0675 KOps/s 39.5879 KOps/s $\color{#d91a1a}-3.84\%$
test_step_mdp_speed[False-True-False-True-False] 39.7240μs 17.3803μs 57.5362 KOps/s 59.5516 KOps/s $\color{#d91a1a}-3.38\%$
test_step_mdp_speed[False-True-False-False-True] 58.2480μs 17.6547μs 56.6420 KOps/s 58.7014 KOps/s $\color{#d91a1a}-3.51\%$
test_step_mdp_speed[False-True-False-False-False] 75.5140μs 11.6376μs 85.9282 KOps/s 89.7495 KOps/s $\color{#d91a1a}-4.26\%$
test_step_mdp_speed[False-False-True-True-True] 74.7690μs 27.0737μs 36.9363 KOps/s 37.8616 KOps/s $\color{#d91a1a}-2.44\%$
test_step_mdp_speed[False-False-True-True-False] 71.5600μs 18.7578μs 53.3112 KOps/s 54.6296 KOps/s $\color{#d91a1a}-2.41\%$
test_step_mdp_speed[False-False-True-False-True] 39.5130μs 17.5138μs 57.0980 KOps/s 58.4825 KOps/s $\color{#d91a1a}-2.37\%$
test_step_mdp_speed[False-False-True-False-False] 38.4710μs 11.6634μs 85.7380 KOps/s 88.2413 KOps/s $\color{#d91a1a}-2.84\%$
test_step_mdp_speed[False-False-False-True-True] 72.3040μs 27.9268μs 35.8079 KOps/s 36.4896 KOps/s $\color{#d91a1a}-1.87\%$
test_step_mdp_speed[False-False-False-True-False] 49.9830μs 19.7985μs 50.5090 KOps/s 51.9299 KOps/s $\color{#d91a1a}-2.74\%$
test_step_mdp_speed[False-False-False-False-True] 70.3010μs 18.5317μs 53.9615 KOps/s 55.0574 KOps/s $\color{#d91a1a}-1.99\%$
test_step_mdp_speed[False-False-False-False-False] 39.8450μs 12.7183μs 78.6268 KOps/s 81.2215 KOps/s $\color{#d91a1a}-3.19\%$
test_values[generalized_advantage_estimate-True-True] 9.8173ms 9.3665ms 106.7634 Ops/s 106.9607 Ops/s $\color{#d91a1a}-0.18\%$
test_values[vec_generalized_advantage_estimate-True-True] 36.5676ms 34.9805ms 28.5874 Ops/s 28.5927 Ops/s $\color{#d91a1a}-0.02\%$
test_values[td0_return_estimate-False-False] 0.2488ms 0.1862ms 5.3706 KOps/s 6.1045 KOps/s $\textbf{\color{#d91a1a}-12.02\%}$
test_values[td1_return_estimate-False-False] 24.2480ms 23.4593ms 42.6269 Ops/s 42.9748 Ops/s $\color{#d91a1a}-0.81\%$
test_values[vec_td1_return_estimate-False-False] 36.2597ms 35.0198ms 28.5553 Ops/s 28.5991 Ops/s $\color{#d91a1a}-0.15\%$
test_values[td_lambda_return_estimate-True-False] 37.1660ms 33.9751ms 29.4334 Ops/s 29.8919 Ops/s $\color{#d91a1a}-1.53\%$
test_values[vec_td_lambda_return_estimate-True-False] 36.4439ms 35.1062ms 28.4850 Ops/s 28.6252 Ops/s $\color{#d91a1a}-0.49\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.7742ms 8.1765ms 122.3024 Ops/s 123.5006 Ops/s $\color{#d91a1a}-0.97\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.5843ms 2.0178ms 495.6011 Ops/s 505.5291 Ops/s $\color{#d91a1a}-1.96\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4198ms 0.3506ms 2.8525 KOps/s 2.8554 KOps/s $\color{#d91a1a}-0.10\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 47.1950ms 44.2662ms 22.5906 Ops/s 22.4911 Ops/s $\color{#35bf28}+0.44\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.6747ms 3.0560ms 327.2202 Ops/s 329.8754 Ops/s $\color{#d91a1a}-0.80\%$
test_dqn_speed 84.9814ms 1.4822ms 674.6808 Ops/s 732.5672 Ops/s $\textbf{\color{#d91a1a}-7.90\%}$
test_ddpg_speed 3.2131ms 2.7066ms 369.4704 Ops/s 365.3198 Ops/s $\color{#35bf28}+1.14\%$
test_sac_speed 9.6430ms 8.5465ms 117.0065 Ops/s 114.6919 Ops/s $\color{#35bf28}+2.02\%$
test_redq_speed 14.8184ms 13.4319ms 74.4496 Ops/s 74.7168 Ops/s $\color{#d91a1a}-0.36\%$
test_redq_deprec_speed 15.2998ms 13.8768ms 72.0626 Ops/s 72.5528 Ops/s $\color{#d91a1a}-0.68\%$
test_td3_speed 9.3515ms 8.6912ms 115.0594 Ops/s 114.8332 Ops/s $\color{#35bf28}+0.20\%$
test_cql_speed 39.4712ms 37.2444ms 26.8497 Ops/s 27.2565 Ops/s $\color{#d91a1a}-1.49\%$
test_a2c_speed 8.7128ms 7.4287ms 134.6128 Ops/s 136.1814 Ops/s $\color{#d91a1a}-1.15\%$
test_ppo_speed 8.1016ms 7.7143ms 129.6295 Ops/s 130.0405 Ops/s $\color{#d91a1a}-0.32\%$
test_reinforce_speed 7.6342ms 6.6915ms 149.4426 Ops/s 151.0444 Ops/s $\color{#d91a1a}-1.06\%$
test_iql_speed 34.2456ms 33.4238ms 29.9188 Ops/s 30.3905 Ops/s $\color{#d91a1a}-1.55\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.2675ms 2.8712ms 348.2844 Ops/s 371.0268 Ops/s $\textbf{\color{#d91a1a}-6.13\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7482ms 0.5151ms 1.9415 KOps/s 1.9315 KOps/s $\color{#35bf28}+0.52\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.9524ms 0.4942ms 2.0237 KOps/s 2.0183 KOps/s $\color{#35bf28}+0.26\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 4.0792ms 2.8373ms 352.4420 Ops/s 358.2492 Ops/s $\color{#d91a1a}-1.62\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7298ms 0.5054ms 1.9784 KOps/s 1.9577 KOps/s $\color{#35bf28}+1.06\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.8015ms 0.4848ms 2.0625 KOps/s 2.0213 KOps/s $\color{#35bf28}+2.04\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.8742ms 3.0171ms 331.4438 Ops/s 353.1929 Ops/s $\textbf{\color{#d91a1a}-6.16\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9537ms 0.6376ms 1.5684 KOps/s 1.5661 KOps/s $\color{#35bf28}+0.15\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8676ms 0.6085ms 1.6434 KOps/s 1.6381 KOps/s $\color{#35bf28}+0.32\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.0556ms 2.8194ms 354.6880 Ops/s 366.1384 Ops/s $\color{#d91a1a}-3.13\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.9372ms 0.5115ms 1.9550 KOps/s 1.9388 KOps/s $\color{#35bf28}+0.84\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7935ms 0.4973ms 2.0108 KOps/s 2.0061 KOps/s $\color{#35bf28}+0.24\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.3149ms 2.8464ms 351.3206 Ops/s 361.2859 Ops/s $\color{#d91a1a}-2.76\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6591ms 0.5066ms 1.9738 KOps/s 1.9443 KOps/s $\color{#35bf28}+1.52\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.8796ms 0.4823ms 2.0735 KOps/s 2.0574 KOps/s $\color{#35bf28}+0.78\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.2413ms 2.9166ms 342.8699 Ops/s 346.4617 Ops/s $\color{#d91a1a}-1.04\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.7657ms 0.6329ms 1.5801 KOps/s 1.4825 KOps/s $\textbf{\color{#35bf28}+6.58\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8842ms 0.6083ms 1.6439 KOps/s 1.6124 KOps/s $\color{#35bf28}+1.95\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 97.2091ms 9.6169ms 103.9837 Ops/s 128.4099 Ops/s $\textbf{\color{#d91a1a}-19.02\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 15.8766ms 13.4454ms 74.3747 Ops/s 75.5098 Ops/s $\color{#d91a1a}-1.50\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 4.2742ms 2.5296ms 395.3126 Ops/s 389.2594 Ops/s $\color{#35bf28}+1.56\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 98.8180ms 7.8372ms 127.5966 Ops/s 101.7218 Ops/s $\textbf{\color{#35bf28}+25.44\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 0.1089s 15.3476ms 65.1567 Ops/s 75.9572 Ops/s $\textbf{\color{#d91a1a}-14.22\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 4.4154ms 2.5563ms 391.1906 Ops/s 390.2340 Ops/s $\color{#35bf28}+0.25\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1010s 8.1973ms 121.9918 Ops/s 100.2805 Ops/s $\textbf{\color{#35bf28}+21.65\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 16.2835ms 13.9803ms 71.5294 Ops/s 74.5045 Ops/s $\color{#d91a1a}-3.99\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 5.1354ms 2.7910ms 358.2999 Ops/s 363.3867 Ops/s $\color{#d91a1a}-1.40\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 15, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}9$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1171s 0.1166s 8.5790 Ops/s 8.7313 Ops/s $\color{#d91a1a}-1.74\%$
test_sync 0.1726s 0.1033s 9.6817 Ops/s 9.6864 Ops/s $\color{#d91a1a}-0.05\%$
test_async 0.1814s 91.8909ms 10.8825 Ops/s 10.9853 Ops/s $\color{#d91a1a}-0.94\%$
test_single_pixels 0.1372s 0.1369s 7.3049 Ops/s 7.7975 Ops/s $\textbf{\color{#d91a1a}-6.32\%}$
test_sync_pixels 83.0982ms 81.6980ms 12.2402 Ops/s 12.6030 Ops/s $\color{#d91a1a}-2.88\%$
test_async_pixels 0.1973s 73.8068ms 13.5489 Ops/s 13.5869 Ops/s $\color{#d91a1a}-0.28\%$
test_simple 0.9017s 0.8312s 1.2030 Ops/s 1.2530 Ops/s $\color{#d91a1a}-3.99\%$
test_transformed 1.1457s 1.0783s 0.9274 Ops/s 0.9473 Ops/s $\color{#d91a1a}-2.11\%$
test_serial 2.4850s 2.4173s 0.4137 Ops/s 0.4305 Ops/s $\color{#d91a1a}-3.91\%$
test_parallel 2.2438s 2.1003s 0.4761 Ops/s 0.4849 Ops/s $\color{#d91a1a}-1.81\%$
test_step_mdp_speed[True-True-True-True-True] 0.1087ms 33.1781μs 30.1404 KOps/s 31.6514 KOps/s $\color{#d91a1a}-4.77\%$
test_step_mdp_speed[True-True-True-True-False] 45.1200μs 20.0778μs 49.8062 KOps/s 52.0555 KOps/s $\color{#d91a1a}-4.32\%$
test_step_mdp_speed[True-True-True-False-True] 44.7400μs 18.8198μs 53.1355 KOps/s 54.8190 KOps/s $\color{#d91a1a}-3.07\%$
test_step_mdp_speed[True-True-True-False-False] 29.5800μs 11.0963μs 90.1197 KOps/s 90.8613 KOps/s $\color{#d91a1a}-0.82\%$
test_step_mdp_speed[True-True-False-True-True] 65.1510μs 35.1316μs 28.4644 KOps/s 29.4216 KOps/s $\color{#d91a1a}-3.25\%$
test_step_mdp_speed[True-True-False-True-False] 43.9800μs 21.8550μs 45.7560 KOps/s 48.1179 KOps/s $\color{#d91a1a}-4.91\%$
test_step_mdp_speed[True-True-False-False-True] 52.2100μs 20.6961μs 48.3184 KOps/s 49.6362 KOps/s $\color{#d91a1a}-2.66\%$
test_step_mdp_speed[True-True-False-False-False] 45.1400μs 13.0495μs 76.6315 KOps/s 78.1083 KOps/s $\color{#d91a1a}-1.89\%$
test_step_mdp_speed[True-False-True-True-True] 66.6910μs 36.8048μs 27.1704 KOps/s 27.6057 KOps/s $\color{#d91a1a}-1.58\%$
test_step_mdp_speed[True-False-True-True-False] 38.0700μs 23.6619μs 42.2621 KOps/s 42.9265 KOps/s $\color{#d91a1a}-1.55\%$
test_step_mdp_speed[True-False-True-False-True] 43.3610μs 20.2099μs 49.4807 KOps/s 49.7479 KOps/s $\color{#d91a1a}-0.54\%$
test_step_mdp_speed[True-False-True-False-False] 33.9600μs 13.2426μs 75.5139 KOps/s 77.6545 KOps/s $\color{#d91a1a}-2.76\%$
test_step_mdp_speed[True-False-False-True-True] 67.8700μs 38.9478μs 25.6754 KOps/s 26.2797 KOps/s $\color{#d91a1a}-2.30\%$
test_step_mdp_speed[True-False-False-True-False] 48.8000μs 25.4825μs 39.2426 KOps/s 39.8893 KOps/s $\color{#d91a1a}-1.62\%$
test_step_mdp_speed[True-False-False-False-True] 44.7300μs 21.8639μs 45.7376 KOps/s 45.4866 KOps/s $\color{#35bf28}+0.55\%$
test_step_mdp_speed[True-False-False-False-False] 31.0000μs 14.9287μs 66.9851 KOps/s 68.5984 KOps/s $\color{#d91a1a}-2.35\%$
test_step_mdp_speed[False-True-True-True-True] 61.2200μs 37.4921μs 26.6723 KOps/s 27.2273 KOps/s $\color{#d91a1a}-2.04\%$
test_step_mdp_speed[False-True-True-True-False] 45.5110μs 23.4904μs 42.5706 KOps/s 42.8097 KOps/s $\color{#d91a1a}-0.56\%$
test_step_mdp_speed[False-True-True-False-True] 50.7600μs 24.4130μs 40.9618 KOps/s 41.5490 KOps/s $\color{#d91a1a}-1.41\%$
test_step_mdp_speed[False-True-True-False-False] 41.4110μs 15.1756μs 65.8953 KOps/s 68.3145 KOps/s $\color{#d91a1a}-3.54\%$
test_step_mdp_speed[False-True-False-True-True] 74.9610μs 38.2747μs 26.1269 KOps/s 26.2526 KOps/s $\color{#d91a1a}-0.48\%$
test_step_mdp_speed[False-True-False-True-False] 48.8500μs 25.8778μs 38.6432 KOps/s 39.5310 KOps/s $\color{#d91a1a}-2.25\%$
test_step_mdp_speed[False-True-False-False-True] 46.8300μs 26.4293μs 37.8368 KOps/s 38.4507 KOps/s $\color{#d91a1a}-1.60\%$
test_step_mdp_speed[False-True-False-False-False] 38.6290μs 16.9721μs 58.9203 KOps/s 60.3212 KOps/s $\color{#d91a1a}-2.32\%$
test_step_mdp_speed[False-False-True-True-True] 72.1010μs 40.8893μs 24.4563 KOps/s 25.0030 KOps/s $\color{#d91a1a}-2.19\%$
test_step_mdp_speed[False-False-True-True-False] 49.4300μs 27.6709μs 36.1391 KOps/s 36.6937 KOps/s $\color{#d91a1a}-1.51\%$
test_step_mdp_speed[False-False-True-False-True] 48.6100μs 25.5553μs 39.1309 KOps/s 38.8748 KOps/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[False-False-True-False-False] 40.3510μs 16.6479μs 60.0676 KOps/s 60.5300 KOps/s $\color{#d91a1a}-0.76\%$
test_step_mdp_speed[False-False-False-True-True] 71.7810μs 41.6364μs 24.0175 KOps/s 24.2425 KOps/s $\color{#d91a1a}-0.93\%$
test_step_mdp_speed[False-False-False-True-False] 56.0900μs 29.3063μs 34.1223 KOps/s 34.6779 KOps/s $\color{#d91a1a}-1.60\%$
test_step_mdp_speed[False-False-False-False-True] 54.3400μs 26.7308μs 37.4101 KOps/s 37.1442 KOps/s $\color{#35bf28}+0.72\%$
test_step_mdp_speed[False-False-False-False-False] 40.1600μs 18.3888μs 54.3808 KOps/s 55.0246 KOps/s $\color{#d91a1a}-1.17\%$
test_values[generalized_advantage_estimate-True-True] 25.8157ms 25.2773ms 39.5612 Ops/s 40.3367 Ops/s $\color{#d91a1a}-1.92\%$
test_values[vec_generalized_advantage_estimate-True-True] 84.9018ms 3.2666ms 306.1264 Ops/s 297.1593 Ops/s $\color{#35bf28}+3.02\%$
test_values[td0_return_estimate-False-False] 0.1005ms 62.4134μs 16.0222 KOps/s 16.3744 KOps/s $\color{#d91a1a}-2.15\%$
test_values[td1_return_estimate-False-False] 58.0219ms 56.8943ms 17.5764 Ops/s 19.1727 Ops/s $\textbf{\color{#d91a1a}-8.33\%}$
test_values[vec_td1_return_estimate-False-False] 2.1456ms 1.8030ms 554.6284 Ops/s 569.7081 Ops/s $\color{#d91a1a}-2.65\%$
test_values[td_lambda_return_estimate-True-False] 93.0446ms 92.4699ms 10.8143 Ops/s 12.0133 Ops/s $\textbf{\color{#d91a1a}-9.98\%}$
test_values[vec_td_lambda_return_estimate-True-False] 3.8526ms 1.8048ms 554.0852 Ops/s 558.0733 Ops/s $\color{#d91a1a}-0.71\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 25.8219ms 25.2145ms 39.6596 Ops/s 43.0827 Ops/s $\textbf{\color{#d91a1a}-7.95\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.8882ms 0.7397ms 1.3519 KOps/s 1.4317 KOps/s $\textbf{\color{#d91a1a}-5.57\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7149ms 0.6600ms 1.5151 KOps/s 1.5435 KOps/s $\color{#d91a1a}-1.84\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5355ms 1.4649ms 682.6274 Ops/s 689.8699 Ops/s $\color{#d91a1a}-1.05\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9526ms 0.6778ms 1.4754 KOps/s 1.5019 KOps/s $\color{#d91a1a}-1.76\%$
test_dqn_speed 7.6218ms 1.4542ms 687.6615 Ops/s 709.8561 Ops/s $\color{#d91a1a}-3.13\%$
test_ddpg_speed 2.9403ms 2.7984ms 357.3429 Ops/s 366.6892 Ops/s $\color{#d91a1a}-2.55\%$
test_sac_speed 9.2945ms 8.5025ms 117.6121 Ops/s 121.2321 Ops/s $\color{#d91a1a}-2.99\%$
test_redq_speed 11.2397ms 10.4562ms 95.6374 Ops/s 95.9347 Ops/s $\color{#d91a1a}-0.31\%$
test_redq_deprec_speed 11.7906ms 11.3444ms 88.1489 Ops/s 86.4788 Ops/s $\color{#35bf28}+1.93\%$
test_td3_speed 8.7752ms 8.5569ms 116.8648 Ops/s 117.8800 Ops/s $\color{#d91a1a}-0.86\%$
test_cql_speed 26.1641ms 25.3395ms 39.4641 Ops/s 39.1418 Ops/s $\color{#35bf28}+0.82\%$
test_a2c_speed 96.3064ms 5.9065ms 169.3058 Ops/s 188.7909 Ops/s $\textbf{\color{#d91a1a}-10.32\%}$
test_ppo_speed 5.9689ms 5.6793ms 176.0772 Ops/s 178.4285 Ops/s $\color{#d91a1a}-1.32\%$
test_reinforce_speed 4.7286ms 4.4667ms 223.8812 Ops/s 224.1307 Ops/s $\color{#d91a1a}-0.11\%$
test_iql_speed 20.3487ms 19.5888ms 51.0497 Ops/s 50.5973 Ops/s $\color{#35bf28}+0.89\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.8642ms 3.6070ms 277.2396 Ops/s 284.3004 Ops/s $\color{#d91a1a}-2.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7249ms 0.5645ms 1.7716 KOps/s 1.7956 KOps/s $\color{#d91a1a}-1.34\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6725ms 0.5341ms 1.8722 KOps/s 1.8937 KOps/s $\color{#d91a1a}-1.13\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.8791ms 3.6223ms 276.0676 Ops/s 281.0566 Ops/s $\color{#d91a1a}-1.78\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6986ms 0.5533ms 1.8074 KOps/s 1.8251 KOps/s $\color{#d91a1a}-0.97\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7070ms 0.5270ms 1.8975 KOps/s 1.9264 KOps/s $\color{#d91a1a}-1.50\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.8393ms 3.7058ms 269.8497 Ops/s 273.1752 Ops/s $\color{#d91a1a}-1.22\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8208ms 0.6821ms 1.4661 KOps/s 1.4757 KOps/s $\color{#d91a1a}-0.65\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7811ms 0.6547ms 1.5274 KOps/s 1.3591 KOps/s $\textbf{\color{#35bf28}+12.38\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.7324ms 3.5949ms 278.1736 Ops/s 281.1753 Ops/s $\color{#d91a1a}-1.07\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6928ms 0.5601ms 1.7854 KOps/s 1.7996 KOps/s $\color{#d91a1a}-0.79\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6475ms 0.5338ms 1.8734 KOps/s 1.6241 KOps/s $\textbf{\color{#35bf28}+15.35\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.7836ms 3.6127ms 276.7989 Ops/s 278.4871 Ops/s $\color{#d91a1a}-0.61\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6853ms 0.5530ms 1.8084 KOps/s 1.8239 KOps/s $\color{#d91a1a}-0.85\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.8475ms 0.5294ms 1.8889 KOps/s 1.9093 KOps/s $\color{#d91a1a}-1.07\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.8331ms 3.7386ms 267.4806 Ops/s 271.8693 Ops/s $\color{#d91a1a}-1.61\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.9131ms 0.6828ms 1.4646 KOps/s 1.4799 KOps/s $\color{#d91a1a}-1.03\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7988ms 0.6616ms 1.5115 KOps/s 1.5354 KOps/s $\color{#d91a1a}-1.56\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1242s 10.1556ms 98.4677 Ops/s 81.5797 Ops/s $\textbf{\color{#35bf28}+20.70\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 19.3582ms 16.7313ms 59.7681 Ops/s 63.5015 Ops/s $\textbf{\color{#d91a1a}-5.88\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 8.0115ms 3.1132ms 321.2125 Ops/s 335.7060 Ops/s $\color{#d91a1a}-4.32\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1187s 12.2197ms 81.8350 Ops/s 100.5829 Ops/s $\textbf{\color{#d91a1a}-18.64\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 18.6853ms 16.4032ms 60.9637 Ops/s 63.8073 Ops/s $\color{#d91a1a}-4.46\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 8.0192ms 3.0970ms 322.8921 Ops/s 333.5678 Ops/s $\color{#d91a1a}-3.20\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1192s 10.2588ms 97.4775 Ops/s 96.0935 Ops/s $\color{#35bf28}+1.44\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 19.5362ms 16.8432ms 59.3712 Ops/s 62.6563 Ops/s $\textbf{\color{#d91a1a}-5.24\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 8.5091ms 3.4337ms 291.2339 Ops/s 304.9046 Ops/s $\color{#d91a1a}-4.48\%$

@vmoens vmoens added bug Something isn't working Suitable for minor Suitable to be integrated in minor release (no new feature) labels Feb 15, 2024
@vmoens vmoens merged commit 0314e05 into main Feb 15, 2024
64 of 68 checks passed
@vmoens vmoens deleted the fix-penv branch February 27, 2024 00:47
vmoens added a commit that referenced this pull request Feb 28, 2024
@vmoens
[BugFix] Make sure ParallelEnv does not overflow mem when policy requ…
2ed996c 
…ires grad (#1909)
@vmoens vmoens mentioned this pull request Feb 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Suitable for minor Suitable to be integrated in minor release (no new feature)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vmoens @facebook-github-bot