Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BugFix] Non exclusive terminated and truncated #1911

Merged
merged 2 commits into from
Feb 15, 2024
Merged

[BugFix] Non exclusive terminated and truncated #1911

merged 2 commits into from
Feb 15, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Feb 15, 2024 •
edited
Loading

Closes #1899

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Feb 15, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1911

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit 4e48531 with merge base 45764b5 (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 15, 2024
@vmoens vmoens added bug Something isn't working Suitable for minor Suitable to be integrated in minor release (no new feature) labels Feb 15, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 15, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $\large\color{#35bf28}1$. Worsened: $\large\color{#d91a1a}8$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1484s 68.9460ms 14.5041 Ops/s 15.4034 Ops/s $\textbf{\color{#d91a1a}-5.84\%}$
test_sync 39.4706ms 33.8572ms 29.5358 Ops/s 28.6841 Ops/s $\color{#35bf28}+2.97\%$
test_async 96.5699ms 33.5238ms 29.8295 Ops/s 29.8330 Ops/s $\color{#d91a1a}-0.01\%$
test_simple 0.5239s 0.4518s 2.2133 Ops/s 2.2451 Ops/s $\color{#d91a1a}-1.41\%$
test_transformed 0.6802s 0.6152s 1.6255 Ops/s 1.6095 Ops/s $\color{#35bf28}+0.99\%$
test_serial 1.5558s 1.4921s 0.6702 Ops/s 0.6874 Ops/s $\color{#d91a1a}-2.50\%$
test_parallel 1.4872s 1.4157s 0.7063 Ops/s 0.7197 Ops/s $\color{#d91a1a}-1.86\%$
test_step_mdp_speed[True-True-True-True-True] 0.1122ms 21.0309μs 47.5491 KOps/s 45.6306 KOps/s $\color{#35bf28}+4.20\%$
test_step_mdp_speed[True-True-True-True-False] 39.6930μs 13.0719μs 76.5001 KOps/s 74.4176 KOps/s $\color{#35bf28}+2.80\%$
test_step_mdp_speed[True-True-True-False-True] 41.4970μs 12.4849μs 80.0967 KOps/s 77.9793 KOps/s $\color{#35bf28}+2.72\%$
test_step_mdp_speed[True-True-True-False-False] 51.6460μs 7.6215μs 131.2083 KOps/s 128.1148 KOps/s $\color{#35bf28}+2.41\%$
test_step_mdp_speed[True-True-False-True-True] 71.7530μs 22.7601μs 43.9365 KOps/s 43.1854 KOps/s $\color{#35bf28}+1.74\%$
test_step_mdp_speed[True-True-False-True-False] 45.2040μs 14.4495μs 69.2066 KOps/s 68.1652 KOps/s $\color{#35bf28}+1.53\%$
test_step_mdp_speed[True-True-False-False-True] 53.3590μs 13.5947μs 73.5578 KOps/s 70.7149 KOps/s $\color{#35bf28}+4.02\%$
test_step_mdp_speed[True-True-False-False-False] 34.2440μs 8.8113μs 113.4912 KOps/s 110.0686 KOps/s $\color{#35bf28}+3.11\%$
test_step_mdp_speed[True-False-True-True-True] 56.4940μs 24.0759μs 41.5353 KOps/s 40.8622 KOps/s $\color{#35bf28}+1.65\%$
test_step_mdp_speed[True-False-True-True-False] 54.8520μs 15.8800μs 62.9724 KOps/s 62.0280 KOps/s $\color{#35bf28}+1.52\%$
test_step_mdp_speed[True-False-True-False-True] 48.1990μs 13.9046μs 71.9187 KOps/s 71.3276 KOps/s $\color{#35bf28}+0.83\%$
test_step_mdp_speed[True-False-True-False-False] 35.3160μs 8.8601μs 112.8659 KOps/s 109.7356 KOps/s $\color{#35bf28}+2.85\%$
test_step_mdp_speed[True-False-False-True-True] 61.8150μs 25.2263μs 39.6411 KOps/s 38.7321 KOps/s $\color{#35bf28}+2.35\%$
test_step_mdp_speed[True-False-False-True-False] 42.2690μs 17.0434μs 58.6738 KOps/s 58.1212 KOps/s $\color{#35bf28}+0.95\%$
test_step_mdp_speed[True-False-False-False-True] 52.6170μs 15.2059μs 65.7640 KOps/s 66.1839 KOps/s $\color{#d91a1a}-0.63\%$
test_step_mdp_speed[True-False-False-False-False] 71.1020μs 10.0749μs 99.2570 KOps/s 97.7429 KOps/s $\color{#35bf28}+1.55\%$
test_step_mdp_speed[False-True-True-True-True] 69.7200μs 24.2714μs 41.2008 KOps/s 40.9036 KOps/s $\color{#35bf28}+0.73\%$
test_step_mdp_speed[False-True-True-True-False] 49.8720μs 15.9450μs 62.7156 KOps/s 62.2344 KOps/s $\color{#35bf28}+0.77\%$
test_step_mdp_speed[False-True-True-False-True] 61.8950μs 16.1767μs 61.8174 KOps/s 61.4745 KOps/s $\color{#35bf28}+0.56\%$
test_step_mdp_speed[False-True-True-False-False] 44.3820μs 10.1290μs 98.7265 KOps/s 97.0974 KOps/s $\color{#35bf28}+1.68\%$
test_step_mdp_speed[False-True-False-True-True] 36.0770μs 25.5689μs 39.1100 KOps/s 38.1534 KOps/s $\color{#35bf28}+2.51\%$
test_step_mdp_speed[False-True-False-True-False] 47.1480μs 17.0815μs 58.5429 KOps/s 57.6562 KOps/s $\color{#35bf28}+1.54\%$
test_step_mdp_speed[False-True-False-False-True] 67.5750μs 17.3824μs 57.5293 KOps/s 56.4720 KOps/s $\color{#35bf28}+1.87\%$
test_step_mdp_speed[False-True-False-False-False] 58.2970μs 11.4017μs 87.7059 KOps/s 86.6363 KOps/s $\color{#35bf28}+1.23\%$
test_step_mdp_speed[False-False-True-True-True] 54.0300μs 26.6145μs 37.5735 KOps/s 36.9538 KOps/s $\color{#35bf28}+1.68\%$
test_step_mdp_speed[False-False-True-True-False] 47.9090μs 18.3493μs 54.4981 KOps/s 53.9246 KOps/s $\color{#35bf28}+1.06\%$
test_step_mdp_speed[False-False-True-False-True] 0.4325ms 19.6152μs 50.9808 KOps/s 57.3526 KOps/s $\textbf{\color{#d91a1a}-11.11\%}$
test_step_mdp_speed[False-False-True-False-False] 0.2443ms 14.4589μs 69.1614 KOps/s 86.8047 KOps/s $\textbf{\color{#d91a1a}-20.33\%}$
test_step_mdp_speed[False-False-False-True-True] 59.1790μs 27.7281μs 36.0645 KOps/s 35.2763 KOps/s $\color{#35bf28}+2.23\%$
test_step_mdp_speed[False-False-False-True-False] 59.3600μs 19.4669μs 51.3691 KOps/s 50.5830 KOps/s $\color{#35bf28}+1.55\%$
test_step_mdp_speed[False-False-False-False-True] 55.3020μs 18.1433μs 55.1166 KOps/s 54.2653 KOps/s $\color{#35bf28}+1.57\%$
test_step_mdp_speed[False-False-False-False-False] 45.3940μs 12.4467μs 80.3425 KOps/s 79.0846 KOps/s $\color{#35bf28}+1.59\%$
test_values[generalized_advantage_estimate-True-True] 9.3901ms 9.1658ms 109.1017 Ops/s 104.8242 Ops/s $\color{#35bf28}+4.08\%$
test_values[vec_generalized_advantage_estimate-True-True] 45.0181ms 35.5819ms 28.1042 Ops/s 30.0004 Ops/s $\textbf{\color{#d91a1a}-6.32\%}$
test_values[td0_return_estimate-False-False] 0.2102ms 0.1818ms 5.5015 KOps/s 5.1874 KOps/s $\textbf{\color{#35bf28}+6.05\%}$
test_values[td1_return_estimate-False-False] 25.0682ms 22.9506ms 43.5719 Ops/s 42.0706 Ops/s $\color{#35bf28}+3.57\%$
test_values[vec_td1_return_estimate-False-False] 36.3428ms 35.1057ms 28.4854 Ops/s 29.8149 Ops/s $\color{#d91a1a}-4.46\%$
test_values[td_lambda_return_estimate-True-False] 36.2617ms 33.2059ms 30.1151 Ops/s 29.0400 Ops/s $\color{#35bf28}+3.70\%$
test_values[vec_td_lambda_return_estimate-True-False] 36.6988ms 35.1603ms 28.4412 Ops/s 30.0524 Ops/s $\textbf{\color{#d91a1a}-5.36\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 7.9920ms 7.8807ms 126.8928 Ops/s 121.4465 Ops/s $\color{#35bf28}+4.48\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.2665ms 1.9568ms 511.0342 Ops/s 504.9719 Ops/s $\color{#35bf28}+1.20\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4156ms 0.3478ms 2.8755 KOps/s 2.8546 KOps/s $\color{#35bf28}+0.73\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 44.4779ms 42.3862ms 23.5926 Ops/s 25.4430 Ops/s $\textbf{\color{#d91a1a}-7.27\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 3.6041ms 3.0322ms 329.7938 Ops/s 327.0295 Ops/s $\color{#35bf28}+0.85\%$
test_dqn_speed 76.0531ms 1.4895ms 671.3626 Ops/s 721.1250 Ops/s $\textbf{\color{#d91a1a}-6.90\%}$
test_ddpg_speed 3.8089ms 2.7479ms 363.9120 Ops/s 360.9758 Ops/s $\color{#35bf28}+0.81\%$
test_sac_speed 10.4924ms 8.6746ms 115.2791 Ops/s 113.8523 Ops/s $\color{#35bf28}+1.25\%$
test_redq_speed 16.8298ms 14.2095ms 70.3753 Ops/s 71.3898 Ops/s $\color{#d91a1a}-1.42\%$
test_redq_deprec_speed 15.5397ms 13.8750ms 72.0718 Ops/s 69.6641 Ops/s $\color{#35bf28}+3.46\%$
test_td3_speed 9.9358ms 8.8660ms 112.7907 Ops/s 110.7066 Ops/s $\color{#35bf28}+1.88\%$
test_cql_speed 39.9851ms 37.5049ms 26.6632 Ops/s 26.8448 Ops/s $\color{#d91a1a}-0.68\%$
test_a2c_speed 8.5899ms 7.5632ms 132.2189 Ops/s 133.5850 Ops/s $\color{#d91a1a}-1.02\%$
test_ppo_speed 9.0971ms 7.7902ms 128.3668 Ops/s 127.4364 Ops/s $\color{#35bf28}+0.73\%$
test_reinforce_speed 7.7862ms 6.6920ms 149.4329 Ops/s 147.1938 Ops/s $\color{#35bf28}+1.52\%$
test_iql_speed 35.0745ms 33.8077ms 29.5790 Ops/s 29.5639 Ops/s $\color{#35bf28}+0.05\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 4.7795ms 2.9201ms 342.4523 Ops/s 336.7860 Ops/s $\color{#35bf28}+1.68\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8004ms 0.5357ms 1.8668 KOps/s 1.8821 KOps/s $\color{#d91a1a}-0.81\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8625ms 0.5081ms 1.9680 KOps/s 1.9871 KOps/s $\color{#d91a1a}-0.96\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.2501ms 2.9887ms 334.5909 Ops/s 330.4060 Ops/s $\color{#35bf28}+1.27\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.8403ms 0.5296ms 1.8882 KOps/s 1.8920 KOps/s $\color{#d91a1a}-0.20\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7755ms 0.5018ms 1.9928 KOps/s 1.9978 KOps/s $\color{#d91a1a}-0.25\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.6140ms 3.0534ms 327.5014 Ops/s 324.6593 Ops/s $\color{#35bf28}+0.88\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0523ms 0.6524ms 1.5329 KOps/s 1.5664 KOps/s $\color{#d91a1a}-2.14\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7662ms 0.6215ms 1.6089 KOps/s 1.6335 KOps/s $\color{#d91a1a}-1.50\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.0695ms 2.8948ms 345.4493 Ops/s 344.9983 Ops/s $\color{#35bf28}+0.13\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.8595ms 0.5365ms 1.8639 KOps/s 1.8839 KOps/s $\color{#d91a1a}-1.07\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.6884ms 0.5110ms 1.9571 KOps/s 1.9736 KOps/s $\color{#d91a1a}-0.84\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.3604ms 3.0412ms 328.8147 Ops/s 333.7174 Ops/s $\color{#d91a1a}-1.47\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7496ms 0.5437ms 1.8393 KOps/s 1.8949 KOps/s $\color{#d91a1a}-2.93\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6491ms 0.5115ms 1.9549 KOps/s 2.0128 KOps/s $\color{#d91a1a}-2.88\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 4.3131ms 3.1361ms 318.8655 Ops/s 331.2982 Ops/s $\color{#d91a1a}-3.75\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.0297ms 0.6753ms 1.4808 KOps/s 1.5611 KOps/s $\textbf{\color{#d91a1a}-5.15\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 1.0254ms 0.6430ms 1.5552 KOps/s 1.6014 KOps/s $\color{#d91a1a}-2.88\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1311s 9.0042ms 111.0593 Ops/s 115.7743 Ops/s $\color{#d91a1a}-4.07\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 16.0246ms 13.7243ms 72.8636 Ops/s 73.9515 Ops/s $\color{#d91a1a}-1.47\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 5.3729ms 2.6136ms 382.6176 Ops/s 394.6046 Ops/s $\color{#d91a1a}-3.04\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1115s 10.4677ms 95.5315 Ops/s 96.6780 Ops/s $\color{#d91a1a}-1.19\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 16.1398ms 13.7004ms 72.9905 Ops/s 74.5636 Ops/s $\color{#d91a1a}-2.11\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 3.7395ms 2.5897ms 386.1494 Ops/s 393.4928 Ops/s $\color{#d91a1a}-1.87\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1152s 8.9714ms 111.4654 Ops/s 116.1482 Ops/s $\color{#d91a1a}-4.03\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 16.8498ms 14.1759ms 70.5423 Ops/s 72.7842 Ops/s $\color{#d91a1a}-3.08\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 4.6170ms 2.9479ms 339.2251 Ops/s 331.4373 Ops/s $\color{#35bf28}+2.35\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 15, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $\large\color{#35bf28}2$. Worsened: $\large\color{#d91a1a}2$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_single 0.1159s 0.1153s 8.6716 Ops/s 8.4654 Ops/s $\color{#35bf28}+2.44\%$
test_sync 0.1714s 0.1025s 9.7605 Ops/s 9.7296 Ops/s $\color{#35bf28}+0.32\%$
test_async 0.2519s 91.5469ms 10.9234 Ops/s 10.9905 Ops/s $\color{#d91a1a}-0.61\%$
test_single_pixels 0.1277s 0.1270s 7.8736 Ops/s 7.4671 Ops/s $\textbf{\color{#35bf28}+5.44\%}$
test_sync_pixels 82.5197ms 80.8543ms 12.3679 Ops/s 12.6434 Ops/s $\color{#d91a1a}-2.18\%$
test_async_pixels 0.2201s 75.6187ms 13.2242 Ops/s 13.4102 Ops/s $\color{#d91a1a}-1.39\%$
test_simple 0.9113s 0.8418s 1.1879 Ops/s 1.2095 Ops/s $\color{#d91a1a}-1.79\%$
test_transformed 1.1347s 1.0718s 0.9330 Ops/s 0.9069 Ops/s $\color{#35bf28}+2.88\%$
test_serial 2.5603s 2.4905s 0.4015 Ops/s 0.3998 Ops/s $\color{#35bf28}+0.43\%$
test_parallel 2.3384s 2.1801s 0.4587 Ops/s 0.4684 Ops/s $\color{#d91a1a}-2.07\%$
test_step_mdp_speed[True-True-True-True-True] 92.0520μs 33.2495μs 30.0756 KOps/s 30.3635 KOps/s $\color{#d91a1a}-0.95\%$
test_step_mdp_speed[True-True-True-True-False] 35.5810μs 20.1242μs 49.6915 KOps/s 50.1742 KOps/s $\color{#d91a1a}-0.96\%$
test_step_mdp_speed[True-True-True-False-True] 33.6110μs 18.8088μs 53.1665 KOps/s 53.3767 KOps/s $\color{#d91a1a}-0.39\%$
test_step_mdp_speed[True-True-True-False-False] 32.8010μs 11.3211μs 88.3309 KOps/s 89.3863 KOps/s $\color{#d91a1a}-1.18\%$
test_step_mdp_speed[True-True-False-True-True] 72.4520μs 34.7426μs 28.7831 KOps/s 28.8145 KOps/s $\color{#d91a1a}-0.11\%$
test_step_mdp_speed[True-True-False-True-False] 46.0910μs 21.6588μs 46.1706 KOps/s 46.5469 KOps/s $\color{#d91a1a}-0.81\%$
test_step_mdp_speed[True-True-False-False-True] 43.0210μs 20.6610μs 48.4003 KOps/s 48.5435 KOps/s $\color{#d91a1a}-0.30\%$
test_step_mdp_speed[True-True-False-False-False] 30.2910μs 13.0549μs 76.5997 KOps/s 75.7127 KOps/s $\color{#35bf28}+1.17\%$
test_step_mdp_speed[True-False-True-True-True] 60.0510μs 37.1986μs 26.8827 KOps/s 27.0108 KOps/s $\color{#d91a1a}-0.47\%$
test_step_mdp_speed[True-False-True-True-False] 47.0910μs 23.2793μs 42.9566 KOps/s 42.0981 KOps/s $\color{#35bf28}+2.04\%$
test_step_mdp_speed[True-False-True-False-True] 42.4510μs 20.5573μs 48.6445 KOps/s 48.1493 KOps/s $\color{#35bf28}+1.03\%$
test_step_mdp_speed[True-False-True-False-False] 42.1910μs 13.2555μs 75.4403 KOps/s 75.5830 KOps/s $\color{#d91a1a}-0.19\%$
test_step_mdp_speed[True-False-False-True-True] 69.1610μs 38.3596μs 26.0691 KOps/s 25.6241 KOps/s $\color{#35bf28}+1.74\%$
test_step_mdp_speed[True-False-False-True-False] 38.9600μs 25.1271μs 39.7976 KOps/s 39.3988 KOps/s $\color{#35bf28}+1.01\%$
test_step_mdp_speed[True-False-False-False-True] 40.5810μs 22.4858μs 44.4725 KOps/s 44.7270 KOps/s $\color{#d91a1a}-0.57\%$
test_step_mdp_speed[True-False-False-False-False] 36.0710μs 14.9718μs 66.7922 KOps/s 66.5512 KOps/s $\color{#35bf28}+0.36\%$
test_step_mdp_speed[False-True-True-True-True] 70.3310μs 36.6716μs 27.2690 KOps/s 27.0575 KOps/s $\color{#35bf28}+0.78\%$
test_step_mdp_speed[False-True-True-True-False] 49.5110μs 23.7276μs 42.1450 KOps/s 42.2860 KOps/s $\color{#d91a1a}-0.33\%$
test_step_mdp_speed[False-True-True-False-True] 77.9510μs 24.3474μs 41.0722 KOps/s 40.9052 KOps/s $\color{#35bf28}+0.41\%$
test_step_mdp_speed[False-True-True-False-False] 28.9100μs 14.9894μs 66.7137 KOps/s 67.4386 KOps/s $\color{#d91a1a}-1.07\%$
test_step_mdp_speed[False-True-False-True-True] 63.5010μs 38.8975μs 25.7086 KOps/s 25.4414 KOps/s $\color{#35bf28}+1.05\%$
test_step_mdp_speed[False-True-False-True-False] 48.7210μs 25.7666μs 38.8100 KOps/s 39.0039 KOps/s $\color{#d91a1a}-0.50\%$
test_step_mdp_speed[False-True-False-False-True] 48.5610μs 25.7637μs 38.8144 KOps/s 38.6328 KOps/s $\color{#35bf28}+0.47\%$
test_step_mdp_speed[False-True-False-False-False] 40.4810μs 16.4289μs 60.8682 KOps/s 59.8374 KOps/s $\color{#35bf28}+1.72\%$
test_step_mdp_speed[False-False-True-True-True] 61.2410μs 40.1926μs 24.8802 KOps/s 24.7183 KOps/s $\color{#35bf28}+0.66\%$
test_step_mdp_speed[False-False-True-True-False] 44.3600μs 27.4640μs 36.4113 KOps/s 36.4635 KOps/s $\color{#d91a1a}-0.14\%$
test_step_mdp_speed[False-False-True-False-True] 47.7800μs 25.7447μs 38.8430 KOps/s 38.5364 KOps/s $\color{#35bf28}+0.80\%$
test_step_mdp_speed[False-False-True-False-False] 35.4100μs 16.5775μs 60.3227 KOps/s 59.8199 KOps/s $\color{#35bf28}+0.84\%$
test_step_mdp_speed[False-False-False-True-True] 66.1810μs 41.6915μs 23.9857 KOps/s 23.9653 KOps/s $\color{#35bf28}+0.09\%$
test_step_mdp_speed[False-False-False-True-False] 46.2810μs 29.3750μs 34.0425 KOps/s 34.0080 KOps/s $\color{#35bf28}+0.10\%$
test_step_mdp_speed[False-False-False-False-True] 54.9410μs 27.3430μs 36.5724 KOps/s 36.4487 KOps/s $\color{#35bf28}+0.34\%$
test_step_mdp_speed[False-False-False-False-False] 43.2100μs 18.4984μs 54.0586 KOps/s 54.4748 KOps/s $\color{#d91a1a}-0.76\%$
test_values[generalized_advantage_estimate-True-True] 27.2934ms 26.9092ms 37.1621 Ops/s 39.0352 Ops/s $\color{#d91a1a}-4.80\%$
test_values[vec_generalized_advantage_estimate-True-True] 86.0913ms 3.2953ms 303.4652 Ops/s 304.3166 Ops/s $\color{#d91a1a}-0.28\%$
test_values[td0_return_estimate-False-False] 0.1162ms 63.0573μs 15.8586 KOps/s 15.6126 KOps/s $\color{#35bf28}+1.58\%$
test_values[td1_return_estimate-False-False] 57.5863ms 56.8388ms 17.5936 Ops/s 18.5031 Ops/s $\color{#d91a1a}-4.92\%$
test_values[vec_td1_return_estimate-False-False] 2.1422ms 1.7774ms 562.6209 Ops/s 565.6019 Ops/s $\color{#d91a1a}-0.53\%$
test_values[td_lambda_return_estimate-True-False] 91.1652ms 88.4048ms 11.3116 Ops/s 11.6331 Ops/s $\color{#d91a1a}-2.76\%$
test_values[vec_td_lambda_return_estimate-True-False] 3.9731ms 1.8102ms 552.4375 Ops/s 553.7202 Ops/s $\color{#d91a1a}-0.23\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 25.9558ms 25.1044ms 39.8337 Ops/s 41.8861 Ops/s $\color{#d91a1a}-4.90\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 0.8938ms 0.7296ms 1.3706 KOps/s 1.4090 KOps/s $\color{#d91a1a}-2.72\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.7341ms 0.6622ms 1.5102 KOps/s 1.5252 KOps/s $\color{#d91a1a}-0.98\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.4972ms 1.4686ms 680.9357 Ops/s 683.9371 Ops/s $\color{#d91a1a}-0.44\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.9561ms 0.6834ms 1.4633 KOps/s 1.4767 KOps/s $\color{#d91a1a}-0.91\%$
test_dqn_speed 1.7670ms 1.4379ms 695.4377 Ops/s 680.2748 Ops/s $\color{#35bf28}+2.23\%$
test_ddpg_speed 3.0278ms 2.7784ms 359.9172 Ops/s 361.5743 Ops/s $\color{#d91a1a}-0.46\%$
test_sac_speed 9.1964ms 8.5985ms 116.2988 Ops/s 117.2908 Ops/s $\color{#d91a1a}-0.85\%$
test_redq_speed 11.2799ms 10.5380ms 94.8948 Ops/s 94.5377 Ops/s $\color{#35bf28}+0.38\%$
test_redq_deprec_speed 12.7063ms 11.9973ms 83.3520 Ops/s 79.5348 Ops/s $\color{#35bf28}+4.80\%$
test_td3_speed 9.0258ms 8.8113ms 113.4902 Ops/s 114.7062 Ops/s $\color{#d91a1a}-1.06\%$
test_cql_speed 0.1194s 28.6163ms 34.9451 Ops/s 38.7488 Ops/s $\textbf{\color{#d91a1a}-9.82\%}$
test_a2c_speed 5.8738ms 5.3894ms 185.5495 Ops/s 182.9680 Ops/s $\color{#35bf28}+1.41\%$
test_ppo_speed 5.9967ms 5.7262ms 174.6352 Ops/s 171.9439 Ops/s $\color{#35bf28}+1.57\%$
test_reinforce_speed 5.3435ms 4.5379ms 220.3682 Ops/s 223.3837 Ops/s $\color{#d91a1a}-1.35\%$
test_iql_speed 20.8411ms 19.8616ms 50.3484 Ops/s 50.4062 Ops/s $\color{#d91a1a}-0.11\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.7332ms 3.5858ms 278.8778 Ops/s 280.1155 Ops/s $\color{#d91a1a}-0.44\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.7475ms 0.5598ms 1.7864 KOps/s 1.8026 KOps/s $\color{#d91a1a}-0.90\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7098ms 0.5296ms 1.8881 KOps/s 1.8892 KOps/s $\color{#d91a1a}-0.06\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.8351ms 3.6094ms 277.0580 Ops/s 281.5115 Ops/s $\color{#d91a1a}-1.58\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7693ms 0.5527ms 1.8093 KOps/s 1.8211 KOps/s $\color{#d91a1a}-0.65\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6788ms 0.5244ms 1.9069 KOps/s 1.9045 KOps/s $\color{#35bf28}+0.13\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.8816ms 3.7400ms 267.3807 Ops/s 269.1451 Ops/s $\color{#d91a1a}-0.66\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8600ms 0.6825ms 1.4652 KOps/s 1.4646 KOps/s $\color{#35bf28}+0.04\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.7848ms 0.6559ms 1.5245 KOps/s 1.5192 KOps/s $\color{#35bf28}+0.35\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 3.7015ms 3.6049ms 277.4029 Ops/s 279.8454 Ops/s $\color{#d91a1a}-0.87\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.6830ms 0.5617ms 1.7804 KOps/s 1.7969 KOps/s $\color{#d91a1a}-0.92\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.7229ms 0.5311ms 1.8830 KOps/s 1.8853 KOps/s $\color{#d91a1a}-0.12\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 3.7695ms 3.6076ms 277.1893 Ops/s 278.1162 Ops/s $\color{#d91a1a}-0.33\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.6746ms 0.5496ms 1.8194 KOps/s 1.8085 KOps/s $\color{#35bf28}+0.60\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.6602ms 0.5267ms 1.8987 KOps/s 1.9058 KOps/s $\color{#d91a1a}-0.37\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 3.8552ms 3.7464ms 266.9259 Ops/s 270.8081 Ops/s $\color{#d91a1a}-1.43\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.8758ms 0.6816ms 1.4672 KOps/s 1.4569 KOps/s $\color{#35bf28}+0.71\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8505ms 0.6554ms 1.5258 KOps/s 1.5156 KOps/s $\color{#35bf28}+0.68\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 0.1279s 10.1385ms 98.6339 Ops/s 97.4323 Ops/s $\color{#35bf28}+1.23\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 0.1250s 18.7512ms 53.3300 Ops/s 61.7086 Ops/s $\textbf{\color{#d91a1a}-13.58\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 5.5731ms 3.0107ms 332.1468 Ops/s 325.3158 Ops/s $\color{#35bf28}+2.10\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.1148s 9.8610ms 101.4092 Ops/s 99.6556 Ops/s $\color{#35bf28}+1.76\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 19.0748ms 16.3432ms 61.1875 Ops/s 61.5364 Ops/s $\color{#d91a1a}-0.57\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 5.9710ms 3.0045ms 332.8340 Ops/s 334.0117 Ops/s $\color{#d91a1a}-0.35\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 0.1188s 10.2471ms 97.5889 Ops/s 98.2185 Ops/s $\color{#d91a1a}-0.64\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 19.2474ms 16.8413ms 59.3778 Ops/s 53.0329 Ops/s $\textbf{\color{#35bf28}+11.96\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 6.9659ms 3.3061ms 302.4683 Ops/s 304.6033 Ops/s $\color{#d91a1a}-0.70\%$

@vmoens vmoens merged commit bd7e268 into main Feb 15, 2024
65 of 68 checks passed
skandermoalla
skandermoalla reviewed Feb 15, 2024
View reviewed changes
torchrl/envs/transforms/transforms.py
Comment on lines +5181 to +5184
# we can have terminated and truncated
# terminated = next_tensordict.get(terminated_key, None)
# if terminated is not None:
# truncated = truncated & ~terminated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really get the comment left here.

@skandermoalla
Copy link
Contributor

Line 5081+ can be removed I think

@property
    def terminated_keys(self):
        terminated_keys = self.__dict__.get("_terminated_keys", None)
        if terminated_keys is None:
            # make the default terminated keys
            terminated_keys = []
            for reset_key in self.parent._filtered_reset_keys:
                if isinstance(reset_key, str):
                    key = "terminated"
                else:
                    key = (*reset_key[:-1], "terminated")
                terminated_keys.append(key)
        self.__dict__["_terminated_keys"] = terminated_keys
return terminated_keys

@skandermoalla
Copy link
Contributor

A test would be nice? To check if when truncated_key="terminated" everything still works fine.

@vmoens vmoens deleted the non-mutually-exclusive-terminated-truncated branch February 27, 2024 00:47
vmoens added a commit that referenced this pull request Feb 28, 2024
@vmoens
[BugFix] Non exclusive terminated and truncated (#1911)
24d7391
@vmoens vmoens mentioned this pull request Feb 28, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@skandermoalla skandermoalla skandermoalla left review comments

Assignees
No one assigned
Labels
bug Something isn't working CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Suitable for minor Suitable to be integrated in minor release (no new feature)
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] Why have mutually exclusive terminated and truncated?
3 participants
@vmoens @skandermoalla @facebook-github-bot