Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Test] More comprehensive tests for auto_spec #2640

Merged
merged 4 commits into from
Dec 12, 2024
Merged

[Test] More comprehensive tests for auto_spec #2640

merged 4 commits into from
Dec 12, 2024

Conversation

vmoens
Copy link
Contributor

@vmoens vmoens commented Dec 6, 2024 •
edited
Loading

@vmoens
Update
6fa9aed 
[ghstack-poisoned]
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Dec 6, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2640

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 20 Unrelated Failures

As of commit 0f67077 with merge base 4bc40a8 (image):

NEW FAILURES - The following jobs have failed:

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Dec 6, 2024
Merged
Merged
Merged
Closed
Merged
Merged
Merged
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 6, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 6, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}8$. Worsened: $\large\color{#d91a1a}12$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.4336s 0.4315s 2.3176 Ops/s 2.2539 Ops/s $\color{#35bf28}+2.82\%$
test_transformed 0.7195s 0.6325s 1.5811 Ops/s 1.5954 Ops/s $\color{#d91a1a}-0.89\%$
test_serial 1.3737s 1.3551s 0.7379 Ops/s 0.7403 Ops/s $\color{#d91a1a}-0.32\%$
test_parallel 1.3045s 1.2923s 0.7738 Ops/s 0.7684 Ops/s $\color{#35bf28}+0.71\%$
test_step_mdp_speed[True-True-True-True-True] 0.2543ms 29.3002μs 34.1295 KOps/s 33.7318 KOps/s $\color{#35bf28}+1.18\%$
test_step_mdp_speed[True-True-True-True-False] 56.8960μs 17.2187μs 58.0765 KOps/s 58.5448 KOps/s $\color{#d91a1a}-0.80\%$
test_step_mdp_speed[True-True-True-False-True] 45.2950μs 16.3853μs 61.0302 KOps/s 59.8399 KOps/s $\color{#35bf28}+1.99\%$
test_step_mdp_speed[True-True-True-False-False] 34.5740μs 9.5795μs 104.3896 KOps/s 103.6284 KOps/s $\color{#35bf28}+0.73\%$
test_step_mdp_speed[True-True-False-True-True] 98.2530μs 31.2254μs 32.0252 KOps/s 32.0049 KOps/s $\color{#35bf28}+0.06\%$
test_step_mdp_speed[True-True-False-True-False] 70.0900μs 19.0044μs 52.6193 KOps/s 52.3652 KOps/s $\color{#35bf28}+0.49\%$
test_step_mdp_speed[True-True-False-False-True] 50.3040μs 18.2145μs 54.9014 KOps/s 54.8349 KOps/s $\color{#35bf28}+0.12\%$
test_step_mdp_speed[True-True-False-False-False] 47.1280μs 11.3140μs 88.3858 KOps/s 86.3860 KOps/s $\color{#35bf28}+2.31\%$
test_step_mdp_speed[True-False-True-True-True] 74.7900μs 33.0177μs 30.2868 KOps/s 29.7725 KOps/s $\color{#35bf28}+1.73\%$
test_step_mdp_speed[True-False-True-True-False] 54.5620μs 20.7913μs 48.0969 KOps/s 47.5740 KOps/s $\color{#35bf28}+1.10\%$
test_step_mdp_speed[True-False-True-False-True] 51.2250μs 18.1927μs 54.9672 KOps/s 54.9046 KOps/s $\color{#35bf28}+0.11\%$
test_step_mdp_speed[True-False-True-False-False] 42.2580μs 11.4471μs 87.3587 KOps/s 87.0099 KOps/s $\color{#35bf28}+0.40\%$
test_step_mdp_speed[True-False-False-True-True] 0.5537ms 34.3439μs 29.1173 KOps/s 28.6232 KOps/s $\color{#35bf28}+1.73\%$
test_step_mdp_speed[True-False-False-True-False] 56.2450μs 22.5097μs 44.4252 KOps/s 43.9672 KOps/s $\color{#35bf28}+1.04\%$
test_step_mdp_speed[True-False-False-False-True] 82.3740μs 19.6922μs 50.7815 KOps/s 49.7673 KOps/s $\color{#35bf28}+2.04\%$
test_step_mdp_speed[True-False-False-False-False] 54.0710μs 13.0356μs 76.7129 KOps/s 74.8229 KOps/s $\color{#35bf28}+2.53\%$
test_step_mdp_speed[False-True-True-True-True] 87.8740μs 32.7149μs 30.5671 KOps/s 29.8915 KOps/s $\color{#35bf28}+2.26\%$
test_step_mdp_speed[False-True-True-True-False] 52.5480μs 20.9894μs 47.6432 KOps/s 47.3240 KOps/s $\color{#35bf28}+0.67\%$
test_step_mdp_speed[False-True-True-False-True] 50.3440μs 20.8113μs 48.0509 KOps/s 48.0400 KOps/s $\color{#35bf28}+0.02\%$
test_step_mdp_speed[False-True-True-False-False] 47.5690μs 12.7170μs 78.6351 KOps/s 78.0742 KOps/s $\color{#35bf28}+0.72\%$
test_step_mdp_speed[False-True-False-True-True] 85.8000μs 34.8715μs 28.6767 KOps/s 28.6616 KOps/s $\color{#35bf28}+0.05\%$
test_step_mdp_speed[False-True-False-True-False] 59.6020μs 22.7409μs 43.9736 KOps/s 43.8515 KOps/s $\color{#35bf28}+0.28\%$
test_step_mdp_speed[False-True-False-False-True] 0.1078s 26.1122μs 38.2963 KOps/s 43.5187 KOps/s $\textbf{\color{#d91a1a}-12.00\%}$
test_step_mdp_speed[False-True-False-False-False] 46.1360μs 14.2345μs 70.2521 KOps/s 69.6753 KOps/s $\color{#35bf28}+0.83\%$
test_step_mdp_speed[False-False-True-True-True] 68.1770μs 36.4155μs 27.4608 KOps/s 27.6331 KOps/s $\color{#d91a1a}-0.62\%$
test_step_mdp_speed[False-False-True-True-False] 58.4390μs 24.3915μs 40.9979 KOps/s 41.2247 KOps/s $\color{#d91a1a}-0.55\%$
test_step_mdp_speed[False-False-True-False-True] 63.5110μs 22.1671μs 45.1118 KOps/s 44.9797 KOps/s $\color{#35bf28}+0.29\%$
test_step_mdp_speed[False-False-True-False-False] 41.1970μs 14.3290μs 69.7885 KOps/s 69.5300 KOps/s $\color{#35bf28}+0.37\%$
test_step_mdp_speed[False-False-False-True-True] 77.6350μs 37.7741μs 26.4732 KOps/s 26.6052 KOps/s $\color{#d91a1a}-0.50\%$
test_step_mdp_speed[False-False-False-True-False] 0.6053ms 26.1880μs 38.1854 KOps/s 38.9108 KOps/s $\color{#d91a1a}-1.86\%$
test_step_mdp_speed[False-False-False-False-True] 90.8790μs 23.4175μs 42.7031 KOps/s 42.0523 KOps/s $\color{#35bf28}+1.55\%$
test_step_mdp_speed[False-False-False-False-False] 46.4560μs 15.8058μs 63.2677 KOps/s 62.4881 KOps/s $\color{#35bf28}+1.25\%$
test_values[generalized_advantage_estimate-True-True] 10.5597ms 9.5966ms 104.2040 Ops/s 105.4639 Ops/s $\color{#d91a1a}-1.19\%$
test_values[vec_generalized_advantage_estimate-True-True] 37.5921ms 35.9154ms 27.8432 Ops/s 29.3443 Ops/s $\textbf{\color{#d91a1a}-5.12\%}$
test_values[td0_return_estimate-False-False] 0.2652ms 0.1977ms 5.0569 KOps/s 5.3905 KOps/s $\textbf{\color{#d91a1a}-6.19\%}$
test_values[td1_return_estimate-False-False] 25.0725ms 23.7714ms 42.0674 Ops/s 42.3265 Ops/s $\color{#d91a1a}-0.61\%$
test_values[vec_td1_return_estimate-False-False] 37.8137ms 36.0944ms 27.7052 Ops/s 29.5469 Ops/s $\textbf{\color{#d91a1a}-6.23\%}$
test_values[td_lambda_return_estimate-True-False] 37.4983ms 34.2847ms 29.1675 Ops/s 29.1554 Ops/s $\color{#35bf28}+0.04\%$
test_values[vec_td_lambda_return_estimate-True-False] 37.7055ms 36.0989ms 27.7017 Ops/s 29.4598 Ops/s $\textbf{\color{#d91a1a}-5.97\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512] 8.3869ms 8.3008ms 120.4708 Ops/s 120.9856 Ops/s $\color{#d91a1a}-0.43\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 2.3631ms 2.0444ms 489.1485 Ops/s 508.9308 Ops/s $\color{#d91a1a}-3.89\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.4763ms 0.3639ms 2.7483 KOps/s 2.7690 KOps/s $\color{#d91a1a}-0.75\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 47.4734ms 45.6301ms 21.9153 Ops/s 23.3992 Ops/s $\textbf{\color{#d91a1a}-6.34\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 4.8779ms 3.1953ms 312.9634 Ops/s 327.7740 Ops/s $\color{#d91a1a}-4.52\%$
test_dqn_speed[False-None] 2.9279ms 1.3910ms 718.8976 Ops/s 712.7508 Ops/s $\color{#35bf28}+0.86\%$
test_dqn_speed[False-backward] 2.6911ms 1.9281ms 518.6391 Ops/s 534.5896 Ops/s $\color{#d91a1a}-2.98\%$
test_dqn_speed[True-None] 1.0586ms 0.4710ms 2.1231 KOps/s 2.1459 KOps/s $\color{#d91a1a}-1.06\%$
test_dqn_speed[True-backward] 0.9890ms 0.9318ms 1.0732 KOps/s 1.0829 KOps/s $\color{#d91a1a}-0.90\%$
test_dqn_speed[reduce-overhead-None] 0.5851ms 0.4709ms 2.1235 KOps/s 2.1370 KOps/s $\color{#d91a1a}-0.63\%$
test_dqn_speed[reduce-overhead-backward] 0.9947ms 0.9249ms 1.0812 KOps/s 1.0632 KOps/s $\color{#35bf28}+1.70\%$
test_ddpg_speed[False-None] 3.6214ms 2.8743ms 347.9089 Ops/s 344.8010 Ops/s $\color{#35bf28}+0.90\%$
test_ddpg_speed[False-backward] 4.1547ms 4.0500ms 246.9127 Ops/s 248.5264 Ops/s $\color{#d91a1a}-0.65\%$
test_ddpg_speed[True-None] 1.2950ms 0.9944ms 1.0056 KOps/s 998.8597 Ops/s $\color{#35bf28}+0.68\%$
test_ddpg_speed[True-backward] 2.0940ms 1.9395ms 515.6099 Ops/s 453.2504 Ops/s $\textbf{\color{#35bf28}+13.76\%}$
test_ddpg_speed[reduce-overhead-None] 1.3622ms 0.9945ms 1.0055 KOps/s 990.1099 Ops/s $\color{#35bf28}+1.55\%$
test_ddpg_speed[reduce-overhead-backward] 2.1686ms 1.9928ms 501.8048 Ops/s 520.4020 Ops/s $\color{#d91a1a}-3.57\%$
test_sac_speed[False-None] 9.9391ms 8.4563ms 118.2554 Ops/s 124.4399 Ops/s $\color{#d91a1a}-4.97\%$
test_sac_speed[False-backward] 12.1539ms 11.3676ms 87.9690 Ops/s 93.2371 Ops/s $\textbf{\color{#d91a1a}-5.65\%}$
test_sac_speed[True-None] 2.0785ms 1.8282ms 546.9938 Ops/s 545.1043 Ops/s $\color{#35bf28}+0.35\%$
test_sac_speed[True-backward] 4.5927ms 3.6116ms 276.8825 Ops/s 281.9726 Ops/s $\color{#d91a1a}-1.81\%$
test_sac_speed[reduce-overhead-None] 3.3096ms 1.8342ms 545.1933 Ops/s 545.5596 Ops/s $\color{#d91a1a}-0.07\%$
test_sac_speed[reduce-overhead-backward] 3.8044ms 3.5259ms 283.6142 Ops/s 286.5968 Ops/s $\color{#d91a1a}-1.04\%$
test_redq_speed[False-None] 14.7546ms 13.2884ms 75.2535 Ops/s 76.5807 Ops/s $\color{#d91a1a}-1.73\%$
test_redq_speed[False-backward] 23.8296ms 22.6649ms 44.1211 Ops/s 45.0670 Ops/s $\color{#d91a1a}-2.10\%$
test_redq_speed[True-None] 5.9504ms 5.0202ms 199.1950 Ops/s 202.5237 Ops/s $\color{#d91a1a}-1.64\%$
test_redq_speed[True-backward] 14.4375ms 12.5313ms 79.8004 Ops/s 80.0514 Ops/s $\color{#d91a1a}-0.31\%$
test_redq_speed[reduce-overhead-None] 5.5589ms 4.7651ms 209.8613 Ops/s 210.2669 Ops/s $\color{#d91a1a}-0.19\%$
test_redq_speed[reduce-overhead-backward] 15.2207ms 12.9264ms 77.3611 Ops/s 81.0297 Ops/s $\color{#d91a1a}-4.53\%$
test_redq_deprec_speed[False-None] 14.5661ms 13.0521ms 76.6161 Ops/s 76.2819 Ops/s $\color{#35bf28}+0.44\%$
test_redq_deprec_speed[False-backward] 20.8421ms 18.8226ms 53.1277 Ops/s 53.3710 Ops/s $\color{#d91a1a}-0.46\%$
test_redq_deprec_speed[True-None] 4.2962ms 3.5666ms 280.3800 Ops/s 277.8087 Ops/s $\color{#35bf28}+0.93\%$
test_redq_deprec_speed[True-backward] 9.2524ms 8.6288ms 115.8914 Ops/s 122.3031 Ops/s $\textbf{\color{#d91a1a}-5.24\%}$
test_redq_deprec_speed[reduce-overhead-None] 4.3433ms 3.7206ms 268.7713 Ops/s 278.6749 Ops/s $\color{#d91a1a}-3.55\%$
test_redq_deprec_speed[reduce-overhead-backward] 8.8685ms 8.4613ms 118.1848 Ops/s 123.7491 Ops/s $\color{#d91a1a}-4.50\%$
test_td3_speed[False-None] 34.4520ms 8.3573ms 119.6564 Ops/s 123.4698 Ops/s $\color{#d91a1a}-3.09\%$
test_td3_speed[False-backward] 12.3301ms 10.5605ms 94.6927 Ops/s 95.0130 Ops/s $\color{#d91a1a}-0.34\%$
test_td3_speed[True-None] 1.9347ms 1.6890ms 592.0496 Ops/s 580.3520 Ops/s $\color{#35bf28}+2.02\%$
test_td3_speed[True-backward] 3.7082ms 3.4051ms 293.6747 Ops/s 304.5760 Ops/s $\color{#d91a1a}-3.58\%$
test_td3_speed[reduce-overhead-None] 1.8450ms 1.6855ms 593.2981 Ops/s 583.1714 Ops/s $\color{#35bf28}+1.74\%$
test_td3_speed[reduce-overhead-backward] 4.2112ms 3.3410ms 299.3143 Ops/s 294.7131 Ops/s $\color{#35bf28}+1.56\%$
test_cql_speed[False-None] 38.7373ms 36.5198ms 27.3824 Ops/s 26.3493 Ops/s $\color{#35bf28}+3.92\%$
test_cql_speed[False-backward] 57.5022ms 46.5220ms 21.4952 Ops/s 21.6559 Ops/s $\color{#d91a1a}-0.74\%$
test_cql_speed[True-None] 16.8529ms 15.7806ms 63.3691 Ops/s 62.7217 Ops/s $\color{#35bf28}+1.03\%$
test_cql_speed[True-backward] 23.6340ms 22.7275ms 43.9996 Ops/s 43.5024 Ops/s $\color{#35bf28}+1.14\%$
test_cql_speed[reduce-overhead-None] 18.3664ms 15.8747ms 62.9934 Ops/s 61.5106 Ops/s $\color{#35bf28}+2.41\%$
test_cql_speed[reduce-overhead-backward] 23.2919ms 22.6388ms 44.1720 Ops/s 43.7733 Ops/s $\color{#35bf28}+0.91\%$
test_a2c_speed[False-None] 8.6235ms 7.1951ms 138.9836 Ops/s 134.2204 Ops/s $\color{#35bf28}+3.55\%$
test_a2c_speed[False-backward] 15.5610ms 14.5170ms 68.8846 Ops/s 66.9025 Ops/s $\color{#35bf28}+2.96\%$
test_a2c_speed[True-None] 5.0539ms 4.2380ms 235.9617 Ops/s 237.2311 Ops/s $\color{#d91a1a}-0.54\%$
test_a2c_speed[True-backward] 12.1235ms 11.1822ms 89.4276 Ops/s 92.7257 Ops/s $\color{#d91a1a}-3.56\%$
test_a2c_speed[reduce-overhead-None] 4.6670ms 4.2216ms 236.8769 Ops/s 236.2580 Ops/s $\color{#35bf28}+0.26\%$
test_a2c_speed[reduce-overhead-backward] 11.6686ms 10.8840ms 91.8783 Ops/s 91.8151 Ops/s $\color{#35bf28}+0.07\%$
test_ppo_speed[False-None] 9.4094ms 7.5068ms 133.2118 Ops/s 132.9952 Ops/s $\color{#35bf28}+0.16\%$
test_ppo_speed[False-backward] 15.3849ms 14.9187ms 67.0299 Ops/s 64.3452 Ops/s $\color{#35bf28}+4.17\%$
test_ppo_speed[True-None] 4.2367ms 3.7397ms 267.3991 Ops/s 268.5310 Ops/s $\color{#d91a1a}-0.42\%$
test_ppo_speed[True-backward] 10.3648ms 9.7742ms 102.3102 Ops/s 101.9874 Ops/s $\color{#35bf28}+0.32\%$
test_ppo_speed[reduce-overhead-None] 4.2034ms 3.7451ms 267.0156 Ops/s 266.3690 Ops/s $\color{#35bf28}+0.24\%$
test_ppo_speed[reduce-overhead-backward] 10.8084ms 9.8476ms 101.5475 Ops/s 99.6258 Ops/s $\color{#35bf28}+1.93\%$
test_reinforce_speed[False-None] 8.7324ms 6.6170ms 151.1256 Ops/s 146.9713 Ops/s $\color{#35bf28}+2.83\%$
test_reinforce_speed[False-backward] 10.7036ms 9.9995ms 100.0052 Ops/s 98.4210 Ops/s $\color{#35bf28}+1.61\%$
test_reinforce_speed[True-None] 3.2313ms 2.6704ms 374.4691 Ops/s 351.3041 Ops/s $\textbf{\color{#35bf28}+6.59\%}$
test_reinforce_speed[True-backward] 10.2138ms 9.0173ms 110.8974 Ops/s 109.8803 Ops/s $\color{#35bf28}+0.93\%$
test_reinforce_speed[reduce-overhead-None] 3.0742ms 2.7407ms 364.8693 Ops/s 366.8389 Ops/s $\color{#d91a1a}-0.54\%$
test_reinforce_speed[reduce-overhead-backward] 9.2764ms 8.9279ms 112.0089 Ops/s 111.2800 Ops/s $\color{#35bf28}+0.66\%$
test_iql_speed[False-None] 34.1352ms 32.7167ms 30.5655 Ops/s 30.3774 Ops/s $\color{#35bf28}+0.62\%$
test_iql_speed[False-backward] 47.3817ms 45.8351ms 21.8174 Ops/s 21.6542 Ops/s $\color{#35bf28}+0.75\%$
test_iql_speed[True-None] 12.0704ms 11.0082ms 90.8418 Ops/s 88.6959 Ops/s $\color{#35bf28}+2.42\%$
test_iql_speed[True-backward] 22.9418ms 22.0860ms 45.2775 Ops/s 44.8081 Ops/s $\color{#35bf28}+1.05\%$
test_iql_speed[reduce-overhead-None] 11.8555ms 10.9075ms 91.6802 Ops/s 90.6219 Ops/s $\color{#35bf28}+1.17\%$
test_iql_speed[reduce-overhead-backward] 24.1821ms 22.3161ms 44.8108 Ops/s 43.7174 Ops/s $\color{#35bf28}+2.50\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 7.3514ms 5.0221ms 199.1209 Ops/s 187.1809 Ops/s $\textbf{\color{#35bf28}+6.38\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.3375s 0.8355ms 1.1969 KOps/s 1.8632 KOps/s $\textbf{\color{#d91a1a}-35.76\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8449ms 0.4955ms 2.0182 KOps/s 2.0152 KOps/s $\color{#35bf28}+0.15\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 7.6497ms 4.8624ms 205.6594 Ops/s 192.6957 Ops/s $\textbf{\color{#35bf28}+6.73\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.1224ms 0.5032ms 1.9873 KOps/s 1.8846 KOps/s $\textbf{\color{#35bf28}+5.45\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.7257ms 0.4793ms 2.0864 KOps/s 2.0647 KOps/s $\color{#35bf28}+1.05\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.8435ms 1.6288ms 613.9636 Ops/s 605.3877 Ops/s $\color{#35bf28}+1.42\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 2.0819ms 1.5845ms 631.1034 Ops/s 628.0261 Ops/s $\color{#35bf28}+0.49\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 8.0544ms 5.1860ms 192.8260 Ops/s 189.6269 Ops/s $\color{#35bf28}+1.69\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 3.7080ms 0.6664ms 1.5007 KOps/s 1.5114 KOps/s $\color{#d91a1a}-0.71\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.9421ms 0.6246ms 1.6011 KOps/s 1.5646 KOps/s $\color{#35bf28}+2.34\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 5.4759ms 4.9497ms 202.0306 Ops/s 191.2087 Ops/s $\textbf{\color{#35bf28}+5.66\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 2.6559ms 0.5218ms 1.9164 KOps/s 1.8625 KOps/s $\color{#35bf28}+2.89\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.8065ms 0.5031ms 1.9878 KOps/s 2.0103 KOps/s $\color{#d91a1a}-1.12\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 7.6354ms 4.8885ms 204.5634 Ops/s 197.8116 Ops/s $\color{#35bf28}+3.41\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 0.7698ms 0.4980ms 2.0080 KOps/s 1.9462 KOps/s $\color{#35bf28}+3.17\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 7.4627ms 0.4922ms 2.0318 KOps/s 2.0203 KOps/s $\color{#35bf28}+0.57\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.7452ms 5.0452ms 198.2098 Ops/s 190.4746 Ops/s $\color{#35bf28}+4.06\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 1.4166ms 0.6577ms 1.5206 KOps/s 1.5048 KOps/s $\color{#35bf28}+1.05\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.8762ms 0.6261ms 1.5971 KOps/s 1.5678 KOps/s $\color{#35bf28}+1.87\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 5.5549ms 4.2057ms 237.7740 Ops/s 238.3864 Ops/s $\color{#d91a1a}-0.26\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 3.5151ms 2.3019ms 434.4151 Ops/s 427.5096 Ops/s $\color{#35bf28}+1.62\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 7.4952ms 1.4270ms 700.7497 Ops/s 832.0609 Ops/s $\textbf{\color{#d91a1a}-15.78\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.4475s 13.1046ms 76.3092 Ops/s 242.7734 Ops/s $\textbf{\color{#d91a1a}-68.57\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 8.6511ms 2.4782ms 403.5176 Ops/s 390.7195 Ops/s $\color{#35bf28}+3.28\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 5.9143ms 1.3690ms 730.4791 Ops/s 748.6241 Ops/s $\color{#d91a1a}-2.42\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 6.0365ms 4.4198ms 226.2524 Ops/s 32.2560 Ops/s $\textbf{\color{#35bf28}+601.43\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 8.9678ms 2.6586ms 376.1426 Ops/s 409.1260 Ops/s $\textbf{\color{#d91a1a}-8.06\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 4.6157ms 1.5669ms 638.1979 Ops/s 639.1990 Ops/s $\color{#d91a1a}-0.16\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 11.7853ms 10.9362ms 91.4397 Ops/s 84.5348 Ops/s $\textbf{\color{#35bf28}+8.17\%}$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 15.7259ms 14.2370ms 70.2396 Ops/s 67.8126 Ops/s $\color{#35bf28}+3.58\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 21.2796ms 19.6486ms 50.8941 Ops/s 48.6285 Ops/s $\color{#35bf28}+4.66\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 15.0668ms 14.4446ms 69.2299 Ops/s 66.9044 Ops/s $\color{#35bf28}+3.48\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 20.8490ms 20.1344ms 49.6662 Ops/s 49.2110 Ops/s $\color{#35bf28}+0.93\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 16.9613ms 15.6851ms 63.7547 Ops/s 62.3551 Ops/s $\color{#35bf28}+2.24\%$

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 6, 2024 •
edited
Loading

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 149. Improved: $\large\color{#35bf28}9$. Worsened: $\large\color{#d91a1a}14$.

Expand to view detailed results
Name Max Mean Ops Ops on Repo HEAD Change
test_simple 0.7430s 0.7422s 1.3474 Ops/s 1.3039 Ops/s $\color{#35bf28}+3.33\%$
test_transformed 1.0948s 1.0160s 0.9843 Ops/s 1.0081 Ops/s $\color{#d91a1a}-2.37\%$
test_serial 2.2277s 2.1477s 0.4656 Ops/s 0.4701 Ops/s $\color{#d91a1a}-0.95\%$
test_parallel 2.0513s 1.9904s 0.5024 Ops/s 0.5072 Ops/s $\color{#d91a1a}-0.95\%$
test_step_mdp_speed[True-True-True-True-True] 0.2251ms 40.0147μs 24.9908 KOps/s 25.0630 KOps/s $\color{#d91a1a}-0.29\%$
test_step_mdp_speed[True-True-True-True-False] 67.8910μs 22.4492μs 44.5450 KOps/s 43.8901 KOps/s $\color{#35bf28}+1.49\%$
test_step_mdp_speed[True-True-True-False-True] 48.1600μs 22.2996μs 44.8439 KOps/s 45.6735 KOps/s $\color{#d91a1a}-1.82\%$
test_step_mdp_speed[True-True-True-False-False] 45.4410μs 12.8424μs 77.8670 KOps/s 77.0947 KOps/s $\color{#35bf28}+1.00\%$
test_step_mdp_speed[True-True-False-True-True] 69.3910μs 42.4092μs 23.5798 KOps/s 23.7043 KOps/s $\color{#d91a1a}-0.53\%$
test_step_mdp_speed[True-True-False-True-False] 56.5510μs 24.9912μs 40.0142 KOps/s 41.3666 KOps/s $\color{#d91a1a}-3.27\%$
test_step_mdp_speed[True-True-False-False-True] 49.1710μs 24.2151μs 41.2965 KOps/s 41.3027 KOps/s $\color{#d91a1a}-0.01\%$
test_step_mdp_speed[True-True-False-False-False] 40.6310μs 14.9223μs 67.0138 KOps/s 67.1168 KOps/s $\color{#d91a1a}-0.15\%$
test_step_mdp_speed[True-False-True-True-True] 72.6910μs 44.9587μs 22.2426 KOps/s 22.8019 KOps/s $\color{#d91a1a}-2.45\%$
test_step_mdp_speed[True-False-True-True-False] 60.4210μs 27.3833μs 36.5186 KOps/s 36.9639 KOps/s $\color{#d91a1a}-1.20\%$
test_step_mdp_speed[True-False-True-False-True] 55.8010μs 24.1476μs 41.4120 KOps/s 41.0511 KOps/s $\color{#35bf28}+0.88\%$
test_step_mdp_speed[True-False-True-False-False] 40.1300μs 14.9610μs 66.8404 KOps/s 66.5780 KOps/s $\color{#35bf28}+0.39\%$
test_step_mdp_speed[True-False-False-True-True] 73.6010μs 46.7965μs 21.3691 KOps/s 21.6576 KOps/s $\color{#d91a1a}-1.33\%$
test_step_mdp_speed[True-False-False-True-False] 53.2610μs 29.4080μs 34.0044 KOps/s 34.1865 KOps/s $\color{#d91a1a}-0.53\%$
test_step_mdp_speed[True-False-False-False-True] 54.0510μs 25.8565μs 38.6750 KOps/s 38.1161 KOps/s $\color{#35bf28}+1.47\%$
test_step_mdp_speed[True-False-False-False-False] 51.9010μs 17.1502μs 58.3084 KOps/s 58.2225 KOps/s $\color{#35bf28}+0.15\%$
test_step_mdp_speed[False-True-True-True-True] 83.3520μs 45.2344μs 22.1071 KOps/s 22.6130 KOps/s $\color{#d91a1a}-2.24\%$
test_step_mdp_speed[False-True-True-True-False] 59.0110μs 27.6759μs 36.1325 KOps/s 36.3113 KOps/s $\color{#d91a1a}-0.49\%$
test_step_mdp_speed[False-True-True-False-True] 58.2510μs 28.1352μs 35.5427 KOps/s 35.4618 KOps/s $\color{#35bf28}+0.23\%$
test_step_mdp_speed[False-True-True-False-False] 42.0710μs 16.7250μs 59.7906 KOps/s 60.6180 KOps/s $\color{#d91a1a}-1.36\%$
test_step_mdp_speed[False-True-False-True-True] 76.7710μs 46.9093μs 21.3177 KOps/s 21.7138 KOps/s $\color{#d91a1a}-1.82\%$
test_step_mdp_speed[False-True-False-True-False] 56.5910μs 29.8698μs 33.4786 KOps/s 34.2587 KOps/s $\color{#d91a1a}-2.28\%$
test_step_mdp_speed[False-True-False-False-True] 3.1951ms 30.9281μs 32.3330 KOps/s 32.9166 KOps/s $\color{#d91a1a}-1.77\%$
test_step_mdp_speed[False-True-False-False-False] 54.9510μs 18.8733μs 52.9848 KOps/s 52.5759 KOps/s $\color{#35bf28}+0.78\%$
test_step_mdp_speed[False-False-True-True-True] 85.7810μs 49.7118μs 20.1160 KOps/s 20.8170 KOps/s $\color{#d91a1a}-3.37\%$
test_step_mdp_speed[False-False-True-True-False] 78.6310μs 30.9744μs 32.2847 KOps/s 31.7285 KOps/s $\color{#35bf28}+1.75\%$
test_step_mdp_speed[False-False-True-False-True] 60.2910μs 29.7207μs 33.6465 KOps/s 33.7036 KOps/s $\color{#d91a1a}-0.17\%$
test_step_mdp_speed[False-False-True-False-False] 45.5410μs 18.5595μs 53.8808 KOps/s 54.2509 KOps/s $\color{#d91a1a}-0.68\%$
test_step_mdp_speed[False-False-False-True-True] 89.6220μs 50.3616μs 19.8564 KOps/s 20.3507 KOps/s $\color{#d91a1a}-2.43\%$
test_step_mdp_speed[False-False-False-True-False] 68.0910μs 33.3767μs 29.9610 KOps/s 30.5997 KOps/s $\color{#d91a1a}-2.09\%$
test_step_mdp_speed[False-False-False-False-True] 63.4210μs 32.1071μs 31.1458 KOps/s 32.1544 KOps/s $\color{#d91a1a}-3.14\%$
test_step_mdp_speed[False-False-False-False-False] 50.4610μs 20.7130μs 48.2790 KOps/s 49.8644 KOps/s $\color{#d91a1a}-3.18\%$
test_values[generalized_advantage_estimate-True-True] 24.8406ms 24.2245ms 41.2806 Ops/s 39.9851 Ops/s $\color{#35bf28}+3.24\%$
test_values[vec_generalized_advantage_estimate-True-True] 0.1009s 2.9139ms 343.1798 Ops/s 328.7067 Ops/s $\color{#35bf28}+4.40\%$
test_values[td0_return_estimate-False-False] 0.1063ms 79.6719μs 12.5515 KOps/s 12.6193 KOps/s $\color{#d91a1a}-0.54\%$
test_values[td1_return_estimate-False-False] 54.6480ms 53.8890ms 18.5567 Ops/s 18.6061 Ops/s $\color{#d91a1a}-0.27\%$
test_values[vec_td1_return_estimate-False-False] 1.4100ms 1.0800ms 925.9433 Ops/s 931.0156 Ops/s $\color{#d91a1a}-0.54\%$
test_values[td_lambda_return_estimate-True-False] 85.9438ms 85.2791ms 11.7262 Ops/s 11.5735 Ops/s $\color{#35bf28}+1.32\%$
test_values[vec_td_lambda_return_estimate-True-False] 1.3702ms 1.0746ms 930.5513 Ops/s 931.4681 Ops/s $\color{#d91a1a}-0.10\%$
test_gae_speed[generalized_advantage_estimate-False-1-512] 24.0007ms 23.6981ms 42.1975 Ops/s 42.2094 Ops/s $\color{#d91a1a}-0.03\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512] 1.0679ms 0.7444ms 1.3433 KOps/s 1.3499 KOps/s $\color{#d91a1a}-0.49\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512] 0.8466ms 0.6644ms 1.5050 KOps/s 1.4738 KOps/s $\color{#35bf28}+2.12\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512] 1.5201ms 1.4731ms 678.8233 Ops/s 680.2461 Ops/s $\color{#d91a1a}-0.21\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512] 0.7395ms 0.6755ms 1.4803 KOps/s 1.4867 KOps/s $\color{#d91a1a}-0.43\%$
test_dqn_speed[False-None] 6.9832ms 1.5121ms 661.3490 Ops/s 673.4478 Ops/s $\color{#d91a1a}-1.80\%$
test_dqn_speed[False-backward] 2.1645ms 2.1276ms 470.0137 Ops/s 475.1923 Ops/s $\color{#d91a1a}-1.09\%$
test_dqn_speed[True-None] 0.6713ms 0.5511ms 1.8145 KOps/s 1.8794 KOps/s $\color{#d91a1a}-3.45\%$
test_dqn_speed[True-backward] 1.2697ms 1.2065ms 828.8588 Ops/s 912.5175 Ops/s $\textbf{\color{#d91a1a}-9.17\%}$
test_dqn_speed[reduce-overhead-None] 0.6763ms 0.5532ms 1.8077 KOps/s 1.8227 KOps/s $\color{#d91a1a}-0.82\%$
test_dqn_speed[reduce-overhead-backward] 1.1243ms 1.0712ms 933.5680 Ops/s 1.0350 KOps/s $\textbf{\color{#d91a1a}-9.80\%}$
test_ddpg_speed[False-None] 3.1787ms 2.8540ms 350.3906 Ops/s 351.6401 Ops/s $\color{#d91a1a}-0.36\%$
test_ddpg_speed[False-backward] 4.6501ms 4.2356ms 236.0917 Ops/s 243.2495 Ops/s $\color{#d91a1a}-2.94\%$
test_ddpg_speed[True-None] 1.3315ms 1.0728ms 932.1713 Ops/s 933.4832 Ops/s $\color{#d91a1a}-0.14\%$
test_ddpg_speed[True-backward] 2.4528ms 2.3164ms 431.7106 Ops/s 453.9212 Ops/s $\color{#d91a1a}-4.89\%$
test_ddpg_speed[reduce-overhead-None] 1.1330ms 1.0815ms 924.6214 Ops/s 909.4178 Ops/s $\color{#35bf28}+1.67\%$
test_ddpg_speed[reduce-overhead-backward] 1.8649ms 1.7866ms 559.7247 Ops/s 613.3763 Ops/s $\textbf{\color{#d91a1a}-8.75\%}$
test_sac_speed[False-None] 8.6714ms 8.0566ms 124.1220 Ops/s 125.3293 Ops/s $\color{#d91a1a}-0.96\%$
test_sac_speed[False-backward] 12.0842ms 11.3113ms 88.4071 Ops/s 91.0123 Ops/s $\color{#d91a1a}-2.86\%$
test_sac_speed[True-None] 1.6693ms 1.5482ms 645.9125 Ops/s 652.7716 Ops/s $\color{#d91a1a}-1.05\%$
test_sac_speed[True-backward] 3.5386ms 3.3798ms 295.8753 Ops/s 292.5745 Ops/s $\color{#35bf28}+1.13\%$
test_sac_speed[reduce-overhead-None] 22.4062ms 12.5118ms 79.9247 Ops/s 79.2175 Ops/s $\color{#35bf28}+0.89\%$
test_sac_speed[reduce-overhead-backward] 1.5345ms 1.4832ms 674.2370 Ops/s 700.5250 Ops/s $\color{#d91a1a}-3.75\%$
test_redq_speed[False-None] 8.3115ms 7.4871ms 133.5634 Ops/s 133.1641 Ops/s $\color{#35bf28}+0.30\%$
test_redq_speed[False-backward] 12.5494ms 11.7789ms 84.8978 Ops/s 86.3185 Ops/s $\color{#d91a1a}-1.65\%$
test_redq_speed[True-None] 2.0488ms 1.9727ms 506.9220 Ops/s 508.2243 Ops/s $\color{#d91a1a}-0.26\%$
test_redq_speed[True-backward] 3.8886ms 3.8158ms 262.0708 Ops/s 275.5546 Ops/s $\color{#d91a1a}-4.89\%$
test_redq_speed[reduce-overhead-None] 2.1020ms 1.9750ms 506.3234 Ops/s 507.3732 Ops/s $\color{#d91a1a}-0.21\%$
test_redq_speed[reduce-overhead-backward] 3.8731ms 3.8285ms 261.2002 Ops/s 273.3427 Ops/s $\color{#d91a1a}-4.44\%$
test_redq_deprec_speed[False-None] 9.5784ms 9.0270ms 110.7794 Ops/s 110.7595 Ops/s $\color{#35bf28}+0.02\%$
test_redq_deprec_speed[False-backward] 12.7642ms 12.3419ms 81.0247 Ops/s 82.7070 Ops/s $\color{#d91a1a}-2.03\%$
test_redq_deprec_speed[True-None] 2.3774ms 2.3151ms 431.9390 Ops/s 429.1121 Ops/s $\color{#35bf28}+0.66\%$
test_redq_deprec_speed[True-backward] 4.5888ms 4.1753ms 239.5054 Ops/s 235.8858 Ops/s $\color{#35bf28}+1.53\%$
test_redq_deprec_speed[reduce-overhead-None] 2.3721ms 2.3116ms 432.6024 Ops/s 430.8581 Ops/s $\color{#35bf28}+0.40\%$
test_redq_deprec_speed[reduce-overhead-backward] 4.5784ms 4.1702ms 239.7974 Ops/s 239.4400 Ops/s $\color{#35bf28}+0.15\%$
test_td3_speed[False-None] 8.0966ms 7.8895ms 126.7515 Ops/s 127.0129 Ops/s $\color{#d91a1a}-0.21\%$
test_td3_speed[False-backward] 11.0702ms 10.4906ms 95.3235 Ops/s 95.5068 Ops/s $\color{#d91a1a}-0.19\%$
test_td3_speed[True-None] 1.5953ms 1.5702ms 636.8706 Ops/s 639.0442 Ops/s $\color{#d91a1a}-0.34\%$
test_td3_speed[True-backward] 3.6469ms 3.2707ms 305.7461 Ops/s 303.7660 Ops/s $\color{#35bf28}+0.65\%$
test_td3_speed[reduce-overhead-None] 49.7005ms 25.3135ms 39.5046 Ops/s 38.1766 Ops/s $\color{#35bf28}+3.48\%$
test_td3_speed[reduce-overhead-backward] 1.4790ms 1.4264ms 701.0630 Ops/s 780.2830 Ops/s $\textbf{\color{#d91a1a}-10.15\%}$
test_cql_speed[False-None] 16.4695ms 16.1305ms 61.9943 Ops/s 62.2644 Ops/s $\color{#d91a1a}-0.43\%$
test_cql_speed[False-backward] 22.2915ms 21.8144ms 45.8413 Ops/s 46.8849 Ops/s $\color{#d91a1a}-2.23\%$
test_cql_speed[True-None] 3.0530ms 2.9133ms 343.2548 Ops/s 337.3948 Ops/s $\color{#35bf28}+1.74\%$
test_cql_speed[True-backward] 5.4989ms 5.0850ms 196.6567 Ops/s 190.4035 Ops/s $\color{#35bf28}+3.28\%$
test_cql_speed[reduce-overhead-None] 22.5722ms 13.1718ms 75.9196 Ops/s 76.0739 Ops/s $\color{#d91a1a}-0.20\%$
test_cql_speed[reduce-overhead-backward] 1.5497ms 1.4705ms 680.0607 Ops/s 601.7079 Ops/s $\textbf{\color{#35bf28}+13.02\%}$
test_a2c_speed[False-None] 3.7555ms 3.2041ms 312.1026 Ops/s 312.2737 Ops/s $\color{#d91a1a}-0.05\%$
test_a2c_speed[False-backward] 6.7184ms 6.1770ms 161.8906 Ops/s 156.1419 Ops/s $\color{#35bf28}+3.68\%$
test_a2c_speed[True-None] 1.4384ms 0.9917ms 1.0084 KOps/s 1.0010 KOps/s $\color{#35bf28}+0.73\%$
test_a2c_speed[True-backward] 2.6977ms 2.6249ms 380.9648 Ops/s 361.8923 Ops/s $\textbf{\color{#35bf28}+5.27\%}$
test_a2c_speed[reduce-overhead-None] 0.3980s 12.4231ms 80.4951 Ops/s 87.4308 Ops/s $\textbf{\color{#d91a1a}-7.93\%}$
test_a2c_speed[reduce-overhead-backward] 1.0485ms 0.9889ms 1.0112 KOps/s 879.5270 Ops/s $\textbf{\color{#35bf28}+14.97\%}$
test_ppo_speed[False-None] 3.8412ms 3.6643ms 272.9041 Ops/s 271.7833 Ops/s $\color{#35bf28}+0.41\%$
test_ppo_speed[False-backward] 7.3205ms 6.8742ms 145.4716 Ops/s 141.1564 Ops/s $\color{#35bf28}+3.06\%$
test_ppo_speed[True-None] 1.0628ms 0.9286ms 1.0769 KOps/s 1.0532 KOps/s $\color{#35bf28}+2.25\%$
test_ppo_speed[True-backward] 2.6626ms 2.5714ms 388.8941 Ops/s 390.7025 Ops/s $\color{#d91a1a}-0.46\%$
test_ppo_speed[reduce-overhead-None] 0.5370ms 0.4777ms 2.0933 KOps/s 1.9238 KOps/s $\textbf{\color{#35bf28}+8.81\%}$
test_ppo_speed[reduce-overhead-backward] 0.9979ms 0.9648ms 1.0364 KOps/s 1.0133 KOps/s $\color{#35bf28}+2.29\%$
test_reinforce_speed[False-None] 2.6529ms 2.2918ms 436.3369 Ops/s 441.5218 Ops/s $\color{#d91a1a}-1.17\%$
test_reinforce_speed[False-backward] 3.8104ms 3.3355ms 299.8039 Ops/s 305.6442 Ops/s $\color{#d91a1a}-1.91\%$
test_reinforce_speed[True-None] 0.9215ms 0.8351ms 1.1975 KOps/s 1.2091 KOps/s $\color{#d91a1a}-0.96\%$
test_reinforce_speed[True-backward] 2.4866ms 2.4255ms 412.2942 Ops/s 414.5553 Ops/s $\color{#d91a1a}-0.55\%$
test_reinforce_speed[reduce-overhead-None] 22.3143ms 11.7626ms 85.0155 Ops/s 87.6947 Ops/s $\color{#d91a1a}-3.06\%$
test_reinforce_speed[reduce-overhead-backward] 1.1011ms 1.0422ms 959.5202 Ops/s 951.5767 Ops/s $\color{#35bf28}+0.83\%$
test_iql_speed[False-None] 9.6973ms 9.2281ms 108.3652 Ops/s 108.8553 Ops/s $\color{#d91a1a}-0.45\%$
test_iql_speed[False-backward] 13.5777ms 13.0309ms 76.7409 Ops/s 77.1290 Ops/s $\color{#d91a1a}-0.50\%$
test_iql_speed[True-None] 1.8884ms 1.7542ms 570.0738 Ops/s 578.9852 Ops/s $\color{#d91a1a}-1.54\%$
test_iql_speed[True-backward] 4.8476ms 4.4095ms 226.7830 Ops/s 236.8013 Ops/s $\color{#d91a1a}-4.23\%$
test_iql_speed[reduce-overhead-None] 21.0989ms 11.5382ms 86.6686 Ops/s 111.2333 Ops/s $\textbf{\color{#d91a1a}-22.08\%}$
test_iql_speed[reduce-overhead-backward] 1.6483ms 1.5857ms 630.6191 Ops/s 638.5196 Ops/s $\color{#d91a1a}-1.24\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 8.0395ms 6.4870ms 154.1544 Ops/s 153.0570 Ops/s $\color{#35bf28}+0.72\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 0.5888ms 0.3438ms 2.9086 KOps/s 3.6568 KOps/s $\textbf{\color{#d91a1a}-20.46\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5223ms 0.3253ms 3.0744 KOps/s 3.8900 KOps/s $\textbf{\color{#d91a1a}-20.97\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.4443ms 6.2163ms 160.8687 Ops/s 159.5189 Ops/s $\color{#35bf28}+0.85\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.1847ms 0.2856ms 3.5019 KOps/s 3.4172 KOps/s $\color{#35bf28}+2.48\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5799ms 0.2800ms 3.5714 KOps/s 4.1182 KOps/s $\textbf{\color{#d91a1a}-13.28\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000] 1.4391ms 1.2524ms 798.4890 Ops/s 795.2762 Ops/s $\color{#35bf28}+0.40\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000] 1.4129ms 1.2080ms 827.7945 Ops/s 835.5045 Ops/s $\color{#d91a1a}-0.92\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.6939ms 6.4968ms 153.9211 Ops/s 155.7746 Ops/s $\color{#d91a1a}-1.19\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 2.1081ms 0.4805ms 2.0814 KOps/s 2.3935 KOps/s $\textbf{\color{#d91a1a}-13.04\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.6977ms 0.4491ms 2.2265 KOps/s 2.1300 KOps/s $\color{#35bf28}+4.53\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000] 6.4249ms 6.3052ms 158.5985 Ops/s 160.6960 Ops/s $\color{#d91a1a}-1.31\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000] 1.9417ms 0.3901ms 2.5633 KOps/s 2.7931 KOps/s $\textbf{\color{#d91a1a}-8.23\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000] 0.5310ms 0.3493ms 2.8630 KOps/s 2.9411 KOps/s $\color{#d91a1a}-2.66\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000] 6.4877ms 6.2310ms 160.4890 Ops/s 161.2584 Ops/s $\color{#d91a1a}-0.48\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000] 1.4467ms 0.2858ms 3.4991 KOps/s 3.1462 KOps/s $\textbf{\color{#35bf28}+11.22\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000] 0.5087ms 0.2780ms 3.5967 KOps/s 3.7206 KOps/s $\color{#d91a1a}-3.33\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000] 6.6898ms 6.4712ms 154.5310 Ops/s 156.6811 Ops/s $\color{#d91a1a}-1.37\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000] 0.7659ms 0.4129ms 2.4218 KOps/s 2.3972 KOps/s $\color{#35bf28}+1.03\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000] 0.5410ms 0.3870ms 2.5837 KOps/s 2.5646 KOps/s $\color{#35bf28}+0.75\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400] 6.9909ms 5.3989ms 185.2222 Ops/s 190.6392 Ops/s $\color{#d91a1a}-2.84\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400] 6.2342ms 2.0379ms 490.7126 Ops/s 448.0125 Ops/s $\textbf{\color{#35bf28}+9.53\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400] 9.1248ms 1.2991ms 769.7661 Ops/s 814.4890 Ops/s $\textbf{\color{#d91a1a}-5.49\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400] 0.4877s 15.0814ms 66.3067 Ops/s 189.4513 Ops/s $\textbf{\color{#d91a1a}-65.00\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400] 9.3201ms 1.9822ms 504.4979 Ops/s 449.9380 Ops/s $\textbf{\color{#35bf28}+12.13\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400] 7.8368ms 1.2153ms 822.8194 Ops/s 790.3962 Ops/s $\color{#35bf28}+4.10\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400] 7.7452ms 5.7114ms 175.0889 Ops/s 33.3714 Ops/s $\textbf{\color{#35bf28}+424.67\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400] 8.8043ms 2.2076ms 452.9808 Ops/s 518.4611 Ops/s $\textbf{\color{#d91a1a}-12.63\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400] 7.5353ms 1.5036ms 665.0704 Ops/s 692.5805 Ops/s $\color{#d91a1a}-3.97\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True] 13.4716ms 12.9910ms 76.9762 Ops/s 75.5348 Ops/s $\color{#35bf28}+1.91\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False] 20.9638ms 16.5199ms 60.5329 Ops/s 59.3201 Ops/s $\color{#35bf28}+2.04\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True] 21.9281ms 17.7712ms 56.2708 Ops/s 55.5792 Ops/s $\color{#35bf28}+1.24\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False] 16.8116ms 16.5405ms 60.4578 Ops/s 57.7085 Ops/s $\color{#35bf28}+4.76\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True] 17.8399ms 17.6043ms 56.8044 Ops/s 56.9292 Ops/s $\color{#d91a1a}-0.22\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False] 18.2384ms 17.6590ms 56.6285 Ops/s 53.4169 Ops/s $\textbf{\color{#35bf28}+6.01\%}$

@vmoens
Update
968f774 
[ghstack-poisoned]
@vmoens
Update
b6d765e 
[ghstack-poisoned]
@vmoens
Update
0f67077 
[ghstack-poisoned]
@vmoens vmoens merged commit 0f67077 into gh/vmoens/54/base Dec 12, 2024
1 check passed
vmoens added a commit that referenced this pull request Dec 12, 2024
@vmoens
[Test] More comprehensive tests for auto_spec
6c7d233 
ghstack-source-id: 75352490436fd706af3d36f9b8016e80a8a3f46a
Pull Request resolved: #2640
@vmoens vmoens deleted the gh/vmoens/54/head branch December 12, 2024 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@vmoens @facebook-github-bot