[Feature] PPO Trainer Updates

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] Documentation #3192
• [Feature] SAC Trainer #3191
• -> [Feature] PPO Trainer Updates #3190

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3190

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 12 New Failures, 1 Unrelated Failure

As of commit 70b82ac with merge base 3d1748f ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 139.
• Habitat Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t b60b47c8f423704618bb1581e95ec35cd1fb81eef40960132bb0a0afa982c392 /exec failed with exit code 1
• Libs Tests on Linux / unittests-gym (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t fc06ebc9e0f0f6c18c4ae2870e1521432e51e3718a2ae87e82417596b8c2afae /exec failed with exit code 1
• Push Binary Nightly / build-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / build-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
• Push Binary Nightly / build-wheel-windows (3.9, 3.9) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-unix (linux, ubuntu-22.04, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-unix (macos, macos-latest, 3.9, cp39-cp39, cpu, cpu) (gh)
  Process completed with exit code 1.
• Push Binary Nightly / test-wheel-windows (3.9, 3.9) (gh)
  Process completed with exit code 1.
• SOTA Tests on Linux / tests (3.9, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 2fe1df1d1ad7f0f2a32243dd2a21da9f80b0e6075bb6bb88bbb9e00a291cfaf3 /exec failed with exit code 1
• Unit-tests on Linux / tests-gpu (3.11, 12.8) / linux-job (gh)
  test/test_collector.py::TestPolicyFactory::test_weight_update
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_collector.py::TestPolicyFactory::test_weight_update

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-olddeps (3.9, 11.6) / linux-job (gh) (trunk failure)
  test/test_collector.py::TestPolicyFactory::test_weight_update

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 154. Improved: $\large\color{#35bf28}20$. Worsened: $\large\color{#d91a1a}11$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_tensor_to_bytestream_speed[pickle]	83.6883μs	82.6439μs	12.1001 KOps/s	11.8379 KOps/s	$\color{#35bf28}+2.22\%$
test_tensor_to_bytestream_speed[torch.save]	0.1437ms	0.1431ms	6.9869 KOps/s	7.0507 KOps/s	$\color{#d91a1a}-0.90\%$
test_tensor_to_bytestream_speed[untyped_storage]	0.1217s	0.1207s	8.2835 Ops/s	8.1339 Ops/s	$\color{#35bf28}+1.84\%$
test_tensor_to_bytestream_speed[numpy]	2.8203μs	2.8112μs	355.7261 KOps/s	355.0445 KOps/s	$\color{#35bf28}+0.19\%$
test_tensor_to_bytestream_speed[safetensors]	45.1208μs	44.2957μs	22.5755 KOps/s	22.5187 KOps/s	$\color{#35bf28}+0.25\%$
test_simple	0.5700s	0.5619s	1.7796 Ops/s	1.7259 Ops/s	$\color{#35bf28}+3.11\%$
test_transformed	1.2475s	1.1552s	0.8657 Ops/s	0.8704 Ops/s	$\color{#d91a1a}-0.54\%$
test_serial	1.6976s	1.6908s	0.5914 Ops/s	0.5869 Ops/s	$\color{#35bf28}+0.78\%$
test_parallel	1.1153s	1.0953s	0.9130 Ops/s	0.9211 Ops/s	$\color{#d91a1a}-0.88\%$
test_step_mdp_speed[True-True-True-True-True]	0.2204ms	45.9020μs	21.7855 KOps/s	21.9414 KOps/s	$\color{#d91a1a}-0.71\%$
test_step_mdp_speed[True-True-True-True-False]	2.7121ms	25.5646μs	39.1165 KOps/s	38.1558 KOps/s	$\color{#35bf28}+2.52\%$
test_step_mdp_speed[True-True-True-False-True]	0.1338ms	25.8423μs	38.6962 KOps/s	38.6197 KOps/s	$\color{#35bf28}+0.20\%$
test_step_mdp_speed[True-True-True-False-False]	36.7700μs	14.1680μs	70.5817 KOps/s	70.5144 KOps/s	$\color{#35bf28}+0.10\%$
test_step_mdp_speed[True-True-False-True-True]	94.6710μs	48.4122μs	20.6559 KOps/s	20.6982 KOps/s	$\color{#d91a1a}-0.20\%$
test_step_mdp_speed[True-True-False-True-False]	68.1710μs	28.5655μs	35.0073 KOps/s	35.2924 KOps/s	$\color{#d91a1a}-0.81\%$
test_step_mdp_speed[True-True-False-False-True]	58.6710μs	28.7208μs	34.8179 KOps/s	35.1647 KOps/s	$\color{#d91a1a}-0.99\%$
test_step_mdp_speed[True-True-False-False-False]	96.2310μs	17.5387μs	57.0168 KOps/s	58.9067 KOps/s	$\color{#d91a1a}-3.21\%$
test_step_mdp_speed[True-False-True-True-True]	79.3400μs	51.5736μs	19.3898 KOps/s	19.1495 KOps/s	$\color{#35bf28}+1.25\%$
test_step_mdp_speed[True-False-True-True-False]	65.9510μs	31.0267μs	32.2304 KOps/s	32.1421 KOps/s	$\color{#35bf28}+0.27\%$
test_step_mdp_speed[True-False-True-False-True]	59.1810μs	28.6798μs	34.8678 KOps/s	34.3356 KOps/s	$\color{#35bf28}+1.55\%$
test_step_mdp_speed[True-False-True-False-False]	47.7710μs	17.3096μs	57.7713 KOps/s	58.7866 KOps/s	$\color{#d91a1a}-1.73\%$
test_step_mdp_speed[True-False-False-True-True]	91.9200μs	54.7602μs	18.2614 KOps/s	18.1277 KOps/s	$\color{#35bf28}+0.74\%$
test_step_mdp_speed[True-False-False-True-False]	90.5410μs	34.3047μs	29.1505 KOps/s	29.6354 KOps/s	$\color{#d91a1a}-1.64\%$
test_step_mdp_speed[True-False-False-False-True]	69.9010μs	31.6999μs	31.5459 KOps/s	32.3734 KOps/s	$\color{#d91a1a}-2.56\%$
test_step_mdp_speed[True-False-False-False-False]	54.4600μs	19.9188μs	50.2038 KOps/s	51.1093 KOps/s	$\color{#d91a1a}-1.77\%$
test_step_mdp_speed[False-True-True-True-True]	0.1048ms	51.9782μs	19.2389 KOps/s	19.3878 KOps/s	$\color{#d91a1a}-0.77\%$
test_step_mdp_speed[False-True-True-True-False]	69.0800μs	32.1806μs	31.0746 KOps/s	32.3937 KOps/s	$\color{#d91a1a}-4.07\%$
test_step_mdp_speed[False-True-True-False-True]	2.3386ms	33.1286μs	30.1854 KOps/s	30.5041 KOps/s	$\color{#d91a1a}-1.04\%$
test_step_mdp_speed[False-True-True-False-False]	51.1600μs	19.0862μs	52.3939 KOps/s	53.6206 KOps/s	$\color{#d91a1a}-2.29\%$
test_step_mdp_speed[False-True-False-True-True]	95.8910μs	54.3499μs	18.3993 KOps/s	18.5136 KOps/s	$\color{#d91a1a}-0.62\%$
test_step_mdp_speed[False-True-False-True-False]	68.6500μs	33.5649μs	29.7930 KOps/s	29.1961 KOps/s	$\color{#35bf28}+2.04\%$
test_step_mdp_speed[False-True-False-False-True]	78.3510μs	35.8399μs	27.9018 KOps/s	28.4840 KOps/s	$\color{#d91a1a}-2.04\%$
test_step_mdp_speed[False-True-False-False-False]	55.7910μs	22.0316μs	45.3893 KOps/s	46.7855 KOps/s	$\color{#d91a1a}-2.98\%$
test_step_mdp_speed[False-False-True-True-True]	95.8700μs	56.7155μs	17.6318 KOps/s	17.5664 KOps/s	$\color{#35bf28}+0.37\%$
test_step_mdp_speed[False-False-True-True-False]	68.5110μs	36.9744μs	27.0457 KOps/s	27.1477 KOps/s	$\color{#d91a1a}-0.38\%$
test_step_mdp_speed[False-False-True-False-True]	75.1300μs	35.2956μs	28.3322 KOps/s	28.1739 KOps/s	$\color{#35bf28}+0.56\%$
test_step_mdp_speed[False-False-True-False-False]	55.3410μs	21.7782μs	45.9175 KOps/s	46.1447 KOps/s	$\color{#d91a1a}-0.49\%$
test_step_mdp_speed[False-False-False-True-True]	0.1334ms	59.1435μs	16.9080 KOps/s	16.9280 KOps/s	$\color{#d91a1a}-0.12\%$
test_step_mdp_speed[False-False-False-True-False]	0.1058ms	39.3993μs	25.3812 KOps/s	25.5655 KOps/s	$\color{#d91a1a}-0.72\%$
test_step_mdp_speed[False-False-False-False-True]	79.7400μs	38.0627μs	26.2725 KOps/s	25.8214 KOps/s	$\color{#35bf28}+1.75\%$
test_step_mdp_speed[False-False-False-False-False]	56.7010μs	24.3568μs	41.0563 KOps/s	41.7349 KOps/s	$\color{#d91a1a}-1.63\%$
test_values[generalized_advantage_estimate-True-True]	11.3629ms	10.4197ms	95.9720 Ops/s	100.0800 Ops/s	$\color{#d91a1a}-4.10\%$
test_values[vec_generalized_advantage_estimate-True-True]	19.6338ms	17.6047ms	56.8031 Ops/s	59.0323 Ops/s	$\color{#d91a1a}-3.78\%$
test_values[td0_return_estimate-False-False]	0.2211ms	0.1308ms	7.6466 KOps/s	8.1513 KOps/s	$\textbf{\color{#d91a1a}-6.19\%}$
test_values[td1_return_estimate-False-False]	28.0299ms	27.5105ms	36.3498 Ops/s	36.4356 Ops/s	$\color{#d91a1a}-0.24\%$
test_values[vec_td1_return_estimate-False-False]	19.0013ms	17.6920ms	56.5227 Ops/s	67.6499 Ops/s	$\textbf{\color{#d91a1a}-16.45\%}$
test_values[td_lambda_return_estimate-True-False]	41.4197ms	40.9815ms	24.4012 Ops/s	24.3232 Ops/s	$\color{#35bf28}+0.32\%$
test_values[vec_td_lambda_return_estimate-True-False]	17.9618ms	17.6537ms	56.6455 Ops/s	72.9590 Ops/s	$\textbf{\color{#d91a1a}-22.36\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.9752ms	8.7889ms	113.7802 Ops/s	114.4180 Ops/s	$\color{#d91a1a}-0.56\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.8248ms	1.4983ms	667.4181 Ops/s	670.1406 Ops/s	$\color{#d91a1a}-0.41\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.5001ms	0.4279ms	2.3369 KOps/s	2.3985 KOps/s	$\color{#d91a1a}-2.57\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	34.6087ms	33.9523ms	29.4531 Ops/s	31.9882 Ops/s	$\textbf{\color{#d91a1a}-7.93\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	1.8279ms	1.7245ms	579.8624 Ops/s	582.8449 Ops/s	$\color{#d91a1a}-0.51\%$
test_dqn_speed[False-None]	6.3766ms	1.4309ms	698.8515 Ops/s	698.3552 Ops/s	$\color{#35bf28}+0.07\%$
test_dqn_speed[False-backward]	2.1004ms	1.9590ms	510.4695 Ops/s	514.2518 Ops/s	$\color{#d91a1a}-0.74\%$
test_dqn_speed[True-None]	0.7344ms	0.5074ms	1.9708 KOps/s	1.8753 KOps/s	$\textbf{\color{#35bf28}+5.09\%}$
test_dqn_speed[True-backward]	1.1079ms	0.9636ms	1.0378 KOps/s	907.9464 Ops/s	$\textbf{\color{#35bf28}+14.30\%}$
test_dqn_speed[reduce-overhead-None]	0.9387ms	0.5307ms	1.8843 KOps/s	1.9013 KOps/s	$\color{#d91a1a}-0.89\%$
test_dqn_speed[reduce-overhead-backward]	0.9940ms	0.9597ms	1.0420 KOps/s	1.0409 KOps/s	$\color{#35bf28}+0.10\%$
test_ddpg_speed[False-None]	3.2744ms	2.9105ms	343.5818 Ops/s	346.2821 Ops/s	$\color{#d91a1a}-0.78\%$
test_ddpg_speed[False-backward]	4.2481ms	4.1219ms	242.6061 Ops/s	242.7569 Ops/s	$\color{#d91a1a}-0.06\%$
test_ddpg_speed[True-None]	1.7425ms	1.3690ms	730.4741 Ops/s	717.3474 Ops/s	$\color{#35bf28}+1.83\%$
test_ddpg_speed[True-backward]	2.4120ms	2.3565ms	424.3615 Ops/s	405.0269 Ops/s	$\color{#35bf28}+4.77\%$
test_ddpg_speed[reduce-overhead-None]	1.6380ms	1.3707ms	729.5612 Ops/s	729.5401 Ops/s	$+0.00\%$
test_ddpg_speed[reduce-overhead-backward]	2.4565ms	2.3297ms	429.2315 Ops/s	415.9784 Ops/s	$\color{#35bf28}+3.19\%$
test_sac_speed[False-None]	8.3773ms	7.9494ms	125.7959 Ops/s	123.9942 Ops/s	$\color{#35bf28}+1.45\%$
test_sac_speed[False-backward]	11.7382ms	11.2339ms	89.0161 Ops/s	88.7123 Ops/s	$\color{#35bf28}+0.34\%$
test_sac_speed[True-None]	2.4534ms	2.0713ms	482.7817 Ops/s	473.5930 Ops/s	$\color{#35bf28}+1.94\%$
test_sac_speed[True-backward]	4.1221ms	4.0128ms	249.2017 Ops/s	237.6864 Ops/s	$\color{#35bf28}+4.84\%$
test_sac_speed[reduce-overhead-None]	2.4176ms	2.0868ms	479.1914 Ops/s	471.4835 Ops/s	$\color{#35bf28}+1.63\%$
test_sac_speed[reduce-overhead-backward]	5.9518ms	4.4043ms	227.0523 Ops/s	244.9828 Ops/s	$\textbf{\color{#d91a1a}-7.32\%}$
test_redq_speed[False-None]	13.4350ms	10.5135ms	95.1159 Ops/s	89.1657 Ops/s	$\textbf{\color{#35bf28}+6.67\%}$
test_redq_speed[False-backward]	18.4573ms	17.9351ms	55.7565 Ops/s	54.7554 Ops/s	$\color{#35bf28}+1.83\%$
test_redq_speed[True-None]	4.5477ms	4.3010ms	232.5038 Ops/s	227.2977 Ops/s	$\color{#35bf28}+2.29\%$
test_redq_speed[True-backward]	12.7001ms	10.2381ms	97.6740 Ops/s	100.5498 Ops/s	$\color{#d91a1a}-2.86\%$
test_redq_speed[reduce-overhead-None]	4.8356ms	4.2912ms	233.0335 Ops/s	221.0242 Ops/s	$\textbf{\color{#35bf28}+5.43\%}$
test_redq_speed[reduce-overhead-backward]	10.2584ms	9.9285ms	100.7205 Ops/s	102.7095 Ops/s	$\color{#d91a1a}-1.94\%$
test_redq_deprec_speed[False-None]	11.7119ms	11.1527ms	89.6647 Ops/s	88.4162 Ops/s	$\color{#35bf28}+1.41\%$
test_redq_deprec_speed[False-backward]	16.3296ms	15.9861ms	62.5543 Ops/s	61.0574 Ops/s	$\color{#35bf28}+2.45\%$
test_redq_deprec_speed[True-None]	3.8971ms	3.5478ms	281.8671 Ops/s	281.0559 Ops/s	$\color{#35bf28}+0.29\%$
test_redq_deprec_speed[True-backward]	7.6758ms	7.4748ms	133.7835 Ops/s	137.3818 Ops/s	$\color{#d91a1a}-2.62\%$
test_redq_deprec_speed[reduce-overhead-None]	3.8752ms	3.4737ms	287.8803 Ops/s	262.3511 Ops/s	$\textbf{\color{#35bf28}+9.73\%}$
test_redq_deprec_speed[reduce-overhead-backward]	7.7459ms	7.5142ms	133.0817 Ops/s	127.5223 Ops/s	$\color{#35bf28}+4.36\%$
test_td3_speed[False-None]	8.2684ms	8.0192ms	124.7013 Ops/s	118.6900 Ops/s	$\textbf{\color{#35bf28}+5.06\%}$
test_td3_speed[False-backward]	11.4516ms	10.9336ms	91.4611 Ops/s	90.2885 Ops/s	$\color{#35bf28}+1.30\%$
test_td3_speed[True-None]	1.8278ms	1.7920ms	558.0316 Ops/s	565.2802 Ops/s	$\color{#d91a1a}-1.28\%$
test_td3_speed[True-backward]	3.6824ms	3.5556ms	281.2444 Ops/s	276.6413 Ops/s	$\color{#35bf28}+1.66\%$
test_td3_speed[reduce-overhead-None]	1.7766ms	1.7415ms	574.2100 Ops/s	566.4451 Ops/s	$\color{#35bf28}+1.37\%$
test_td3_speed[reduce-overhead-backward]	3.6395ms	3.5280ms	283.4487 Ops/s	275.7480 Ops/s	$\color{#35bf28}+2.79\%$
test_cql_speed[False-None]	28.6979ms	26.0956ms	38.3207 Ops/s	38.5476 Ops/s	$\color{#d91a1a}-0.59\%$
test_cql_speed[False-backward]	39.3687ms	35.4998ms	28.1691 Ops/s	28.0765 Ops/s	$\color{#35bf28}+0.33\%$
test_cql_speed[True-None]	15.3439ms	12.4761ms	80.1532 Ops/s	79.8585 Ops/s	$\color{#35bf28}+0.37\%$
test_cql_speed[True-backward]	19.4768ms	18.4475ms	54.2080 Ops/s	54.9461 Ops/s	$\color{#d91a1a}-1.34\%$
test_cql_speed[reduce-overhead-None]	15.4969ms	12.4847ms	80.0979 Ops/s	79.9659 Ops/s	$\color{#35bf28}+0.17\%$
test_cql_speed[reduce-overhead-backward]	18.8534ms	18.4525ms	54.1931 Ops/s	57.0810 Ops/s	$\textbf{\color{#d91a1a}-5.06\%}$
test_a2c_speed[False-None]	5.6604ms	5.4534ms	183.3734 Ops/s	184.9723 Ops/s	$\color{#d91a1a}-0.86\%$
test_a2c_speed[False-backward]	12.1802ms	11.8814ms	84.1655 Ops/s	84.1169 Ops/s	$\color{#35bf28}+0.06\%$
test_a2c_speed[True-None]	3.9032ms	3.6964ms	270.5358 Ops/s	285.6193 Ops/s	$\textbf{\color{#d91a1a}-5.28\%}$
test_a2c_speed[True-backward]	8.7263ms	8.4552ms	118.2707 Ops/s	110.6590 Ops/s	$\textbf{\color{#35bf28}+6.88\%}$
test_a2c_speed[reduce-overhead-None]	4.0074ms	3.7007ms	270.2227 Ops/s	270.0261 Ops/s	$\color{#35bf28}+0.07\%$
test_a2c_speed[reduce-overhead-backward]	9.0871ms	8.8351ms	113.1845 Ops/s	105.9593 Ops/s	$\textbf{\color{#35bf28}+6.82\%}$
test_ppo_speed[False-None]	6.3432ms	5.9415ms	168.3083 Ops/s	168.5427 Ops/s	$\color{#d91a1a}-0.14\%$
test_ppo_speed[False-backward]	13.0540ms	12.5788ms	79.4988 Ops/s	80.1079 Ops/s	$\color{#d91a1a}-0.76\%$
test_ppo_speed[True-None]	3.9692ms	3.6212ms	276.1517 Ops/s	261.6567 Ops/s	$\textbf{\color{#35bf28}+5.54\%}$
test_ppo_speed[True-backward]	8.8564ms	8.4382ms	118.5091 Ops/s	118.0066 Ops/s	$\color{#35bf28}+0.43\%$
test_ppo_speed[reduce-overhead-None]	3.8394ms	3.6424ms	274.5422 Ops/s	273.2050 Ops/s	$\color{#35bf28}+0.49\%$
test_ppo_speed[reduce-overhead-backward]	10.4477ms	9.0066ms	111.0298 Ops/s	113.8409 Ops/s	$\color{#d91a1a}-2.47\%$
test_reinforce_speed[False-None]	5.1302ms	4.6538ms	214.8777 Ops/s	216.7507 Ops/s	$\color{#d91a1a}-0.86\%$
test_reinforce_speed[False-backward]	7.8710ms	7.3857ms	135.3975 Ops/s	134.0339 Ops/s	$\color{#35bf28}+1.02\%$
test_reinforce_speed[True-None]	3.0266ms	2.8588ms	349.7933 Ops/s	345.9371 Ops/s	$\color{#35bf28}+1.11\%$
test_reinforce_speed[True-backward]	7.8430ms	7.6205ms	131.2251 Ops/s	124.9818 Ops/s	$\color{#35bf28}+5.00\%$
test_reinforce_speed[reduce-overhead-None]	3.2820ms	2.8255ms	353.9148 Ops/s	350.5493 Ops/s	$\color{#35bf28}+0.96\%$
test_reinforce_speed[reduce-overhead-backward]	8.1225ms	7.8522ms	127.3523 Ops/s	125.5658 Ops/s	$\color{#35bf28}+1.42\%$
test_iql_speed[False-None]	25.6075ms	20.7291ms	48.2414 Ops/s	49.5532 Ops/s	$\color{#d91a1a}-2.65\%$
test_iql_speed[False-backward]	31.2930ms	30.3433ms	32.9562 Ops/s	32.3918 Ops/s	$\color{#35bf28}+1.74\%$
test_iql_speed[True-None]	8.8198ms	8.4218ms	118.7389 Ops/s	118.6283 Ops/s	$\color{#35bf28}+0.09\%$
test_iql_speed[True-backward]	17.1051ms	16.6109ms	60.2013 Ops/s	60.5288 Ops/s	$\color{#d91a1a}-0.54\%$
test_iql_speed[reduce-overhead-None]	8.7449ms	8.4440ms	118.4276 Ops/s	113.5303 Ops/s	$\color{#35bf28}+4.31\%$
test_iql_speed[reduce-overhead-backward]	17.7673ms	17.2228ms	58.0626 Ops/s	63.4426 Ops/s	$\textbf{\color{#d91a1a}-8.48\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.6364ms	6.2496ms	160.0096 Ops/s	163.4262 Ops/s	$\color{#d91a1a}-2.09\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5373ms	0.2797ms	3.5752 KOps/s	3.0817 KOps/s	$\textbf{\color{#35bf28}+16.01\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6573ms	0.2640ms	3.7874 KOps/s	3.1570 KOps/s	$\textbf{\color{#35bf28}+19.97\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.2318ms	5.9427ms	168.2732 Ops/s	169.5163 Ops/s	$\color{#d91a1a}-0.73\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.6284ms	0.2809ms	3.5595 KOps/s	3.0319 KOps/s	$\textbf{\color{#35bf28}+17.40\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6940ms	0.2842ms	3.5186 KOps/s	3.1406 KOps/s	$\textbf{\color{#35bf28}+12.04\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	1.4316ms	1.2562ms	796.0745 Ops/s	736.9547 Ops/s	$\textbf{\color{#35bf28}+8.02\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	1.5654ms	1.1753ms	850.8298 Ops/s	785.3651 Ops/s	$\textbf{\color{#35bf28}+8.34\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	10.8635ms	6.3283ms	158.0195 Ops/s	166.1276 Ops/s	$\color{#d91a1a}-4.88\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.1864ms	0.4820ms	2.0747 KOps/s	2.0863 KOps/s	$\color{#d91a1a}-0.55\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8794ms	0.4672ms	2.1406 KOps/s	2.1982 KOps/s	$\color{#d91a1a}-2.62\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	6.1390ms	5.9647ms	167.6542 Ops/s	170.3047 Ops/s	$\color{#d91a1a}-1.56\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	2.1483ms	0.3809ms	2.6257 KOps/s	793.4076 Ops/s	$\textbf{\color{#35bf28}+230.93\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.5751ms	0.3199ms	3.1262 KOps/s	3.1621 KOps/s	$\color{#d91a1a}-1.13\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.2149ms	5.8959ms	169.6081 Ops/s	168.6021 Ops/s	$\color{#35bf28}+0.60\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	2.2685ms	0.3530ms	2.8331 KOps/s	3.0328 KOps/s	$\textbf{\color{#d91a1a}-6.58\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.5363ms	0.2868ms	3.4868 KOps/s	3.1899 KOps/s	$\textbf{\color{#35bf28}+9.31\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.3132ms	6.1454ms	162.7227 Ops/s	163.0947 Ops/s	$\color{#d91a1a}-0.23\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	2.1499ms	0.4550ms	2.1977 KOps/s	2.2530 KOps/s	$\color{#d91a1a}-2.46\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8277ms	0.4130ms	2.4212 KOps/s	2.3527 KOps/s	$\color{#35bf28}+2.91\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	6.6580ms	5.1713ms	193.3766 Ops/s	190.8059 Ops/s	$\color{#35bf28}+1.35\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	10.4534ms	2.3640ms	423.0062 Ops/s	448.0775 Ops/s	$\textbf{\color{#d91a1a}-5.60\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	2.2326ms	1.0718ms	933.0342 Ops/s	841.8984 Ops/s	$\textbf{\color{#35bf28}+10.83\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.5085s	15.3589ms	65.1087 Ops/s	56.2078 Ops/s	$\textbf{\color{#35bf28}+15.84\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	3.0768ms	1.4096ms	709.3985 Ops/s	480.8743 Ops/s	$\textbf{\color{#35bf28}+47.52\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	12.5979ms	1.2773ms	782.9318 Ops/s	871.9775 Ops/s	$\textbf{\color{#d91a1a}-10.21\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	7.2929ms	5.3072ms	188.4228 Ops/s	183.8616 Ops/s	$\color{#35bf28}+2.48\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	8.5058ms	2.1818ms	458.3472 Ops/s	457.1389 Ops/s	$\color{#35bf28}+0.26\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	7.9518ms	1.3666ms	731.7365 Ops/s	727.8694 Ops/s	$\color{#35bf28}+0.53\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-True]	35.3795ms	33.3438ms	29.9906 Ops/s	30.1754 Ops/s	$\color{#d91a1a}-0.61\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-10000-10000-100-False]	19.8629ms	17.8078ms	56.1553 Ops/s	58.9281 Ops/s	$\color{#d91a1a}-4.71\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-True]	37.8194ms	34.7792ms	28.7528 Ops/s	28.9035 Ops/s	$\color{#d91a1a}-0.52\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-100000-10000-100-False]	19.5545ms	17.9821ms	55.6110 Ops/s	57.8017 Ops/s	$\color{#d91a1a}-3.79\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-True]	37.5067ms	36.0092ms	27.7707 Ops/s	27.5039 Ops/s	$\color{#35bf28}+0.97\%$
test_rb_extend_sample[ReplayBuffer-LazyTensorStorage-RandomSampler-1000000-10000-100-False]	20.7242ms	19.3182ms	51.7645 Ops/s	53.0222 Ops/s	$\color{#d91a1a}-2.37\%$