[Feature] Transform for Recording History Observations

[btx0424]

Description

This draft PR proposes a Transform for recording histories of observations, which is a common practice in robotic tasks. It should also address #1676: to record observation-action history, append the previous action to observation, and then transform the environment with History.

The naming, description (docstring), and tests are for the initial commit. Opinions and suggestions are wanted.

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds core functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1763

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (16 Unrelated Failures)

As of commit 6c1109e with merge base eec9f9e ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Workflow failed! Resource not accessible by integration
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Workflow failed! Resource not accessible by integration
• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t 8e11c6f609a0f621f7138f465216c8228104ae557b84b681df8fe59fc2e6ce57 /exec failed with exit code 139
• Lint / python-source-and-configs / linux-job (gh)
  torchrl/envs/transforms/transforms.py:7274:1: F811 redefinition of unused 'RemoveEmptySpecs' from line 6823
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t b8cae4a63ab641072cc7704337d3da1ea95098a1a8c3395a803f242d1d6e4604 /exec failed with exit code 2
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t 55e573d4efefea44797bedafdca39024f7b2316cb85595b9ac0006589e7250b8 /exec failed with exit code 2
• Unit-tests on Linux / tests-cpu (3.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 8d7b2d1bc6084f48a789228c31fc4eed430510c391c6be21880ba232cdcc97a3 /exec failed with exit code 2
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  RuntimeError: Command docker exec -t 7c6ecc904c85bf1a04e3047f0732cc5fa4d7beaf6d83aaab025302b77244dbc7 /exec failed with exit code 2
• Unit-tests on Linux / tests-gpu (3.8, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 207c278b415e56d8e7646c277d00d9e4d57552c6188dccb8f412b815dcf73987 /exec failed with exit code 2
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t 4209dc37a8899fa68acbb585e943d7aed19d2127226fb459e446093450fa702b /exec failed with exit code 2
• Unit-tests on Linux / tests-optdeps (3.9, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 5bf0c58109a0ca8783a432d373c1a3a6fa6faae62c91db19da56aa6b9bd4b3b4 /exec failed with exit code 2
• Unit-tests on Linux / tests-stable-gpu (3.8, 11.8) / linux-job (gh)
  RuntimeError: Command docker exec -t bd6f6bc7f2392079dd37a1b35cd6d1bd5581896a06d91bc34f75ffc9e9c25ef3 /exec failed with exit code 2
• Unit-tests on MacOS CPU / tests (3.11) / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
• Unit-tests on MacOS CPU / tests (3.8) / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 2
• Unit-tests on Windows / unittests-cpu / windows-job (gh)
  Process completed with exit code 2.
• Unit-tests on Windows / unittests-gpu / windows-job (gh)
  Process completed with exit code 2.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Cool stuff!
Is this the stack version of CatFrames? Do we want to have two separate transforms for that? If what we want is for CatFrames to have the option to discard the last observation or not, or use stack instead of cat, these could be passed as arguments (though I can see why one would rather have two separate classes). I can give some advice if we want to go down that route.

If we have two separate transforms, maybe we could name this one StackFrames for consistency? Is there any piece of code from CatFrames that we should recycle?

We should add the transform to the doc too.

[vmoens]

Is there a reason why we use 16?

[btx0424]

Thanks for replying!

Yes, I think renaming it to StackFrames is a good idea. However, do you prefer it to be stateful (like CatFrames) or stateless (the current implementation)? I will make it reuse CatFrames more if we go for stateful.

I will complete the tests and doc soon.

[vmoens]

It's good to have it stateless I agree. Let's keep it as it is!
Let's align the doc of this class and CatFrames and refer one to the other, it's likely that users will jump from one to the other.
Thanks again!

[vmoens]

@btx0424 Anything I can help you with here? I can find some time to help if needed

[btx0424]

@btx0424 Anything I can help you with here? I can find some time to help if needed

Hi there. I had some trouble figuring out the no-env usage while trying to align it with CatFrames. I noticed that CatFrames, when used for RBs, do not split between trajectories, i.e., some observations may contain information from the last episode. Is that the desired behavior?

[vmoens]

Just seeing this now sorry
No definitely not!
We should fix this if it's the case