[Enhancement]: Support for OpenAI Gym like API surface

[vyeevani]

Describe the solution you'd like

I'd like there to be a way to create a simulator that's also interface-able through OpenAI gym APIs. It would be great if this simulation environment was parallelizable on a hardware accelerator. Specifically, something like: https://github.com/google/brax given its support for TPU, GPU, and CPU. It would be great if this simulator environment and the real robot shared the same API surface. This repo here uses the OpenAI gym API to interface with a single widow-x: https://github.com/rail-berkeley/bridge_data_robot.

Platform

xsarms, uxarms

Is your feature request related to a problem? Please describe.

It's frustrating that in order to test any control policy, I have to setup the environment, robot, and any cameras. Additionally, most deep reinforcement learning techniques have poor sample efficiency all but requiring the use of simulators.

Suggested code

No response

Additional Info

No response

[lukeschmitt-tr]

Integration into more RL-focused simulators like Isaac Sim, PyBullet, and MuJoCo is something we've discussed internally a number of times but have never had the resources to do. We would be open to contributions if you'd like to take a crack at it.