VR System

[wensi-ai]

Features

• Teleoperation System for OmniGibson
  • Supports OpenXR (Meta Quest), SteamVR (HTC VIVE), SpaceMouse, Keyboard teleoperation
  • Added BehaviorRobot
  • 3 demo provided for scene touring, controller teleoperating, and hand tracking
• Assisted Grasping for ManipulationRobot
  • Fully tested with Fetch, Franka, Allegro, and Behaviorbot
  • Grasping demo updated

Known issues

• [Solved] Tiago IK is not quite working possibly due to low joint stiffness/friction, so not tested with VR
• [Solved] Fetch base is very unstable, especially when moving back and forth repetitively

[cremebrule]

LGTM! Rubber stamped the xr_utils.py since there's a lot of magic number shenanigans going on there but I assume it's all been validated to work. Is there any unit test we can add?

[ChengshuLi]

LGTM, feel free to merge!

[Bailey-24]

hello, I can't use the vr system with Data Collection / Playback, how to solve.

I have try

    cfg = dict(scene=scene_cfg, robots=[robot_cfg], objects=object_cfg)
    # Create the environment
    env = og.Environment(configs=cfg)
    # Wrap the environment with DataCollectionWrapper
    data_env = DataCollectionWrapper(
        env=env,
        output_path=collect_hdf5_path,
        only_successes=False,  # Set to True if you only want to record successful episodes
    )
    # Use the wrapped environment as you would normally
    obs, info = data_env.reset()
    
    # update viewer camera pose
    og.sim.viewer_camera.set_position_orientation(position=[-0.22, 0.99, 1.09], orientation=[-0.14, 0.47, 0.84, -0.23])
    # Start teleoperation system
    robot = env.robots[0]

    # Initialize teleoperation system
    teleop_sys = TeleopSystem(config=teleop_config, robot=robot, show_control_marker=False)
    teleop_sys.start()

    # main simulation loop
    for _ in range(2000):
        # Get observation from the original env's robot
        # obs = teleop_sys.get_obs()
        action = teleop_sys.get_action(obs)
        # Step through the data collection wrapper
        obs, reward, terminated, truncated, info = data_env.step(action)
        
    # Save the collected data
    print(f"Saving data to {collect_hdf5_path}")
    data_env.save_data()
    # Shut down the environment cleanly at the end
    # og.sim.stop()  
    # teleop_sys.stop()
    og.clear()
    print("Data collection finished!")

but the system can't receive the control input.