Pi0 and Lerobot with Rerun

A repo to explore training robots with Pi0 and Lerobot and human pose motion retargeting

Installation

Currently only linux is supported

Using Pixi

Make sure you have the Pixi package manager installed

git clone https://github.com/rerun-io/pi0-lerobot.git
cd pi0-lerobot

Usage

Camera Calibration using VGGT

Get camera intrinsics/extrinsics, depth maps, and pointcloud for a set of multiview images from hocap

To run camera calibration on a sequence run

pixi run calibrate-cameras-vggt

Human Pose and Kinematics

For HOCap dataset, sample is automatically downloaded from dataset on command run. Checkout example rrd file here

For the full assembly101 dataset go to this link to get/download dataset

To run dataset visualization

pixi run visualize-hocap-dataset # recommended
pixi run visualize-assembly101-dataset

Current Pipeline

To run 2D pose estimation, tracking, and triangulation

pixi run pose-estimation-assembly101

Jupyter Notebook Tutorials

pixi run notebook_tutorial

TODO

Part 1. Human Pose and Kinematics

• Basic Triangulation from 2D Detection (body pose)
• Basic Triangulation from 2D detection (hand pose)
• Detection by Tracking (extrapolate 3d views-> Generate bounding box based on extrapolated -> check based on kpts confidence)
• Mano + SMPL fitting for skeleton kinematics
• Add EgoCentric (first person) headset views and poses

Part 2. Body and Hand Pose Retargeting

• Implement pose retargeting to isaac sim

Part 3. Immitation Learning and Teleoperation

• Notebook Tutorials
• Finetune and evaluation on Aloha simulator with Pi0
• Finetune base Pi0 on SO-ARM100 from lerobot