👔 UniFolding

This repo contains official code for the CoRL 2023 paper: UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding. UniFolding is a sample-efficient, scalable, and generalizable robotic system for unfolding and folding various garments with large variations in shapes, textures and materials.

• Paper: UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding
• Website: https://unifolding.robotflow.ai

Introduction

Challenges

Method & Advantages

Experiments

⚙Environment Setup

This section describes how to setup neccessary environments for this project.

🧠 Learning Environment

The learning code should work on environments that meet the following requirements:

• Modern Linux Distributions that are not in EOL.
• Python >= 3.8
• Pytorch >= 1.11.0
• CUDA >= 11.3

We recommend these combinations:

• Ubuntu 20.04
• Python = 3.8
• Pytorch = 1.11.0
• CUDA = 11.3

To setup the learning environment, you need to download and install CUDA from here in advance. Then, you should run the setup-env.sh script to setup all basic requirments except for GroundedSAM.

bash setup-env.sh

This script will automatically create an conda environment named unifolding and install dependent packages in it. You can modify this script to make it behaves differently.

Finally, see GroundedSAM for installation of Grounded-DINO and Segment-Anything.

🔽 Pre-trained Models

Please download the pre-trained models for long-sleeve Shirts and short-sleeve T-shirts from here.

💻 Simulation Environment

The simulation environment depends on RFUniverse. To set up the simulation environment, please refer to Simulation Environment.

CLOTH3D dataset is used in our simulation. You can download it from here.

📷🦾🗒 Real Environment

📷Camera

Our project should work on any commercial depth cameras systems that produce colorful point cloud and RGB images. However, for the best performance, we recommend high-precision and high-resolution depth cameras. In our experiment, we adopted Photoneo MotionCam3D M+ as the main depth camera and MindVision SUA202GC as the subordinate color camera. However, if you are using custom cameras, please re-implement capture_pcd() method in manipulation.experiment_real::ExperimentReal.

🦾Robot

Our experiments are conducted using two Flexiv Rizon 4 robot arms. To operate these robot arms, you should install Flexiv RDK from here. If you are using custom robot arms, please re-implement execute_action() method in manipulation.experiment_real::ExperimentReal.

🗒OMPL Setup

Open Motion Plannig Library (OMPL) is a widely-adopted library for motion plannig. We use this library to plan trajectories for the dual Flexiv arm setup. OMPL can be installed from here.

📕 Usage Example

Here is an minimum example of using our model. The Inference3D class is an interface of our AI model (UFONet), it can work seamlessly for both virtual environments (ExperimentVirtual) and real environments (ExperimentReal). You can simply modify the experiment settings by overriding config parameters with Hydra.

import hydra
from omegaconf import DictConfig

from learning.inference_3d import Inference3D
from manipulation.experiment_real import ExperimentReal
from manipulation.experiment_virtual import ExperimentVirtual


@hydra.main(
    config_path="config/real_experiment", config_name="experiment_real_tshirt_long", version_base="1.1"
)
def main(cfg: DictConfig) -> None:
    # create experiment instance that contains robot controller, camera, environment setup, etc.
    exp = ExperimentReal(config=cfg.experiment)
    # exp = ExperimentVirtual(config=cfg.experiment)  # uncomment this line to use virtual environment

    # the interface of AI model (UFONet)
    inference = Inference3D(experiment=exp, **cfg.inference)

    # capture point cloud
    obs, err = exp.capture_pcd()  # please re-implement this method if you are using custom hardware

    # predict action type
    action_type = inference.predict_raw_action_type(obs)

    # predict action
    prediction_message, action_message, err = inference.predict_action(obs, action_type)

    # execute action
    err = exp.execute_action(action_message)  # please re-implement this method if you are using custom hardware


if __name__ == "__main__":
    main()

🌎 Inference

We have made a Makefile target to run the full inference process (unfolding + folding) in the real world.

make stage3.real.tshirt_long.test

⏳ Training Pipeline

The training pipeline includes 4 stages. Stage 1-3 are wrapped as Makefie targets.

Stage 0 (VR Data Collection)

The VR data recording system used in our paper is based on the VR-Garment project. Please follow its instructions for the hardware setup and guidelines to collect human demonstration data in Virtual Reality.

Stage 1 (Supervised Training)

make stage1.virtual.tshirt_long.train

Stage 2 (Data Collection + Self-Supervised Training)

make stage2.virtual.tshirt_long.run

Stage 3 (Data Collection + Hybrid Training)

make stage3.real.tshirt_long.run &
make stage3.virtual.tshirt_long.run

⚠The two Makefile targets should run simutaneously.

🙏 Acknowlegement

• The baseline model used in our code is adapted from ClothFunnels.

🏷️ License

This project is freely available for non-commercial use, and may be redistributed under these conditions. For commercial queries, please drop an e-mail at xiaoxiaoxh[at]sjtu[dot]edu[dot]cn and cc lucewu[at]sjtu[dot]edu[dot]cn. We will send the detail agreement to you.

🔗Citation

If you find this project helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{xue2023unifolding,
  title={UniFolding: Towards Sample-efficient, Scalable, and Generalizable Robotic Garment Folding},
  author={Xue, Han and Li, Yutong and Xu, Wenqiang and Li, Huanyu and Zheng, Dongzhe and Lu, Cewu},
  booktitle={Conference on Robot Learning},
  pages={3321--3341},
  year={2023},
  organization={PMLR}
}