gcasp-for-zed-camera

This is an implementation of gcasp for ZED camera. The code is tested on Ubuntu 20.04 with ZED 2i camera.

Introduction

The pipeline of this work is as follows:

1. Use ZED camera to capture RGB-D images.
2. Back-project the depth image to get the point cloud.
3. Use instance segmentation(here I use mmdetection) to get the instance masks.
4. Get the point cloud of each instance.
5. Use gcasp to get the 3D pose of each instance.

The result can be used to transform the object into a canonical pose, which is useful for downstream tasks like point cloud completion.

Installation

• Follow the instructions from openmmlab to install mmdetection.
• Other dependencies:

pip install scikit-image

Inference

• Download the pretrained checkpoint for gcasp from here
• Download the pretrained checkpoint for instance segmentation. Modify the path for config file and checkpoint file in the pose_inference.py around line 45. My implementation uses rtmdet-ins_x_8xb16-300e_coco. You can download the checkpoint from here
• Then run the following command:

python pose_inference.py /home/lxt/research-assignment/gcasp/config/gcn3d_seg/gcn3d_seg_all_256.yaml --pretrained ./epoch_29_iters_581280.pth

• The output should look like this:

Training

For training and evaluation, please refer to the original gcasp repository.