English | 简体中文

Segment Any Event Streams via Weighted Adaptation of Pivotal Tokens [CVPR '24]

Zhiwen Chen¹   Zhiyu Zhu²   Yifan Zhang²   Junhui Hou²   Guangming Shi¹   Jinjian Wu^1
1Xidian University    ²City University of Hong Kong   

About

Official Code for Segment Any Events via Weighted Adaptation of Pivotal Tokens [📕Paper]. This paper delves into the nuanced challenge of tailoring the Segment Anything Models (SAMs) for integration with event data, with the overarching objective of attaining robust and universal object segmentation within the event-centric domain.

Getting Started

Installation

Clone the repository locally:

pip install git+https://github.com/happychenpipi/EventSAM.git

Create and activate a conda environment and install the required packages:

conda create -n eventsam python=3.8
conda activate eventsam
bash install_eventsam.sh

Data Preparation

In this work, we collected a large-scale RGB-Event dataset for event-centric segmentation, from current available pixel-level aligned datasets (VisEvent and COESOT), namely RGBE-SEG. To explore the zero-shot performance of our method, we showed more segmentation results on MVSEC, DDD17 and DSEC datasets. In addition, we also provide corresponding groundtruth masks or prediction results for comparison. Please download these data with the link below and put in ./data.

Datasets Code：1234 Groundtruths Code：1234 Predictions Code：1234

Format of All Datasets:

├── RGBE_SEG dataset
    ├── Training Subset (472 sequences)
        ├── dvSave-2021_09_01_06_59_10
            ├── event          # Event Source File： [N,4]-[x,y,t,p]
            ├── rgb_image      # RGB Images, which is the input of teacher network.
            ├── event_image    # Event-oriented Binary Images, which is used for event visualization.
            ├── voxel_image    # Event-oriented Voxel-like Images, which is the input of student network.
        ├── ... 
    ├── Testing Subset For Normal Scenes (104 sequences) # Easy, Medium, Hard
        ├── dvSave-2021_07_30_11_04_12
            ├── event
            ├── rgb_image
            ├── event_image
            ├── voxel_image 
        ├── ...
    ├── Testing Subset For Degraded Scenes (28 sequences) # Low Light, Over Exposure, Motion Blur
        ├── video_0078
            ├── event
            ├── rgb_image
            ├── event_image
            ├── voxel_image 
        ├── ...

├── MVSEC_SEG/DDD17_SEG/DSEC_SEG dataset
    ├── Testing Subset
        ├── seq_name
            ├── event
            ├── rgb_image
            ├── event_image
            ├── voxel_image 
        ├── ...

Format of Groundtruth Masks or Prediction Masks:

├── RGBE_SEG dataset
    ├── Testing Subset For Normal Scenes (108 sequences) # Easy, Medium, Hard
        ├── dvSave-2021_07_30_11_04_12
            ├── **.png     # Groundtruth Masks/Prediction Masks.
        ├── ...

├── MVSEC_SEG/DDD17_SEG/DSEC_SEG dataset
    ├── Testing Subset
        ├── seq_name
            ├── **.png     # Groundtruth Masks/Prediction Masks.
        ├── ...

Training

First download a pre-trained model checkpoint (e.g. sam_vit_b.pth) SAM and put in ./pretrained. Then the model can be used as teacher for rgb-event knowledge distillation:

python ./event_encoder/train.py

Pre-trained Model

Pre-trained EventSAM model (e.g. rgbe_encoder.pth) needs to be downloaded and put in ./checkpoints.

EventSAM Model Code：1234

Evaluation

Predict the segment masks of event images:

python ./evaluate/predict_mask.py

Calculate metrics of predicted masks:

python ./evaluate/calculate_metric.py

Visualization

EventSAM&LLM

To further validate the strong zero-shot object recognition ability of our event-adapt SAM. We integrate it with a vision-language object segmentation framework LISA. Through this, we could further unlock the rich semantic inherent in SAM, for interactive universal object segmentation with Event data. There are some visualizations.

Acknowledgments

Thanks to VisEvent, COESOT, MVSEC, DDD17, DSEC datasets, SAM and LISA projects.

Contact

Feedbacks and comments are welcome! Feel free to contact us via [email protected] and [email protected].

Citing EventSAM

If you use EventSAM in your research, please use the following BibTeX entry.

@article{chen2023segment,
  title={Segment Any Events via Weighted Adaptation of Pivotal Tokens},
  author={Chen, Zhiwen and Zhu, Zhiyu and Zhang, Yifan and Hou, Junhui and Shi, Guangming and Wu, Jinjian},
  journal={arXiv preprint arXiv:2312.16222},
  year={2023}
}