InterFusion

This is a repo of InterFusion for 3D object detection.

The code is mainly based on OpenPCDet.

Introduction

Many recent works detect 3D objects by several sensor modalities for autonomous driving, where high-resolution cameras and high-line Lidars are mostly used but relatively expensive. To achieve a balance between overall cost and detection accuracy, many multi-modal fusion techniques have been suggested. In recent years, the fusion of Lidar and Radar has gained ever-increasing attention, especially 4D Radar, which can adapt to bad weather conditions due to its penetrability. Although features have been fused from multiple sensing modalities, most methods cannot learn interactions from different modalities, which does not make for their best use. Inspired by the self-attention mechanism, here we present InterFusion, an interaction-based fusion framework, to fuse 16-line Lidar with 4D Radar. It aggregates features from two modalities and identifies cross-modal relations between Radar and Lidar features. In experimental evaluations on the Astyx HiRes2019 dataset, our method outperformed the baseline by 4.09% mAP in 3D and 10.34% BEV mAP for the car class at the moderate level.

• Model Framework:

Experiment Results:

• All experiments are tested on Astyx Hires2019

Modality	Method 3D				mAP(%) BEV mAP(%)
Radar	Easy	Moderate	Hard	Easy	Moderate	Hard
PointRCNN	12.23	9.1	9.1	14.95	13.82	13.89
SECOND [20]	24.11	18.5	17.77	41.25	30.58	29.33
PVRCNN [33]	28.21	22.29	20.4	46.62	35.1	33.67
PointPillars [30]	30.14	24.06	21.91	45.66	36.71	35.3
Lidar
PointRCNN [23]	30.67	23.69	23.03	35.75	28.13	23.79
SECOND [20]	53.32	44.1	40.16	57.26	47.52	45.4
PVRCNN [33]	54.93	45.29	41.4	56.71	47.55	45.06
PointPillars [30]	53.02	43.56	41.72	55.76	45.81	43.62
Radar+Lidar Ours(InterFusion)	59.04	47.65	46.47	68.1	56.15	55.01
Delta	+6.02	+4.09	+4.75	+12.34	+10.34	+11.39

Installation

a. Clone this repository.

git clone https://github.com/adept-thu/InterFusion.git

b. Install the dependent libraries as follows:

• Install the dependent python libraries:

pip install -r requirements.txt 

c. Generate dataloader

python -m pcdet.datasets.astyx.astyx_dataset create_astyx_infos tools/cfgs/dataset_configs/astyx_dataset.yaml

Training

CUDA_VISIBLE_DEVICES=1 python train.py --cfg_file cfgs/astyx_models/pointpillar.yaml --tcp_port 25851 --extra_tag yourmodelname

Testing

python test.py --cfg_file cfgs/astyx_models/pointpillar.yaml --batch_size 4 --ckpt ##astyx_models/pointpillar/debug/ckpt/checkpoint_epoch_80.pth

Citation

If you find this project useful in your research, please consider cite: