Skip to content

Latest commit

 

History

History
1365 lines (1308 loc) · 44.8 KB

README.md

File metadata and controls

1365 lines (1308 loc) · 44.8 KB
Raw

Paper: (https://arxiv.org/pdf/2310.07602.pdf)

News

[2023.1.15] We are very sorry for the damage to some files in the published data, and the partitioning issue of imagesets has been corrected. And you can replace these images(006702.png, 006708.png, 006710.png, 006713.png, 006717.png) in the original dataset.

[2023.10.29] We have released the dataset download link.

[2023.12.10] Our Code currently supports VFF and M2Fusion.

[2023.10.27] Our Code currently supports some baselines including Voxel RCNN, Second, Pointpillars, RDIOU. Other baselines will be updated soon.

[2023.10.15] Our code and data are still being maintained and will be released soon.

1. Introduction

Dual-Radar is a new dataset based on 4D radar that can be used for studies on 3D object detection and tracking in the field of autonomous driving. The perception system of ego vehicle includes a high-resolution camera, a 80-line LiDAR and two up-to-date and different models of 4D radars operating in different modes(Arbe and ARS548). The dataset comprises of raw data collected from ego vehicle, including scenarios such as urban and tunnels, with weather conditions of rainy, cloudy, sunny and so on. Our dataset also includes data from different time periods, including dusk, nighttime, and daytime. Our collected raw data amounts to a total of 12.5 hours, encompassing a total distance of over 600 kilometers. Our dataset covers a route distance of approximately 50 kilometers. It consists of 151 continuous time sequences, with the majority being 20-second sequences, resulting in a total of 10,007 carefully time-synchronized frames.

Image 1

a) Ego vehicle's work scenario

 Image 1

b) Data projection visualization

Figure 1. Data collection vehicle and data projection visualization

Sensor Configuration

Our ego vehicle’s configuration and the coordinate relationships between multiple sensors are shown in Figure. 2. The platform of our ego vehicle system consists of a high-resolution camera, an new 80-line LiDAR, and two types of 4D radar. All sensors have been carefully calibrated. The camera and LiDAR are mounted directly above the ego vehicle, while the 4D radars are installed in front of it. Due to the range of horizontal view limitations of the camera and 4D radars, we only collect data from the front of our ego vehicle for annotation. The ARS548 RDI captures data within approximately 120° horizontal field of view and 28° vertical field of view in front of the ego vehicle, while the Arbe Phoenix, operating in middle-range mode, collects data within a 100° horizontal field of view and 30° vertical field of view. The LiDAR collects around the ego vehicle in a 360° manner but only retains the data in the approximate 120° field of view in front of it for annotation.

Figure 2. Sensor Configuration and Coordinate Systems

  • The specification of the autonomous vehicle system platform. Our proposed dataset is collected from a high-resolution camera, an 80-line mechanical LiDAR, and two types of 4D radars, the Arbe Phoenix and the ARS548 RDI radar. Our dataset provides GPS information for timing implementation. The sensor configurations are shown in Table 1.

Table 1. The specification of the autonomous vehicle system platform

Sensors Type Resolution Fov FPS
Range Azimuth Elevation Range Azimuth Elevation
camera acA1920-40uc - 1920X 1200X - - - 10
LiDAR RS-Ruby Lite 0.05m 0.2° 0.2° 230m 360° 40° 10
4D radar ARS 548RDI 0.22m 1.2°@0...±15°
1.68°@±45°		 2.3° 300m ±60° ±4°@300m
±14°@<140m		 20
Arbe Phoenix 0.3m 1.25° 2° 153.6m 100° 30° 20
  • The statistics of number of points cloud per frame. In addition,we analyze the distribution density of the point clouds and the number of point clouds per frame, as shown in Table 2.

Table 2. The statistics of number of points cloud per frame

Transducers Minimum Values Average Values Maximum Values
LiDAR 74386 116096 133538
Arbe Phnoeix 898 11172 93721
ARS548 RDI 243 523 800

2. Data Acquisition Scenario

  • The visualization of raw data collected under different weather conditions and the visualization of annotated 3D bounding boxes are performed separately. The three types of data(Lidar, Arbe and Ars548) are transformed onto a unified coordinate system.
Image 1

a) Sunny,Daytime

 Image 1

b) Sunny,Nightime
Image 1

c) Rainy,Daytime

 Image 1

d) Cloudy,Daytime

Figure 3. Visualization of raw data sequences under different weather conditions. On the left is a color RGB image, while on the right side, the cyan represents the Arbe point cloud, the white represents the LiDAR point cloud, and the yellow represents the ARS548 point cloud.

Figure 4. Visualization of 3D bounding box projection on data. The first column represents the 3D frame markers on the image. The Column 2, 3, and 4 represent the point cloud from Lidar, Arbe Phoenix radar point cloud, and ARS548 RDI radar point cloud, respectively. Each row represents a scenario type.(a): downtown daytime normal light. (b): downtown daytime backlight. (c): downtown dusk normal light. (d): downtown dusk backlight. (e): downtown clear night. (f): downtown daytime cloudy. (g): downtown rainy day. (h): downtown cloudy dusk. (i): downtown cloudy night. (j): downtown rainy night. (k): daytime tunnel. (l): nighttime tunnel.

3. Download Link

  • Our dataset is freely available to researchers. Please download and sign our agreement and send it to the provided email address ([email protected]). You will receive the download link within one week.

  • When unzipping the data, please file and organize it by following the format below:

    └─Dual Radar
    ├─ImageSets.zip
    ├─testing
    │  ├─testing_arbe.zip
    │  ├─testing_ars548.zip
    │  ├─testing_calib.zip
    │  ├─testing_image.zip
    │  ├─testing_label.zip
    │  ├─testing_robosense.zip
    ├─training
    │  ├─training_arbe.zip
    │  ├─training_ars548.zip
    │  ├─training_calib.zip
    │  ├─training_image.zip
    │  ├─training_label.zip
    │  ├─training_robosense.zip
    └─README_dual_radar.txt
  • This folder contains 10007 frames of labeled pointclouds and image data. The structure of the folder is shown as blow:
    └─Dual Radar
    ├─ImageSets
    │      test.txt
    │      train.txt
    │      trainval.txt
    │      val.txt
    ├─testing
    │  ├─arbe
    │  │      000000.bin	# Raw pointclouds (removed None) of the Arbe.
    │  │      ...............
    │  ├─ars548
    │  │      000000.bin	# Raw pointclouds (removed None) of the ARS548.
    │  │      ...............
    │  ├─calib
    │  │      000000.txt
    │  │      ...............
    │  ├─image
    │  │      000000.png	# Undistort images of the camera.
    │  │      ...............
    │  ├─label
    │  │      000000.txt	# Label in txt format, explain later.
    │  │      ...............
    │  ├─robosense
    │  │      000000.bin	# Raw pointclouds (removed None) of the LiDAR.
    │  │      ...............
    ├─training
    │  ├─arbe
    │  │      000000.bin
    │  │      ...............
    │  ├─ars548
    │  │      000000.bin
    │  │      ...............
    │  ├─calib
    │  │      000000.txt
    │  │      ...............
    │  ├─image
    │  │      000000.png
    │  │      ...............
    │  ├─label
    │  │      000000.txt
    │  │      ...............
    │  ├─robosense
    │  │      000000.bin
    │  │      ...............
    └─README.txt

4. The Description of Calib Format

  • The calib.txt contains tree parts. The dataset consists of two parts: the data part and the alignment calibration file. The data part is image data in png format and point cloud data in bin format. The alignment calibration file includes calibration parameters for the four sensors. The camera-LiDAR, camera-4D radar joint calibration are shown here as examples for illustration.
   Dual Radar_cam.Intrinsics.RadialDistortion: Barrel distortion of Dual Radar_cam [ k1, k2, k3 ]
   Dual Radar_cam.Intrinsics.TangentialDistortion: radial distortion of Dual Radar_cam [ p1, p2 ]
   Dual Radar_cam.IntrinsicMatrix: Dual Radar_cam's intrinsic matrix [ af, 0, 0; 0, bf, 0; u, v, 1]
   Dual Radar_LiDAR-->Dual Radar_cam: Dual Radar_lidar to Dual Radar cam's single response matrix P(4×4)
   Dual Radar_radar--> Dual Radar_cam: Dual Radar_radar to Dual Radar_cam rotation matrix + translation matrix P(3×4)

5. Label Files Discription

  • All values (numerical or strings) are separated via spaces, each row corresponds to one object. The 19 columns represent:
  Value       Name             Description
  -------------------------------------------------------------------------------------------------------
  1        type               Describes the type of object: 'Car', 'Van', 'Truck', 'Pedestrian',  'Person_sitting', 'Cyclist', 'Tram', 'Misc' or 'DonCare'
  1        truncated          Float from 0 (non-truncated) to 1 (truncated), where truncated refers to the object leaving image boundaries
  1        occluded           Integer (0,1,2,3) indicating occlusion state: 0 = fully visible, 1 = partly ccluded, 2 = largely occluded, 3 = unknown
  1        alpha              Observation angle of object, ranging [-pi..pi]
  4        bbox               2D bounding box of object in the image (0-based index): contains left, top, right, bottom pixel coordinates. 
  3        dimensions         3D object dimensions: height, width, length (in meters).
  3        location           3D object location x,y,z in camera coordinates (in meters).
  1        rotation_y         Rotation ry around Y-axis in camera coordinates [-pi..pi].
  1        score              Only for results: Float,indicating confidence in detection, needed for p/r curves , higher is better.
  1        track_id           Path tracking of the same object

  • Since the labeling work is done in label coordinate, the bounding box out of the image FOV(1920×1080) needs to be cut.

  • Location mean the xyz in label coordinate. The same coordinate origen and the relation of axis is shown below.

Figure 5. Illustration of sensor coordinate systems

  • The difference between rotation_y and alpha are, that rotation_y is directly given in camera coordinates, while alpha also considers the vector from the camera center to the object center, to compute the relative orientation of the object with respect to the camera. For example, a car which is facing along the X-axis of the camera coordinate system corresponds to rotation_y=0, no matter where it is located in the X/Z plane (bird's eye view), while alpha is zero only, when this object is located along the Z-axis of the camera. When moving the car away from the Z-axis, the observation angle will change.

6. Data Statistics

Figure 6. Distribution of weather conditions.

  • We separately count the number of instances for each category in the Dual-Radar dataset and the distribution of different types of weather. About two-thirds of our data are collected under normal weather conditions, and about one-third are collected under rainy and cloudy conditions. We collect 577 frames in rainy weather, which is about 5.5% of the total dataset. The rainy weather data we collect can be used to test the performance of different 4D radars in adverse weather conditions.

Figure 7. Distribution of instance conditions.

  • We also conduct a statistical analysis of the number of objects with each label at different distance ranges from our vehicle, as shown in Figure 7. Most objects are within 60 meters of our ego vehicle.

7. Getting Started

Environment

This is the documentation for how to use our detection frameworks with Dual-Radar dataset. We test the Dual-Radar detection frameworks on the following environment:

  • Python 3.8.16 (3.10+ does not support open3d.)
  • Ubuntu 18.04/20.04
  • Torch 1.10.1+cu113
  • CUDA 11.3
  • opencv 4.2.0.32

Preparing The Dataset

  • After all files are downloaded, please arrange the workspace directory with the following structure:

  • Organize your code structure as follows

    Frameworks
      ├── checkpoints
      ├── data
      ├── docs
      ├── pcdet
      ├── output
  • Organize the dataset according to the following file structure
    dual_radar
      ├── lidar
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── velodyne
        ├── testing
            ├── calib
            ├── image
            ├── velodyne
      ├── radar_arbe
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── arbe
        ├── testing
            ├── calib
            ├── image
            ├── arbe
      ├── radar_ars548
        ├── ImageSets
            ├── train.txt
            ├── trainval.txt
            ├── val.txt
            ├── test.txt
        ├── training
            ├── calib
            ├── image
            ├── label
            ├── ars548
        ├── testing
            ├── calib
            ├── image
            ├── ars548

Requirements

  • Clone the repository
 git clone https://github.com/adept-thu/Dual-Radar.git
 cd Dual-Radar
  • Create a conda environment
conda create -n Dual-Radardet python=3.8.16
conda activate Dual-Radardet

  • Install PyTorch (We recommend pytorch 1.10.1.)

  • Install the dependencies

pip install -r requirements.txt
  • Install Spconv（our cuda version is 113）
pip install spconv-cu113
  • Build packages for Dual-Radardet
python setup.py develop

Train & Evaluation

  • Generate the data infos by running the following command:
using lidar data
python -m pcdet.datasets.dual_radar.dual_radar_dataset create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset.yaml

using arbe data
python -m pcdet.datasets.dual_radar.dual_radar_dataset_arbe create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_arbe.yaml

using ars548 data
python -m pcdet.datasets.dual_radar.dual_radar_dataset_ars548 create_dual_radar_infos tools/cfgs/dataset_configs/dual_radar_dataset_ars548.yaml
  • To train the model on single GPU, prepare the total dataset and run
python train.py --cfg_file ${CONFIG_FILE}
  • To train the model on multi-GPUs, prepare the total dataset and run
sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE}
  • To evaluate the model on single GPU, modify the path and run
python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT}
  • To evaluate the model on multi-GPUs, modify the path and run
sh scripts/dist_test.sh ${NUM_GPUS} \
    --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}

Quick Demo

Here we provide a quick demo to test a pretrained model on the custom point cloud data and visualize the predicted results

  • Download the pretrained model as shown in Table 4~8.
  • Make sure you have installed the Open3d and mayavi visualization tools. If not, you could install it as follow:
pip install open3d
pip install mayavi
  • prepare your point cloud data
points[:, 3] = 0 
np.save(`my_data.npy`, points)
  • Run the demo with a pretrained model and point cloud data as follows
python demo.py --cfg_file ${CONFIG_FILE} \
    --ckpt ${CKPT} \
    --data_path ${POINT_CLOUD_DATA}

8. Experimental Results

Table 3. Multi-modal experimental results([email protected] 0.25 0.25)

Baseline Data Car Pedestrain Cyclist model pth
[email protected] [email protected] [email protected]
Easy Mod. Hard Easy Mod. Hard Easy Mod. Hard
VFF camera+LiDAR 94.60 84.14 78.77 39.79 35.99 36.54 55.87 51.55 51.00 model
camera+Arbe 31.83 14.43 11.30 0.01 0.01 0.01 0.20 0.07 0.08 model
camera+ARS548 12.60 6.53 4.51 0.00 0.00 0.00 0.00 0.00 0.00 model
M2-Fusion LiDAR+Arbe 89.71 79.70 64.32 27.79 20.41 19.58 41.85 36.20 35.14 model
LiDAR+ARS548 89.91 78.17 62.37 34.28 29.89 29.17 42.42 40.92 39.98 model

Table 4. Multi-modal experimental results([email protected] 0.25 0.25)

Baseline Data Car Pedestrain Cyclist model pth
[email protected] [email protected] [email protected]
Easy Mod. Hard Easy Mod. Hard Easy Mod. Hard
VFF camera+Lidar 94.60 84.28 80.55 40.32 36.59 37.28 55.87 51.55 51.00 model
camera+Arbe 36.09 17.20 13.23 0.01 0.01 0.01 0.20 0.08 0.08 model
camera+ARS548 16.34 9.58 6.61 0.00 0.00 0.00 0.00 0.00 0.00 model
M2-Fusion LiDAR+Arbe 90.91 85.73 70.16 28.05 20.68 20.47 53.06 47.83 46.32 model
LiDAR+ARS548 91.14 82.57 66.65 34.98 30.28 29.92 43.12 41.57 40.29 model

Table 5. Single-modal experimental results([email protected] 0.25 0.25)

Baseline Data Car Pedestrain Cyclist model pth
[email protected] [email protected] [email protected]
Easy Mod. Hard Easy Mod. Hard Easy Mod. Hard
Pointpillars LiDAR 81.78 55.40 44.53 43.22 38.87 38.45 25.60 24.35 23.97 model
Arbe 49.06 27.64 18.63 0.00 0.00 0.00 0.19 0.12 0.12 model
ARS548 11.94 6.12 3.76 0.00 0.00 0.00 0.99 0.63 0.58 model
RDIou LiDAR 63.43 40.80 32.92 33.71 29.35 28.96 38.26 35.62 35.02 model
Arbe 51.49 26.74 17.83 0.00 0.00 0.00 0.51 0.37 0.35 model
ARS548 5.96 3.77 2.29 0.00 0.00 0.00 0.21 0.15 0.15 model
VoxelRCNN LiDAR 86.41 56.91 42.38 52.65 46.33 45.80 38.89 35.13 34.52 model
Arbe 55.47 30.17 19.82 0.03 0.02 0.02 0.15 0.06 0.06 model
ARS548 18.37 8.24 4.97 0.00 0.00 0.00 0.24 0.21 0.21 model
Cas-V LiDAR 80.60 58.98 49.83 55.43 49.11 48.47 42.84 40.32 39.09 model
Arbe 27.96 10.27 6.21 0.02 0.01 0.01 0.05 0.04 0.04 model
ARS548 7.71 3.05 1.86 0.00 0.00 0.00 0.08 0.06 0.06 model
Cas-T LiDAR 73.41 45.74 35.09 58.84 52.08 51.45 35.42 33.78 33.36 model
Arbe 14.15 6.38 4.27 0.00 0.00 0.00 0.09 0.06 0.05 model
ARS548 3.16 1.60 1.00 0.00 0.00 0.00 0.36 0.20 0.20 model

Table 6. Single-modal experimental results([email protected] 0.25 0.25)

Baseline Data Car Pedestrain Cyclist model pth
[email protected] [email protected] [email protected]
Easy Mod. Hard Easy Mod. Hard Easy Mod. Hard
Pointpillars LiDAR 81.81 55.49 45.69 43.60 39.59 38.92 38.78 38.74 38.42 model
Arbe 54.63 35.09 25.19 0.00 0.00 0.00 0.41 0.24 0.23 model
ARS548 14.40 8.14 5.26 0.00 0.00 0.00 2.27 1.64 1.53 model
RDIou LiDAR 63.44 41.25 33.74 33.97 29.62 29.22 49.33 47.48 46.85 model
Arbe 55.27 31.48 21.80 0.01 0.01 0.01 0.84 0.66 0.65 model
ARS548 7.13 5.00 3.21 0.00 0.00 0.00 0.61 0.46 0.44 model
VoxelRCNN LiDAR 86.41 56.95 42.43 41.21 53.50 45.93 47.47 45.43 43.85 model
Arbe 59.32 34.86 23.77 0.02 0.02 0.02 0.21 0.15 0.15 model
ARS548 21.34 9.81 6.11 0.00 0.00 0.00 0.33 0.30 0.30 model
Cas-V LiDAR 80.60 59.12 51.17 55.66 49.35 48.72 51.51 50.03 49.35 model
Arbe 30.52 12.28 7.82 0.02 0.02 0.02 0.13 0.05 0.05 model
ARS548 8.81 3.74 2.38 0.00 0.00 0.00 0.25 0.21 0.19 model
Cas-T LiDAR 73.42 45.79 35.31 59.06 52.36 51.74 44.35 44.41 42.88 model
Arbe 22.85 13.06 9.18 0.00 0.00 0.00 0.17 0.08 0.08 model
ARS548 4.21 2.21 1.49 0.00 0.00 0.00 0.68 0.43 0.42 model

Table 7. Single-modal experimental results in the rainy scenario([email protected] 0.25 0.25)

Baseline Data Car Pedestrain Cyclist model pth
[email protected] [email protected] [email protected]
Easy Mod. Hard Easy Mod. Hard Easy Mod. Hard
Pointpillars LiDAR 60.57 44.31 41.91 32.74 28.82 28.67 29.12 25.75 24.24 model
Arbe 68.24 48.98 42.80 0.00 0.00 0.00 0.19 0.10 0.09 model
ARS548 11.87 8.41 7.32 0.11 0.09 0.08 0.93 0.36 0.30 model
RDIou LiDAR 44.93 39.32 39.09 24.28 21.63 21.43 52.64 43.92 42.04 model
Arbe 67.81 49.59 43.24 0.00 0.00 0.00 0.38 0.30 0.28 model
ARS548 5.87 5.48 4.68 0.00 0.00 0.00 0.09 0.01 0.01 model

Table 8. Single-modal experimental results([email protected] 0.25 0.25) in the rainy scenario

Baseline Data Car Pedestrain Cyclist model pth
[email protected] [email protected] [email protected]
Easy Mod. Hard Easy Mod. Hard Easy Mod. Hard
Pointpillars LiDAR 60.57 44.56 42.49 32.74 28.82 28.67 44.39 40.36 38.64 model
Arbe 74.50 59.68 54.34 0.00 0.00 0.00 0.32 0.16 0.15 model
ARS548 14.16 11.32 9.82 0.11 0.09 0.08 2.26 1.43 1.20 model
RDIou LiDAR 44.93 39.39 39.86 24.28 21.63 21.43 10.80 52.44 50.28 model
Arbe 70.09 54.17 47.64 0.00 0.00 0.00 0.63 0.45 0.45 model
ARS548 6.36 6.51 5.46 0.00 0.00 0.00 0.13 0.08 0.08 model

9. Acknowledgement

  • Thanks for the sensor support provided by Beijing Jingwei Hirain Technologies Co., Inc.

10. Citation

  • If you find this work is useful for your research, please consider citing:
@article{zhang2023dual,
  title={Dual Radar: A Multi-modal Dataset with Dual 4D Radar for Autononous Driving},
  author={Zhang, Xinyu and Wang, Li and Chen, Jian and Fang, Cheng and Yang, Lei and Song, Ziying and Yang, Guangqi and Wang, Yichen and Zhang, Xiaofei and Yang,Qingshan and Li, Jun},
  journal={arXiv preprint arXiv:2310.07602},
  year={2023}
}