Skip to content

Source code of PivotNet (ICCV2023, PivotNet: Vectorized Pivot Learning for End-to-end HD Map Construction)

License

Notifications You must be signed in to change notification settings

wenjie710/PivotNet

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

NEWS !!!

  • Jan. 15th, 2024: 🚀 🚀 🚀 The official implementation of our PivotNet is released now. Enjoy it!
  • Jul. 14th, 2023: 👏 Our PivotNet is accepted by ICCV 2023 ! Refer to the Long-version Paper for more details.
  • May. 26th, 2023: 🏆 Our team win the Championship and the Innovation-Award of the CVPR23 Online HD Map Construction Challenge ! [ Leaderboard / Tech-Report]
  • Feb. 28th, 2023: 👏 Our BeMapNet is accepted by CVPR 2023 ! Refer to the Paper for more details. [Long version on Arxiv / Code]

Introduction

Vectorized high-definition map (HD-map) construction has garnered considerable attention in the field of autonomous driving research. Towards precise map element learning, we propose a simple yet effective architecture named PivotNet, which adopts unified pivot-based map representations and is formulated as a direct set prediction paradigm. Concretely, we first propose a novel Point-to-Line Mask module to encode both the subordinate and geometrical point-line priors in the network. Then, a well-designed Pivot Dynamic Matching module is proposed to model the topology in dynamic point sequences by introducing the concept of sequence matching. Furthermore, to supervise the position and topology of the vectorized point predictions, we propose a Dynamic Vectorized Sequence loss. PivotNet contains four primary components: Camera Feature Extractor, BEV Feature Decoder, Line-aware Point Decoder, and Pivotal Point Predictor. It takes the RGB images as inputs and generates flexible and compact vectorized representation without any post-processing.

Documentation

We build the released version of PivotNet upon BeMapNet project. Therefore, this project supports the reproduction of both PivotNet and BeMapNet.

Step-by-step Installation <\br>
  • a. Check Environment

    Python >= 3.8
    CUDA 11.1
    # other versions of python/cuda have not been fully tested, but I think they should work as well.
  • b. Create a conda virtual environment and activate it. (Optional)

    conda create -n pivotnet python=3.8 -y
    conda activate pivotnet
  • c. Install PyTorch and torchvision following the official instructions.

    pip3 install torch==1.10.1+cu111 torchvision==0.11.2+cu111 -f https://download.pytorch.org/whl/torch_stable.html
  • d. Install MMCV following the official instructions. (need GPU)

    pip3 install -U openmim
    mim install mmcv==1.7.1
  • e. Install Detectron2 following the official instructions.

    python3 -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu111/torch1.10/index.html
  • f. Compiling CUDA operators for Deformable Transformer.

    cd mapmaster/models/bev_decoder/deform_transformer/ops
    bash make.sh
  • g. Install PivotNet.

    git clone [email protected]:wenjie710/PivotNet.git
    cd pivotnet
    pip3 install -r requirement.txt
Material Preparation <\br>
  • a. Data: NuScenes

    • Download&Unzip the NuScenes dataset into your server and link it to desirable path.
      cd /path/to/pivotnet
      mkdir data
      ln -s /any/path/to/your/nuscenes data/nuscenes
    • Generate Pivot/Bezier-annotations from NuScenes's raw-annotations.
      cd /path/to/pivotnet
      python3 tools/anno_converter/nuscenes/convert.py -d ./data # generate pivot-annotation only
      OR python3 tools/anno_converter/nuscenes/convert.py -d ./data --bezier   # generate both pivot and bezier annotatation (more time needed)
  • b. Weights: Public-Pretrain-Models

    • Download public pretrain-weights as backbone initialization.
      cd /path/to/pivotnet
      cd assets/weights
      wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/efficientnet-b0-355c32eb.pth .
      wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/resnet50-0676ba61.pth .
      wget https://github.com/wenjie710/PivotNet/releases/download/v1.0/upernet_swin_tiny_patch4_window7_512x512.pth .
  • c. Check: Project-Structure

    • Your project directory should be,
        assets
          | -- weights (resnet, swin-t, efficient-b0, ...)
          | -- 
        mapmaster
        configs
        data
          | -- nuscenes
            | -- samples (CAM_FRONT, CAM_FRONT_LEFT, CAM_FRONT_RIGHT, ...)
            | -- annotations
            | -- v1.0-trainval
            | -- ...
            | -- customer
              | -- pivot-bezier
                | -- *.npz
        tools
Training and Evluation <\br>
  • a. Model Training

    bash run.sh train pivotnet_nuscenes_swint 30  # pivotnet, default: 8GPUs, bs=1, epochs=30
    bash run.sh train bemapnet_nuscenes_swint 30  # bemapnet, default: 8GPUs, bs=1, epochs=30
  • b. Model Evaluation

    bash run.sh test pivotnet_nuscenes_swint ${checkpoint-path}  # for pivotnet
    bash run.sh test bemapnet_nuscenes_swint ${checkpoint-path}  # for bemapnet
  • c. Reproduce with one command

    bash run.sh reproduce pivotnet_nuscenes_swint  # for pivotnet
    bash run.sh reproduce bemapnet_nuscenes_swint  # for bemapnet

Models & Results

Results on NuScenes Val Set <\br>

Citation

If you find PivotNet/BeMapNet/MachMap is useful in your research or applications, please consider giving us a star ⭐ and citing them by the following BibTeX entries:

@inproceedings{ding2023pivotnet,
  title={Pivotnet: Vectorized pivot learning for end-to-end hd map construction},
  author={Ding, Wenjie and Qiao, Limeng and Qiu, Xi and Zhang, Chi},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={3672--3682},
  year={2023}
}

@InProceedings{Qiao_2023_CVPR,
    author    = {Qiao, Limeng and Ding, Wenjie and Qiu, Xi and Zhang, Chi},
    title     = {End-to-End Vectorized HD-Map Construction With Piecewise Bezier Curve},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {13218-13228}
}

@article{qiao2023machmap,
    author={Limeng Qiao and Yongchao Zheng and Peng Zhang and Wenjie Ding and Xi Qiu and Xing Wei and Chi Zhang},
    title={MachMap: End-to-End Vectorized Solution for Compact HD-Map Construction}, 
    journal={arXiv preprint arXiv:2306.10301},
    year={2023},
}

About

Source code of PivotNet (ICCV2023, PivotNet: Vectorized Pivot Learning for End-to-end HD Map Construction)

Resources

License

Stars

Watchers

Forks

Packages

No packages published