Seasonally Invariant Deep Transform for Visual Terrain-Relative Navigation

This is the official training code for the models presented in the Science Robotics paper: A seasonally invariant deep transform for visual terrain-relative navigation

Example Usage

We will work with a toy dataset of raw orthorectified image pairs stored in data/coregistered_images.

Environment setup

Create a python3 anaconda environment and use the requirements.txt file provided. Install any still-missing packages as needed.

pip3 install -r requirements.txt

Processing large orthorectified data

To create the training dataset, run

python createTiledDataset.py --raw_data_dir=data/coregistered_images/off --save_data_dir=data/training_pairs/off --overlap_ratio=0 --crop_width=600 --crop_height 600

python createTiledDataset.py --raw_data_dir=data/coregistered_images/on --save_data_dir=data/training_pairs/on --overlap_ratio=0 --crop_width=600 --crop_height 600

Training a deep transform for NCC-based registration

To train a deep image transform to optimize downstream image registration based on normalized cross-correlation (NCC), run

### Just as an example, use training dataset for validation

python siamese-ncc.py --exp_name=correlation-toy-example --training_data_dir=data/training_pairs/ --validation_data_dir=data/training_pairs/ --batch-size=4 --epochs=100 --device=0 --num_workers=4

Perform inference using the learned weights on a sample image via

python siamese-inference.py --data_dir data/samples/fakeplaceid_fakequad_000015_on.png --output_dir correlation-toy-example/sample-outputs --weights_path correlation-toy-example/weights/best_test_weights.pt

Training a deep transform for SIFT-based registration

To optimize the image transform for a feature matching registration objective, run

python siamese-sift.py --exp_name sift-toy-example --training_data_dir=data/training_pairs/ --validation_data_dir=data/training_pairs/ --subsamples=100 --crop_width=64 --batch-size=2 --zeta=10 --gamma=1 --epochs=100

Notes

• Training a NCC-optimized image transform should not be difficult.
• The feature-based transform is harder to train, and may require some hyperparameter tuning and loss balancing depending on your dataset (tip: start with just the detector loss and add in descriptor loss in small amounts if needed).
• For both transforms, the appearance of the transformed images may vary, depending on the characteristics (size, terrain variety, etc...) of the dataset used during training.

Working with a custom dataset

Create the dataset

Store your coregistered dataset under a directory structure like so

RAW_DATA_ROOT/
    on/
        image1.png
        image2.png
        ...
        imageN.png
    off/
        image1.png
        image2.png
        ...
        imageN.png

If the images are very large, then process then into smaller crops via createTiledDataset.py.

Citation

If you find our code or paper useful, please consider citing our paper:

@article{fragoso2021seasonally,
  title={A seasonally invariant deep transform for visual terrain-relative navigation},
  author={Fragoso, Anthony T and Lee, Connor T and McCoy, Austin S and Chung, Soon-Jo},
  journal={Science Robotics},
  volume={6},
  number={55},
  pages={eabf3320},
  year={2021},
  publisher={American Association for the Advancement of Science}
}

Acknowledgements

Our U-Net code is based on Pytorch-UNet.