This is a part of the bigger project to bring the more realistic pedestrian movements to CARLA. It isn't intended for fully standalone use. Please see the main project README.md or Adversarial Cases for Autonomous Vehicles (ARCANE) project website for details.
Copy the .env.template file to .env and edit the values as required.
Run the CARLA server (optional) & the containers with our code as described in [Step 2 of main project README.md](https://github.com/wielgosz-info/carla-pedestrians/blob/main/README.md#Step 2). This should result in the carla-pedestrians_video2carla_1 container running.
If you don't have the Nvidia GPU, there is CPU-only version available docker-compose.cpu.yml.
Please note that currently running CARLA server requires GPU, so without it the source_carla
carla renderers shouldn't be used, since it would result in errors.
If needed (you want to use JAADOpenPoseDataModule), inside the container run the
python src/pedestrians_video_2_carla/data/utils/jaad_annotations_xml_2_csv.pyscript to convert the JAAD annotations from XML to CSV. The output will be in /outputs/JAADOpenPoseDataModule and this is where JAADOpenPoseDataModule will look for it by default. Please note that the data module expects to find the keypoints files in /outputs/JAADOpenPoseDataModule/openpose, you need to process the dataset with OpenPose to get them.
(For now) the annotations.csv file needs to have video, frame, x1,y1, x2, y2, id, action, gender, age, group_size and speed columns, where x1,y1, x2, y2 define pedestrian bounding box, id is the pedestrian id, action is what the pedestrian is doing (since right now only the walking ones will be used) and speed is the car speed category (for now only stopped cars will be used). For now we are also only using fragments where group_size=1.
The preferred way of running is via Docker. If conda is used, in addition to creating the env from the provided environment.yml file, following steps need to be done:
-
Create and activate conda environment with the following command:
conda create -f environment.yml conda activate pedestrians
-
Install the
pedestrians_video_2_carlapackage with:COMMIT=$(git rev-parse --short HEAD) SETUPTOOLS_SCM_PRETEND_VERSION="0.0.post0.dev38+${COMMIT}.dirty" pip install -e .
-
Run
pytest teststo see if everything is working.
Please note that conda env is not actively maintained.
Full list of options is available by running inside the container:
python -m pedestrians_video_2_carla --helpPlease note that data module and model specific options may change if you switch the DataModule or Model.
python -m pedestrians_video_2_carla \
--flow=pose_lifting \
--mode=train \
--data_module_name=CarlaRecorded \
--movements_model_name=LinearAE \
--batch_size=256 \
--num_workers=32 \
--clip_length=16 \
--data_nodes=CARLA_SKELETON \
--input_nodes=CARLA_SKELETON \
--output_nodes=CARLA_SKELETON \
--max_epochs=500 \
--loss_modes=loc_2d_3d \
--renderers none \
--gpus=0,1 \
--accelerator=ddppython -m pedestrians_video_2_carla \
--flow=autoencoder \
--mode=train \
--data_module_name=CarlaRecorded \
--batch_size=256 \
--num_workers=32 \
--data_nodes=CARLA_SKELETON \
--input_nodes=CARLA_SKELETON \
--output_nodes=CARLA_SKELETON \
--loss_modes=loc_2d_3d \
--gpus=0,1 \
--accelerator=ddp \
--log_every_n_steps=16 \
--renderers source_videos input_points projection_points carla \
--source_videos_dir=/datasets/CARLA/BinarySinglePedestrian \
--source_videos_overlay_skeletons \
\
--movements_model_name=Seq2SeqEmbeddings \
--movements_output_type=pose_2d \
--max_epochs=200 \
--clip_length=16Reference skeleton data in src/pedestrians_video_2_carla/data/carla/files are extracted form CARLA project Walkers *.uasset files.
If you use this repo please cite:
@misc{wielgosz2023carlabsp,
title={{CARLA-BSP}: a simulated dataset with pedestrians},
author={Maciej Wielgosz and Antonio M. López and Muhammad Naveed Riaz},
month={May},
year={2023},
eprint={2305.00204},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Our code is released under MIT License.
This project uses (and is developed to work with) CARLA Simulator, which is released under MIT License.
This project uses videos and annotations from JAAD dataset, created by Amir Rasouli, Iuliia Kotseruba, and John K. Tsotsos, to extract pedestrians movements and attributes. The videos and annotations are released under MIT License.
This project uses OpenPose, created by Ginés Hidalgo, Zhe Cao, Tomas Simon, Shih-En Wei, Yaadhav Raaj, Hanbyul Joo, and Yaser Sheikh, to extract pedestrians skeletons from videos. OpenPose has its own licensing (basically, academic or non-profit organization noncommercial research use only).
This project uses software, models and datasets from Max-Planck Institute for Intelligent Systems, namely VPoser: Variational Human Pose Prior for Body Inverse Kinematics, Body Visualizer, Configer and Perceiving Systems Mesh Package, which have their own licenses (non-commercial scientific research purposes, see each repo for details). The models can be downloaded from "Expressive Body Capture: 3D Hands, Face, and Body from a Single Image" website. Required are the "SMPL-X with removed head bun" or other SMPL-based model that can be fed into BodyModel - right now our code utilizes only first 22 common SMPL basic joints. For VPoser, the "VPoser v2.0" model is used. Both downloaded models need to be put in models directory. If using other SMPL models, the defaults in src/pedestrians_video_2_carla/data/smpl/constants.py may need to be modified. SMPL-compatible datasets can be obtained from AMASS: Archive of Motion Capture As Surface Shapes. Each available dataset has its own license / citing requirements. During the development of this project, we mainly used CMU and Human Eva SMPL-X Gender Specific datasets.
 |
 |
 |
This project has been set up using PyScaffold 4.0.2. For details and usage information on PyScaffold see https://pyscaffold.org/.