Invalid objects warning occured when loading waymo training data

[zerlinwang]

Hi Alan, Thank you for your excellent work!
I encountered a problem when I wanna train wayformer on the waymo dataset. It would be grateful if you can offer any help.

What is the problem

When I run python train.py
The outputs are

Seed set to 42
Loading training data...
Using 64 processes to load data...
Warning: obj_idx=177 is not valid at time step 9, scene_id=a796796f4a048d62
0/7610 data processed
Warning: obj_idx=40 is not valid at time step 9, scene_id=8702fb2a32bb71b1
Warning: obj_idx=26 is not valid at time step 9, scene_id=ec8bfc734c194445
Warning: obj_idx=4 is not valid at time step 9, scene_id=386f1987ce0ac975
Warning: obj_idx=12 is not valid at time step 9, scene_id=2ffcf84b9ccbc646
Warning: obj_idx=159 is not valid at time step 9, scene_id=a90b538976da7b95
Warning: obj_idx=13 is not valid at time step 9, scene_id=69eb1d8e88941a82
Warning: obj_idx=33 is not valid at time step 9, scene_id=4e833e33f9ea0e48
Warning: obj_idx=32 is not valid at time step 9, scene_id=c0903535ea655ea6
Warning: obj_idx=9 is not valid at time step 9, scene_id=ee28e59c2b44efd1
Warning: obj_idx=18 is not valid at time step 9, scene_id=4a266beef119d131
Warning: obj_idx=159 is not valid at time step 9, scene_id=1bb2559a10e6ec0c
Warning: obj_idx=15 is not valid at time step 9, scene_id=c409b46c149703b7
Warning: obj_idx=3 is not valid at time step 9, scene_id=c409b46c149703b7
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_e75022217efdf6bf.pkl
Warning: obj_idx=43 is not valid at time step 9, scene_id=c12a9f85df574fa5
Warning: obj_idx=2 is not valid at time step 9, scene_id=a59b2c32e340b2a3
Warning: no center objects at time step 9, scene_id=a59b2c32e340b2a3
Warning: 'NoneType' object is not iterable in sd_waymo_v1.2_a59b2c32e340b2a3.pkl
Warning: obj_idx=106 is not valid at time step 9, scene_id=715ad5c57bdcc6e
Warning: obj_idx=4 is not valid at time step 9, scene_id=859d80bbf53923a8
Warning: obj_idx=55 is not valid at time step 9, scene_id=9e22b91b294d3a12
Warning: obj_idx=11 is not valid at time step 9, scene_id=97800e3c6548cd85
Warning: obj_idx=19 is not valid at time step 9, scene_id=258587cdee55e595
Warning: obj_idx=2 is not valid at time step 9, scene_id=fc268c0790864a3f
Warning: obj_idx=112 is not valid at time step 9, scene_id=bfba0ae3b01a9d37
...

How to reproduce the problem

1. Download waymo open motion dataset v1.2.0;

gsutil -m cp -r \
  "gs://waymo_open_dataset_motion_v_1_2_0/uncompressed/scenario/training" \
  "gs://waymo_open_dataset_motion_v_1_2_0/uncompressed/scenario/validation" \
  .

2. Process the dataset with ScenarioNet

python -m scenarionet.convert_waymo -d /path/to/unitraj/data_samples/waymo --raw_data_path /path/to/Waymo_v1.2/training --num_workers 64
python -m scenarionet.convert_waymo -d /path/to/unitraj/data_samples/waymo_val --raw_data_path /path/to/Waymo_v1.2/validation --num_workers 64

3. Train

My config.yaml is

# exp setting
exp_name: 'waymo_debug' # name that will be used in wandb
ckpt_path: null # checkpoint path, used for evaluation
seed: 42 # random seed
debug: False # debug mode, will use cpu only
devices: [ 0, 1, 2, 3 ] # gpu ids

# data related
load_num_workers: 8 # number of workers for loading data
train_data_path: [ "data_samples/waymo" ] # list of paths to the training data
val_data_path: [ "data_samples/waymo_val" ] # list of paths to the validation data
cache_path: "./cache"
max_data_num: [ null ] # maximum number of data for each training dataset, null means all data
starting_frame: [ 0 ] # history trajectory starts at this frame for each training dataset
past_len: 10 # history trajectory length, 2.1s
future_len: 80 # future trajectory length, 6s
object_type: [ 'VEHICLE' , 'PEDESTRIAN', 'CYCLIST'] # object types included in the training set
line_type: [ 'lane','stop_sign','road_edge','road_line','crosswalk','speed_bump' ] # line type to be considered in the input
masked_attributes: [ 'z_axis', 'size' ] # attributes to be masked in the input
trajectory_sample_interval: 1 # sample interval for the trajectory
only_train_on_ego: False # only train on AV
center_offset_of_map: [ 30.0, 0.0 ] # center offset of the map
use_cache: False # use cache for data loading
overwrite_cache: True # overwrite existing cache
store_data_in_memory: False # store data in memory

# official evaluation
# nuscenes_dataroot: '/mnt/nas3_rcp_enac_u0900_vita_scratch/datasets/Prediction-Dataset/nuscenes/nuscenes_root'
eval_nuscenes: False # whether to evaluate with nuscenes evaluation tool
eval_waymo: False # whether to evaluate with waymo evaluation tool
eval_argoverse2: False # whether to evaluate with argoverse evaluation tool

defaults:
  - method: wayformer

Then run python train.py

Possible reasons

1. The version of waymo dataset:
   I find that some people use v1.2.0 while other use v1.2.1 or v1.1.0. In addition, could I ask uncompressed/scenario/training and uncompressed/scenario/training_20s, which one is what we need?
2. Problems in my config.yaml
3. This is actually not a problem.

[Alan-LanFeng]

Hi,

the past_len should be set as 11 for Waymo official split. Can you try again?

[zerlinwang]

Hi Alan, thank you for your reply. After setting the past_len=11, the output was:

Seed set to 42                                                                                                                                                               
Loading training data...                                                                                                                                                     
Using 64 processes to load data...                                                                                                                                           
0/7610 data processed                                                                                                                                                        
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_e75022217efdf6bf.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_dd601c11d0c5c533.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_89f5f3bfcc69c563.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_4124d5ed7578138e.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_97060d12ad9366fb.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_2eb685c7caf8d082.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_fd800f4ac842b394.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_1384d925f2468856.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_87ac32ed8c4bdfa1.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_74b169a0e1eb617a.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_3a74a2b074b26a0c.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_b5f926e4a40bc446.pkl                                                                               
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_87dd0163cbd198b3.pkl                                                                            
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_76895172f40c4f67.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_b8e6e2e8736d50c5.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_b839720835af7116.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_2cd38954abaddbb0.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_67811ca4b2ea12d1.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_371ba4878ddc05a3.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_277927e328bdf914.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_ce3c4dd7fa4188e2.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_9b7879c050b7fbd2.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_e3976888355f406f.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_268b533682c60ed5.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_51693c81509fb649.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_45cb4670438f104e.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_a0a03dd38638bb84.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_324be8c95d2a6b66.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_5562f4f607f6ec32.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_8733974e00c59297.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_162def94bc5f6c62.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_5a9e9f1f125dba0f.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_82b4e554c130c874.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_5fa21a5565450c15.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_bc3b1deb708fe3d9.pkl
Warning: index 0 is out of bounds for axis 0 with size 0 in sd_waymo_v1.2_f1fdd1083cc502e9.pkl
...

• Could you tell me which version of Waymo Dataset do you use? 1.2 or 1.1?
• In addition, the data loading process seems consume more than 250GB memory, is it normal?

Thank you for your time on this matter.

[Alan-LanFeng]

1. We are using 1.1
2. The memory usage is normal because of multiprocessing.

The result is not normal. I will check the code. Thanks for pointing out.