Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forecast eval #9

Open
wants to merge 57 commits into
base: master
Choose a base branch
from
Open
Changes from 11 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
04dd869
Added jdmp head, added gt_instance_ids to dataset infos and passed to…
sandropapais Aug 1, 2024
5094d5f
added attention forecasting head
sandropapais Aug 3, 2024
1408955
finished forecasting model and loss implementation
sandropapais Aug 13, 2024
1051dd2
reverted changes to old config
sandropapais Aug 15, 2024
e9e6789
updated new cc run script for mini
sandropapais Aug 15, 2024
a1a31f5
removed old configs
sandropapais Aug 15, 2024
0d45250
Merge branch 'forecast_attn' into master2attnforecast
sandropapais Aug 15, 2024
6537595
Merge pull request #6 from TRAILab/master2attnforecast
sandropapais Aug 15, 2024
d87f15e
removed gt_instance_ids and updated config naming
sandropapais Aug 20, 2024
4ba4c75
updated dec num_layer hyperparameter mapping
sandropapais Aug 22, 2024
2a478f6
Merge branch 'forecast_attn' into master2forecast
sandropapais Aug 22, 2024
241cf94
Merge pull request #7 from TRAILab/master2forecast
sandropapais Aug 22, 2024
9ce3513
forecast with propogating
Aug 22, 2024
75476b9
removed propogation
Aug 23, 2024
ff76400
updated prop/noprop and qemb model and config switching
sandropapais Aug 27, 2024
52681f0
added map encoder
sandropapais Aug 29, 2024
222fea3
added new experiment configs
Aug 29, 2024
6d58a7c
modified configs
Aug 30, 2024
54ba64f
fixed rec_rotation size issue
sandropapais Aug 30, 2024
1d7618a
uppdate forecast gt for 6 seconds
sandropapais Aug 30, 2024
6e80b07
forecast eval initial commit
RobertRen1122 Aug 31, 2024
718cab0
add pgp evaluator
RobertRen1122 Aug 31, 2024
4a9546b
added map debug plot
sandropapais Sep 1, 2024
c8f6e15
updated map encoder implementation
sandropapais Sep 3, 2024
3cccc30
updated map encoder
sandropapais Sep 3, 2024
3995a0b
removed print statement
sandropapais Sep 3, 2024
7756094
"added additional forecast gt states"
sandropapais Sep 4, 2024
29bd36e
forecast evaluation
RobertRen1122 Sep 5, 2024
f80f5cb
added all states for forecasting gt and fixed predicted augmentations
sandropapais Sep 5, 2024
e437d9a
Merge branch 'forecast_attn' into master2forecast_attn
sandropapais Sep 6, 2024
8a29cd0
Merge pull request #8 from TRAILab/master2forecast_attn
sandropapais Sep 6, 2024
c86305e
added vov run configs
sandropapais Sep 6, 2024
72fd256
fixed bug with new forecast augmentation and multigpu training
sandropapais Sep 6, 2024
d9da898
fixed eval frequency
sandropapais Sep 6, 2024
28b749e
added constant velocity propogation
sandropapais Sep 17, 2024
76a61d1
added cv prop
sandropapais Sep 17, 2024
4339310
bug fix on tensor size
sandropapais Sep 17, 2024
c9e0ad3
fixed config typo, updated docker run for test
sandropapais Sep 19, 2024
7d62447
changed forecast evaluation to use nuscenes instead of pgp
sandropapais Sep 24, 2024
ab02419
reverted changes from initial branch work
sandropapais Sep 24, 2024
c2d0ec0
fixed typo in config
sandropapais Sep 24, 2024
9856965
fixed typo in config
sandropapais Sep 24, 2024
24a5040
Merge branch 'forecast_eval' into master2forecast_eval
sandropapais Sep 24, 2024
1027334
Merge pull request #10 from TRAILab/master2forecast_eval
sandropapais Sep 24, 2024
33c0639
updated cc scripts
Sep 25, 2024
8ced39d
Merge branch 'forecast_attn' of github.com:TRAILab/StreamPETR-JDMP in…
Sep 25, 2024
4eb4547
added new vov config files
sandropapais Sep 25, 2024
5ef73be
updated cc runs
Sep 25, 2024
eb5c1e1
error handling for evaluation on samples with no gt
sandropapais Sep 25, 2024
499f77e
Merge branch 'forecast_eval' of github.com:TRAILab/StreamPETR-JDMP in…
sandropapais Sep 25, 2024
65c482d
updated script and configs
sandropapais Oct 25, 2024
c9c3bb4
Merge branch 'forecast_eval' into master2eval
sandropapais Oct 25, 2024
0294b79
Merge pull request #11 from TRAILab/master2eval
sandropapais Oct 25, 2024
14f4378
updated local train mini files
sandropapais Oct 28, 2024
00d4aa8
Merge branch 'forcast_attn_new' into master2attnforecast
sandropapais Oct 28, 2024
21e13e4
Merge pull request #12 from TRAILab/master2attnforecast
sandropapais Oct 28, 2024
60450be
Merge branch 'forcast_attn_new' into forecast_eval
sandropapais Oct 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,292 @@
_base_ = [
'../../../mmdetection3d/configs/_base_/datasets/nus-3d.py',
'../../../mmdetection3d/configs/_base_/default_runtime.py'
]
log_config = dict(
interval=50,
hooks=[
dict(type='TextLoggerHook'),
dict(type='WandbLoggerHook',
init_kwargs=dict(
entity='trailab',
project='JDMP',
name='jdmp_attforecast_graddetach_qembshare_6lay_attmem_bs16_1gpu',),
interval=50)
])
backbone_norm_cfg = dict(type='LN', requires_grad=True)
plugin=True
plugin_dir='projects/mmdet3d_plugin/'

# If point cloud range is changed, the models should also change their point
# cloud range accordingly
point_cloud_range = [-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]
voxel_size = [0.2, 0.2, 8]
img_norm_cfg = dict(
mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
# For nuScenes we usually do 10-class detection
class_names = [
'car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier',
'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'
]

num_gpus = 1
batch_size = 16
num_iters_per_epoch = 28130 // (num_gpus * batch_size)
num_epochs = 60

queue_length = 1
num_frame_losses = 1
collect_keys=['lidar2img', 'intrinsics', 'extrinsics','timestamp', 'img_timestamp', 'ego_pose', 'ego_pose_inv']
input_modality = dict(
use_lidar=False,
use_camera=True,
use_radar=False,
use_map=False,
use_external=True)
model = dict(
type='Petr3D',
num_frame_head_grads=num_frame_losses,
num_frame_backbone_grads=num_frame_losses,
num_frame_losses=num_frame_losses,
use_grid_mask=True,
img_backbone=dict(
init_cfg=dict(
type='Pretrained', checkpoint="ckpts/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth",
prefix='backbone.'),
type='ResNet',
depth=50,
num_stages=4,
out_indices=(2, 3),
frozen_stages=-1,
norm_cfg=dict(type='BN2d', requires_grad=False),
norm_eval=True,
with_cp=True,
style='pytorch'),
img_neck=dict(
type='CPFPN', ###remove unused parameters
in_channels=[1024, 2048],
out_channels=256,
num_outs=2),
img_roi_head=dict(
type='FocalHead',
num_classes=10,
in_channels=256,
loss_cls2d=dict(
type='QualityFocalLoss',
use_sigmoid=True,
beta=2.0,
loss_weight=2.0),
loss_centerness=dict(type='GaussianFocalLoss', reduction='mean', loss_weight=1.0),
loss_bbox2d=dict(type='L1Loss', loss_weight=5.0),
loss_iou2d=dict(type='GIoULoss', loss_weight=2.0),
loss_centers2d=dict(type='L1Loss', loss_weight=10.0),
train_cfg=dict(
assigner2d=dict(
type='HungarianAssigner2D',
cls_cost=dict(type='FocalLossCost', weight=2.),
reg_cost=dict(type='BBoxL1Cost', weight=5.0, box_format='xywh'),
iou_cost=dict(type='IoUCost', iou_mode='giou', weight=2.0),
centers2d_cost=dict(type='BBox3DL1Cost', weight=10.0)))
),
pts_bbox_head=dict(
type='JDMPPETRHead',
num_classes=10,
in_channels=256,
num_query=300,
memory_len=512,
topk_proposals=128,
num_propagated=128,
with_ego_pos=True,
match_with_velo=False,
scalar=10, ##noise groups
noise_scale = 1.0,
dn_weight= 1.0, ##dn loss weight
split = 0.75, ###positive rate
LID=True,
with_position=True,
position_range=[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0],
code_weights = [2.0, 2.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
detect_transformer=dict(
type='PETRTemporalTransformer',
decoder=dict(
type='PETRTransformerDecoder',
return_intermediate=True,
num_layers=6,
transformerlayers=dict(
type='PETRTemporalDecoderLayer',
attn_cfgs=[
dict(
type='MultiheadAttention',
embed_dims=256,
num_heads=8,
dropout=0.1),
dict(
type='PETRMultiheadFlashAttention',
embed_dims=256,
num_heads=8,
dropout=0.1),
],
feedforward_channels=2048,
ffn_dropout=0.1,
with_cp=True, ###use checkpoint to save memory
operation_order=('self_attn', 'norm', 'cross_attn', 'norm',
'ffn', 'norm')),
)),
forecast_transformer=dict(
type='JDMPTemporalTransformer',
decoder=dict(
type='PETRTransformerDecoder',
return_intermediate=True,
num_layers=6,
transformerlayers=dict(
type='PETRTemporalDecoderLayer',
attn_cfgs=[
dict(
type='MultiheadAttention',
embed_dims=256,
num_heads=8,
dropout=0.1),
],
feedforward_channels=2048,
ffn_dropout=0.1,
with_cp=True, ###use checkpoint to save memory
operation_order=('self_attn', 'norm',
'ffn', 'norm')),
)),
bbox_coder=dict(
type='NMSFreeCoder',
post_center_range=[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0],
pc_range=point_cloud_range,
max_num=300,
voxel_size=voxel_size,
num_classes=10),
loss_cls=dict(
type='FocalLoss',
use_sigmoid=True,
gamma=2.0,
alpha=0.25,
loss_weight=2.0),
loss_bbox=dict(type='L1Loss', loss_weight=0.25),
loss_forecast=dict(type='L1Loss', loss_weight=0.25),
loss_iou=dict(type='GIoULoss', loss_weight=0.0),),
# model training and testing settings
train_cfg=dict(pts=dict(
grid_size=[512, 512, 1],
voxel_size=voxel_size,
point_cloud_range=point_cloud_range,
out_size_factor=4,
assigner=dict(
type='HungarianAssigner3D',
cls_cost=dict(type='FocalLossCost', weight=2.0),
reg_cost=dict(type='BBox3DL1Cost', weight=0.25),
iou_cost=dict(type='IoUCost', weight=0.0), # Fake cost. This is just to make it compatible with DETR head.
pc_range=point_cloud_range),)))


dataset_type = 'CustomNuScenesDataset'
data_root = './data/nuscenes/'

file_client_args = dict(backend='disk')


ida_aug_conf = {
"resize_lim": (0.38, 0.55),
"final_dim": (256, 704),
"bot_pct_lim": (0.0, 0.0),
"rot_lim": (0.0, 0.0),
"H": 900,
"W": 1600,
"rand_flip": True,
}
train_pipeline = [
dict(type='LoadMultiViewImageFromFiles', to_float32=True),
dict(type='JDMPLoadAnnotations3D', with_bbox_3d=True, with_label_3d=True, with_bbox=True,
with_label=True, with_bbox_depth=True, with_forecast=True),
dict(type='JDMPObjectRangeFilter', point_cloud_range=point_cloud_range),
dict(type='JDMPObjectNameFilter', classes=class_names),
dict(type='ResizeCropFlipRotImage', data_aug_conf = ida_aug_conf, training=True),
dict(type='GlobalRotScaleTransImage',
rot_range=[-0.3925, 0.3925],
translation_std=[0, 0, 0],
scale_ratio_range=[0.95, 1.05],
reverse_angle=True,
training=True,
),
dict(type='NormalizeMultiviewImage', **img_norm_cfg),
dict(type='PadMultiViewImage', size_divisor=32),
dict(type='JDMPFormatBundle3D', class_names=class_names, collect_keys=collect_keys + ['prev_exists']),
dict(type='Collect3D', keys=['gt_bboxes_3d', 'gt_labels_3d', 'img', 'gt_bboxes', 'gt_labels', 'centers2d', 'depths', 'prev_exists', 'gt_forecasting_locs', 'gt_forecasting_masks'] + collect_keys,
meta_keys=('filename', 'ori_shape', 'img_shape', 'pad_shape', 'scale_factor', 'flip', 'box_mode_3d', 'box_type_3d', 'img_norm_cfg', 'scene_token', 'gt_bboxes_3d','gt_labels_3d'))
]
test_pipeline = [
dict(type='LoadMultiViewImageFromFiles', to_float32=True),
dict(type='ResizeCropFlipRotImage', data_aug_conf = ida_aug_conf, training=False),
dict(type='NormalizeMultiviewImage', **img_norm_cfg),
dict(type='PadMultiViewImage', size_divisor=32),
dict(
type='MultiScaleFlipAug3D',
img_scale=(1333, 800),
pts_scale_ratio=1,
flip=False,
transforms=[
dict(
type='PETRFormatBundle3D',
collect_keys=collect_keys,
class_names=class_names,
with_label=False),
dict(type='Collect3D', keys=['img'] + collect_keys,
meta_keys=('filename', 'ori_shape', 'img_shape','pad_shape', 'scale_factor', 'flip', 'box_mode_3d', 'box_type_3d', 'img_norm_cfg', 'scene_token'))
])
]

data = dict(
samples_per_gpu=batch_size,
workers_per_gpu=4,
train=dict(
type=dataset_type,
data_root=data_root,
ann_file=data_root + 'nuscenes2d_temporal_infos_train.pkl',
num_frame_losses=num_frame_losses,
seq_split_num=2, # streaming video training
seq_mode=True, # streaming video training
pipeline=train_pipeline,
classes=class_names,
modality=input_modality,
collect_keys=collect_keys + ['img', 'prev_exists', 'img_metas'],
queue_length=queue_length,
test_mode=False,
use_valid_flag=True,
filter_empty_gt=False,
box_type_3d='LiDAR'),
val=dict(type=dataset_type, pipeline=test_pipeline, collect_keys=collect_keys + ['img', 'img_metas'], queue_length=queue_length, ann_file=data_root + 'nuscenes2d_temporal_infos_val.pkl', classes=class_names, modality=input_modality),
test=dict(type=dataset_type, pipeline=test_pipeline, collect_keys=collect_keys + ['img', 'img_metas'], queue_length=queue_length, ann_file=data_root + 'nuscenes2d_temporal_infos_val.pkl', classes=class_names, modality=input_modality),
shuffler_sampler=dict(type='InfiniteGroupEachSampleInBatchSampler'),
nonshuffler_sampler=dict(type='DistributedSampler')
)

optimizer = dict(
type='AdamW',
lr=2e-4, # bs 8: 2e-4 || bs 16: 4e-4
paramwise_cfg=dict(
custom_keys={
'img_backbone': dict(lr_mult=0.1), # set to 0.1 always better when apply 2D pretrained.
}),
weight_decay=0.01)

optimizer_config = dict(type='Fp16OptimizerHook', loss_scale='dynamic', grad_clip=dict(max_norm=35, norm_type=2))
# learning policy
lr_config = dict(
policy='CosineAnnealing',
warmup='linear',
warmup_iters=500,
warmup_ratio=1.0 / 3,
min_lr_ratio=1e-3,
)

evaluation = dict(interval=num_iters_per_epoch*num_epochs, pipeline=test_pipeline)
find_unused_parameters=False #### when use checkpoint, find_unused_parameters must be False
checkpoint_config = dict(interval=num_iters_per_epoch, max_keep_ckpts=3)
runner = dict(
type='IterBasedRunner', max_iters=num_epochs * num_iters_per_epoch)
load_from=None
resume_from=None
Loading