Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add iou_complement as a configurable alternative distance function to center_distance (for detection and tracking evaluation) #1075

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

sbarkby
Copy link

@sbarkby sbarkby commented Apr 23, 2024

This PR introduces the Intersection Over Union complement (1.0 - IOU) as an alternative distance function for determining matches and the calculation of metrics, such as MOTP. The use of this distance function has been made configurable in the detection_cvpr_2019.json and tracking_nips_2019.json config files for detection and tracking evaluation respectively. Fortunately the codebase was already setup to cater for alternative distance functions, so the coding change proposed here is minimal.

Also note that this PR poses no change to the default operation of the evaluator, it will continue to use "center_distance" as it's distance function by default.

Copy link
Collaborator

@whyekit-motional whyekit-motional left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this feature @sbarkby!

I've done an initial pass on PR (and will look deeper into your code in subsequent passes)

python-sdk/nuscenes/eval/common/utils.py Outdated Show resolved Hide resolved
python-sdk/nuscenes/eval/common/utils.py Outdated Show resolved Hide resolved
python-sdk/nuscenes/eval/common/utils.py Outdated Show resolved Hide resolved
python-sdk/nuscenes/eval/common/utils.py Outdated Show resolved Hide resolved
:return: 1 - IOU.
"""
# Do a cheaper first pass before calculating IOU i.e. check if the circles that enclose the two
# boxes overlap
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I understand, shapely's intersection method is pretty fast - can all these calculations just to do this check really save much computation time?

If the savings are not significant, I would favor having simpler code here, e.g.:

def bev_iou_complement(...) -> float:
    """..."""
    return 1.0 - bev_iou(...)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this cheap first pass provides a considerable speedup e.g. putting this in decreases the run time of the nuscenes/eval/tracking/tests/test_evaluate.py script (on my local machine) from 17min18sec to 3min46sec.

python-sdk/nuscenes/eval/detection/data_classes.py Outdated Show resolved Hide resolved
python-sdk/nuscenes/eval/detection/tests/test_utils.py Outdated Show resolved Hide resolved
python-sdk/nuscenes/eval/detection/tests/test_utils.py Outdated Show resolved Hide resolved
@sbarkby
Copy link
Author

sbarkby commented Apr 24, 2024

Hi @whyekit-motional , thanks for the review, have made the changes you suggested, let me know if you need anything else.

Cheers. :)

@sbarkby
Copy link
Author

sbarkby commented May 8, 2024

Hi @whyekit-motional, just checking in, any updates?

@sbarkby
Copy link
Author

sbarkby commented Jul 9, 2024

Hi @whyekit-motional, just checking in, any updates? Does this need to be transferred to a different reviewer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants