Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics validate that the matcher is supported #170

Merged
merged 6 commits into from
Nov 26, 2024
Merged

Conversation

msschwartz21
Copy link
Collaborator

@msschwartz21 msschwartz21 commented Nov 25, 2024

Closes #83

Ultimately we are embracing a very manual/hard-coded approach. Each metric has to implement its own _validate_matcher function that can check both the name of the matcher and any associated parameters that may be relevant for that metric. For example, IOUMatcher is only valid for division metrics if one_to_one=True.

Types of Changes

What types of changes does your code introduce? Delete those that do not apply.

  • New feature or enhancement

Which topics does your change affect? Delete those that do not apply.

  • Matchers
  • Metrics

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • I have read the developer/contributing docs.
  • I have added tests that prove that my feature works in various situations or tests the bugfix (if appropriate).
  • I have checked that I maintained or improved code coverage.
  • I have checked the benchmarking action to verify that my changes did not adversely affect performance.
  • I have written docstrings and checked that they render correctly in the Read The Docs build (created after the PR is opened).
  • I have updated the general documentation including Metric descriptions and example notebooks if necessary.

- Add abstract function to base Metric
- Implement _validate_matcher for each metric
- Eliminate uses of CTCMatcher with div metrics
@msschwartz21 msschwartz21 added enhancement New feature or request topic: metrics labels Nov 25, 2024
Copy link
Collaborator

@cmalinmayor cmalinmayor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good! Thanks for implementing :) just left a few comments inline

src/traccuracy/metrics/_base.py Show resolved Hide resolved
valid_matchers = {"IOUMatcher", "CTCMatcher"}
name = matched.matcher_info["name"]

if name in valid_matchers:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could just return name in valid_matchers but this is a style thing

src/traccuracy/metrics/_divisions.py Outdated Show resolved Hide resolved
src/traccuracy/metrics/_track_overlap.py Show resolved Hide resolved
Copy link
Collaborator

@cmalinmayor cmalinmayor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with the caveat that it should be strictly greater than 0.5 for true one-to-one-ness :) but I don't feel the need to review again once that's fixed!

src/traccuracy/metrics/_divisions.py Outdated Show resolved Hide resolved
@msschwartz21 msschwartz21 merged commit a78b48c into main Nov 26, 2024
13 checks passed
@msschwartz21 msschwartz21 deleted the matcher-metric-val branch November 26, 2024 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request topic: metrics
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Validation of Matchers and Metrics that work together
2 participants