Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Handle Monotonicity in Hierarchical Segmentation Metrics #409

Open
tomxi opened this issue Feb 13, 2025 · 0 comments
Open

Comments

@tomxi
Copy link

tomxi commented Feb 13, 2025

Description

The hierarchical segmentation metrics in mir_eval are defined with the assumption that input annotations has monotonic labels and boundaries. However, in practice, hierarchies often don't have montonic labels, or even boundaries (outputs of spectral clustering for example).
While the original papers only define the case for fully monotonic hierarchies, this issue proposes a additional option for another sensible way of handling cases where hierarchies are not monotonic.

To extend flexibility regarding monotonicity, I propose:

  1. Adding a monotonicity check: Before computing metrics, we should validate whether the hierarchical annotation has monotonic labels/boundaries.
  2. Handling different monotonicity cases: The _meet function in hierarchy.py currently records the deepest level where two frames has the same label (meet), however, they don't require all the previous levels to meet (garunteed by monotonic labels, but not otherwise). Giving _meet function a option to record the deepest monotonic level that two frames meet in labels will enforce monotonicity in evaluation that way. This would not change the metric for the monotonic case but provide a different definition for annotation meet matrix that takes monotonicity into account.
  3. Exposing an option to control this behavior: This option of how to build the annotation meet matrix should be elevated to the metric functions so users can specify how to handle monotonicity constraints.

Proposed Changes

  • Implement checks for monotonicity violation.
  • Introduce a new argument (e.g., monotonicity_mode) in _meet to determine how to construct the annotation meet matrix.
  • Update hierarchical segmentation metric functions to expose monotonicity_mode as a user-configurable option.
  • Update documentation and add tests for different monotonicity scenarios.

Next Steps

I will follow up with a pull request implementing these changes. Feedback on the approach and naming conventions would be appreciated! @bmcfee

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant