You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The hierarchical segmentation metrics in mir_eval are defined with the assumption that input annotations has monotonic labels and boundaries. However, in practice, hierarchies often don't have montonic labels, or even boundaries (outputs of spectral clustering for example).
While the original papers only define the case for fully monotonic hierarchies, this issue proposes a additional option for another sensible way of handling cases where hierarchies are not monotonic.
To extend flexibility regarding monotonicity, I propose:
Adding a monotonicity check: Before computing metrics, we should validate whether the hierarchical annotation has monotonic labels/boundaries.
Handling different monotonicity cases: The _meet function in hierarchy.py currently records the deepest level where two frames has the same label (meet), however, they don't require all the previous levels to meet (garunteed by monotonic labels, but not otherwise). Giving _meet function a option to record the deepest monotonic level that two frames meet in labels will enforce monotonicity in evaluation that way. This would not change the metric for the monotonic case but provide a different definition for annotation meet matrix that takes monotonicity into account.
Exposing an option to control this behavior: This option of how to build the annotation meet matrix should be elevated to the metric functions so users can specify how to handle monotonicity constraints.
Proposed Changes
Implement checks for monotonicity violation.
Introduce a new argument (e.g., monotonicity_mode) in _meet to determine how to construct the annotation meet matrix.
Update hierarchical segmentation metric functions to expose monotonicity_mode as a user-configurable option.
Update documentation and add tests for different monotonicity scenarios.
Next Steps
I will follow up with a pull request implementing these changes. Feedback on the approach and naming conventions would be appreciated! @bmcfee
The text was updated successfully, but these errors were encountered:
Description
The hierarchical segmentation metrics in
mir_eval
are defined with the assumption that input annotations has monotonic labels and boundaries. However, in practice, hierarchies often don't have montonic labels, or even boundaries (outputs of spectral clustering for example).While the original papers only define the case for fully monotonic hierarchies, this issue proposes a additional option for another sensible way of handling cases where hierarchies are not monotonic.
To extend flexibility regarding monotonicity, I propose:
_meet
function inhierarchy.py
currently records the deepest level where two frames has the same label (meet), however, they don't require all the previous levels to meet (garunteed by monotonic labels, but not otherwise). Giving_meet
function a option to record the deepest monotonic level that two frames meet in labels will enforce monotonicity in evaluation that way. This would not change the metric for the monotonic case but provide a different definition for annotation meet matrix that takes monotonicity into account.Proposed Changes
monotonicity_mode
) in_meet
to determine how to construct the annotation meet matrix.monotonicity_mode
as a user-configurable option.Next Steps
I will follow up with a pull request implementing these changes. Feedback on the approach and naming conventions would be appreciated! @bmcfee
The text was updated successfully, but these errors were encountered: