Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup 05: Retrieve unique detections in family and in matched_filter #527

Conversation

flixha
Copy link
Collaborator

@flixha flixha commented Dec 12, 2022

What does this PR do?

  • core.match_filter.family._uniq: 1.9x speedup
    • retrieving the unique list of detections is quicker for many detections with list(set) (1.9x speedup for 43000 detections, fastest: 3.1 s), but 1.2x slower for small sets (e.g., 430 detections; 50 ms --> 27 ms).
  • core.match_filter.detect - 1000x speed up for many calls to family._uniq
    • using family._uniq in a loop over all families is still rather slow with _uniq. Checking tuples of (detection.id, detection.detect_time, detection.detect_val) with numpy.unique and avoiding a loop is 1000x faster. From 752 s to <1 s for 82000 detections.

Why was it initiated? Any relevant Issues?

Retrieving unique detections for matched_filter-run was getting slower than needed when there are a lot of detections.

This PR contributes to the summary issue in #522

PR Checklist

  • develop base branch selected?
  • This PR is not directly related to an existing issue (which has no PR yet).
  • All tests still pass.
    - [ ] Any new features or fixed regressions are be covered via new tests.
    - [] Any new or changed features have are fully documented.
  • Significant changes have been added to CHANGES.md.
    - [ ] First time contributors have added your name to CONTRIBUTORS.md.

Copy link
Member

@calum-chamberlain calum-chamberlain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the changes to _uniq shouldn't work, and if they do then we should make sure they don't... Otherwise this is a really useful speedup!

eqcorrscan/core/match_filter/family.py Outdated Show resolved Hide resolved
eqcorrscan/core/match_filter/tribe.py Show resolved Hide resolved
Copy link
Member

@calum-chamberlain calum-chamberlain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Golden, thanks!

@calum-chamberlain calum-chamberlain merged commit 8e5ca9d into eqcorrscan:develop Jan 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants