Speedup 05: Retrieve unique detections in family and in matched_filter
#527
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
core.match_filter.family._uniq
: 1.9x speeduplist(set)
(1.9x speedup for 43000 detections, fastest: 3.1 s), but 1.2x slower for small sets (e.g., 430 detections; 50 ms --> 27 ms).core.match_filter.detect
- 1000x speed up for many calls tofamily._uniq
family._uniq
in a loop over all families is still rather slow with_uniq
. Checking tuples of(detection.id, detection.detect_time, detection.detect_val)
withnumpy.unique
and avoiding a loop is 1000x faster. From 752 s to <1 s for 82000 detections.Why was it initiated? Any relevant Issues?
Retrieving unique detections for
matched_filter
-run was getting slower than needed when there are a lot of detections.This PR contributes to the summary issue in #522
PR Checklist
develop
base branch selected?- [ ] Any new features or fixed regressions are be covered via new tests.- [] Any new or changed features have are fully documented.CHANGES.md
.- [ ] First time contributors have added your name toCONTRIBUTORS.md
.