Add a KBMOD results filter for matching "known objects" #741
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a filter for matching and filtering KBMOD results to "known objects", as defined by a user-provided astropy table specifying a catalog of objects we expect to find in the KBMOD data as part of addressing #528. The catalog of known objects can either be cached information of real known objects from a service such as astroquery or a catalog of inserted synthetic fakes a user has added to the data.
Such a catalog must have columns representing an object's:
On klone/hyak loading and filtering with an approximately 750 mb catalog of cached astroquery results data corresponding to cone searches around bore sights in the DEEP search was tested, and loading and filtering took about 30 seconds. So there is room for optimization but currently would likely not be a scaling bottle neck.
The filter can be called either in the kbmod search wrokflow in src/kbmod/run_search.py where it can be called multiple times for different data sources or from any post-processing steps with a saved KBMOD Results object and its saved WCS.
The filter matches each result observation to potentially multiple objects with the user being able to apply thresholds for how close they need to be both spatially and temporally. Observations that match to known objects are then set as invalid in the Results table's "obs_valid" column, and
remove_match_obs=True
Results table filtering is applied by the filter to remove results that no longer have enough matching observations.The function returns which which known objects matched to which observation for each KBMOD result (regardless of how many observations matched and the truth value of
remove_match_obs
). This preserves as much matching information as possible for cases such as when multiple known objects intersect different parts of a result trajectory. While this PR does not provide a convenient list of which expected recovered objects were not in the KBMOD results as requested in #528, the caller of the filter has all of the information needed to construct that.An example workflow for filtering out known objects, identifying recovered fakes, and then processing potentially real results is provided below: