diff --git a/docs/source/conf.py b/docs/source/conf.py index 75825718d..2f50a2d35 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -6,10 +6,10 @@ # -- Project information ----------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information -project = 'KBMoD' -copyright = '2022, KBMoD Developers' -author = 'KBMoD Developers' -release = '0.9.0' +project = 'KBMOD' +copyright = '2022, KBMOD Developers' +author = 'KBMOD Developers' +release = '1.0.0' # -- General configuration --------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration @@ -32,7 +32,7 @@ # -- Options for HTML output ------------------------------------------------- # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output -html_theme = 'pydata_sphinx_theme' +html_theme = 'sphinx_rtd_theme' html_static_path = ['_static',] html_logo = "_static/kbmod.svg" diff --git a/docs/source/evaluate.rst b/docs/source/evaluate.rst deleted file mode 100644 index 9e04eee7c..000000000 --- a/docs/source/evaluate.rst +++ /dev/null @@ -1,5 +0,0 @@ -Module: evaluate -================== - -.. automodule:: kbmod.evaluate - :members: diff --git a/docs/source/fake_data_creator.rst b/docs/source/fake_data_creator.rst new file mode 100644 index 000000000..b009b181d --- /dev/null +++ b/docs/source/fake_data_creator.rst @@ -0,0 +1,5 @@ +Module: fake_data_creator +========================= + +.. automodule:: kbmod.fake_data_creator + :members: diff --git a/docs/source/index.rst b/docs/source/index.rst index 413e29862..49d6bea1b 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -5,11 +5,11 @@ .. image:: _static/kbmod.svg :width: 400 - :alt: KBMoD logo + :alt: KBMOD logo KBMOD (Kernel Based Moving Object Detection) is a GPU-accelerated framework for the detection of slowly moving asteroids within sequences of images. KBMOD enables users to detect moving objects that are too dim to be detected in a single image without requiring source detection in any individual image, nor at any fixed image cadence. KBMOD achieves this by “shift-and-stacking” images for a range of asteroid velocities and orbits without requiring the input images to be transformed to correct for the asteroid motion. -.. Important:: If you use KBMoD for work presented in a publication or talk please help the project via proper citation or acknowledgement **what is proper citation or acknowledgement** +.. Important:: If you use KBMOD for work presented in a publication or talk please help the project via proper citation or acknowledgement **what is proper citation or acknowledgement** In addition, if you use the known object matching functionality please cite either `IMCCE's SkyBoT VO tool `_ (Berthier et. al. 2006) or JPL’s SSD (Solar System Dynamics) `API service `_ as appropriate. Getting Started @@ -37,18 +37,21 @@ User Documentation run_search_referenceapi search_referenceapi analysis_utils - evaluate image_info jointfit_functions kbmod_info kbmodpy - known_objects + result_list + fake_data_creator For Developers ============== +.. toctree:: + :maxdepth: 1 + overview/release_notes Indices and tables ================== diff --git a/docs/source/known_objects.rst b/docs/source/known_objects.rst deleted file mode 100644 index f3ecc35b4..000000000 --- a/docs/source/known_objects.rst +++ /dev/null @@ -1,5 +0,0 @@ -Module: known_objects -================== - -.. automodule:: kbmod.known_objects - :members: diff --git a/docs/source/overview/input_files.rst b/docs/source/overview/input_files.rst index 1c7e5ded3..ff9ae9da2 100644 --- a/docs/source/overview/input_files.rst +++ b/docs/source/overview/input_files.rst @@ -1,13 +1,13 @@ Input Files =========== -KBMoD expects Vera C. Rubin Science Pipelines calexp-style FITS files. These are multi-extension fits files that contain: +KBMOD expects Vera C. Rubin Science Pipelines calexp-style FITS files. These are multi-extension fits files that contain: * photometrically and astometrically calibrated single-CCD image, usually refered to as the "science image", * variance image, representing per-pixel noise levels, and a * pixel bitmask -stored in 1st, 2nd and 3rd header extension/plane respectively. The zeroth header extension is expected to contain the image metadata. A collection of science images that overlap the same area on the sky at different times are expected to be grouped into directories, usually refered to as "pointing groups". The path to this directory is a required input to KBMoD, see :ref:`Search Parameters`. +stored in 1st, 2nd and 3rd header extension/plane respectively. The zeroth header extension is expected to contain the image metadata. A collection of science images that overlap the same area on the sky at different times are expected to be grouped into directories, usually refered to as "pointing groups". The path to this directory is a required input to KBMOD, see :ref:`Search Parameters`. The images are expected to be warped, i.e. geometrically transformed to a set of images with a consistent and uniform relationship between sky coordinates and image pixels on a shared pixel grid. @@ -25,7 +25,7 @@ Time file There are two cases where you would want to use an external time file: * when the FITS files do not contain timestamp information - If no file is included, kbmod will attempt to extract the timestamp from the FITS file header (in the MJD field). + If no file is included, KBMOD will attempt to extract the timestamp from the FITS file header (in the MJD field). * when you want to prefilter the files based on the parameter `mjd_lims` (see :ref:`Search Parameters`) before loading the file. This reduces loading time when accessing a large directory. diff --git a/docs/source/overview/masking.rst b/docs/source/overview/masking.rst index 05355d2ce..500848ac0 100644 --- a/docs/source/overview/masking.rst +++ b/docs/source/overview/masking.rst @@ -1,19 +1,19 @@ Masking ======= -The kbmod algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by: +The KBMOD algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by: * applying the per-image mask * applying a computed a global mask * growing the computed mask footprint -As stated previously (see :ref:`Input Files`), KBMoD expects Vera C. Rubin Science Pipelines calexp-style FITS files. Therefore each science image has an associated mask. Per-image masks are applied by checking whether the associated mask image contains one or more of the specified flags, see :py:attr:`~kbmod.run_search.run_search.default_flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`. +As stated previously (see :ref:`Input Files`), KBMOD expects Vera C. Rubin Science Pipelines calexp-style FITS files. Therefore each science image has an associated mask. Per-image masks are applied by checking whether the associated mask image contains one or more of the specified flags, see :py:attr:`~kbmod.run_search.run_search.default_flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`. These values can be overwritten by providing the :py:attr:`~kbmod.run_search.run_search.flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`. Futhermore, pixels with values that exceed a threshold value will be masked if a flux threshold is provided, see `mask_threshold` key in :ref:`Search Parameters`. Global mask is computed for all images by counting how many individual masks have flagged a pixel for **any** reason in the list of allowed global masking flags, and masking it out if that count surpasses a given threshold. The number of times a pixel has to be flagged to be masked can be set by `mask_num_images` parameter (see :ref:`Search Parameters`) and the list of flags to use in global masking can be modified via the `repeated_flag_keys` parameter and is by default set to :py:attr:`~kbmod.run_search.run_search.default_repeated_flag_keys`. Global masking is not applied when the list of allowed masking flags is empty. -After the per-image and global masks are applied to every image, kbmod grows the mask to nearby pixels. The parameters `mask_grow` (see :ref:`Search Parameters`) determines the ammount of the growth. +After the per-image and global masks are applied to every image, KBMOD grows the mask to nearby pixels. The parameters `mask_grow` (see :ref:`Search Parameters`) determines the ammount of the growth. The provided pixel bitmask uses the following mapping between flag and bitmask values, which corresponds to the Rubin Science Pipelines mask values: diff --git a/docs/source/overview/output_files.rst b/docs/source/overview/output_files.rst index 61f30ee43..5d7df776c 100644 --- a/docs/source/overview/output_files.rst +++ b/docs/source/overview/output_files.rst @@ -1,7 +1,7 @@ Output Files ============ -KBMoD outputs a range of information about the discovered trajectories. Each filename includes a user defined suffix, allowing user to easily save and compare files from different runs. Below we use SUFFIX to indicate the user-defined suffix. +KBMOD outputs a range of information about the discovered trajectories. Each filename includes a user defined suffix, allowing user to easily save and compare files from different runs. Below we use SUFFIX to indicate the user-defined suffix. The main file that most users will want to access is `results_SUFFIX.txt`. This file contains one line for each trajectory with the trajectory information (x pixel start, y pixel start, x velocity, y velocity), the number of observations seen, the estimated flux, and the estimated likelihood. diff --git a/docs/source/overview/overview.rst b/docs/source/overview/overview.rst index 1e8d64d78..b8301dc39 100644 --- a/docs/source/overview/overview.rst +++ b/docs/source/overview/overview.rst @@ -10,13 +10,14 @@ Overview output_files results_filtering testing + release_notes -Running KBMoD +Running KBMOD ------------- -To run a search, KBMoD must be provided with +To run a search, KBMOD must be provided with * appropriately pre-processed input data (see :ref:`Input Files`) * appropriate search and filter parameters (see :ref:`Masking`) @@ -35,7 +36,7 @@ The search is initiated via the :py:class:`kbmod.run_search.run_search` class an Data Model ---------- -Kbmod uses an hierarchy of three nested data structures to store the image data over which it searches. +KBMOD uses an hierarchy of three nested data structures to store the image data over which it searches. .. image:: ../_static/datamodel.png :alt: schematic @@ -67,4 +68,3 @@ Reference/API :maxdepth: 1 ../run_search_referenceapi - ../search_module_referenceapi diff --git a/docs/source/overview/release_notes.rst b/docs/source/overview/release_notes.rst new file mode 100644 index 000000000..f6958b38e --- /dev/null +++ b/docs/source/overview/release_notes.rst @@ -0,0 +1,51 @@ +Release Notes +============= + +Version 1.0.0 (2023-02-23) +-------------------------- + +Version 1.0.0 represents a full refactoring and redesign of KBMOD, including major changes to the features, functionality, and performance. Significant testing and documentation have also been added to facilitate future maintenance and collaboration. Below we detail many of the individual changes included since the v0.5.0 release. + +**New Features** + +* Add ability to compare results to known objects looked up from SkyBoT or JPL (`133 `_, `198 `_) +* Add JointFit functions (`135 `_) +* Add the ability to load per-image PSFs from an auxiliary file (`139 `_, `161 `_) +* Add ability to pass 8-bit or 16-bit images to GPU functions to save memory (`152 `_) +* Make external time file optional (`165 `_) +* Rework build system to make package pip installable (`169 `_) +* Add Sphinx documentation (`184 `_) +* Remove unused filtering options to simplify configuration (`197 `_) + +**Testing Improvements** + +* Many unit tests added (Various PRs) +* Add a diff test (`124 `_) +* Add a regression test (`144 `_) +* Add continuous integration tests (`201 `_) + +**Efficiency Improvements** + +* Reduce internal copies (`115 `_, `119 `_) +* Speed up growMask function (`129 `_) +* Move growMask to GPU (`153 `_) +* Improve handling of single/multi-threading in post processing filtering (`155 `_) +* Skip masking functions if there are no masking keys (`164 `_) +* Move coadded stamp generation to GPU (`179 `_, `189 `_) + +**Bug Fixes** + +* Update DBSCAN parameters (`116 `_) +* Fix initial value bug in GPU Max Pooling (`120 `_) +* Correctly account for number of good images in median stamp creation (`123 `_) +* Fix unwanted append when saving layered images (`136 `_) +* Fix median computation with masked images (`137 `_) +* Drop masked pixels before conducting sigmaG filtering on GPU (`142 `_, `180 `_, `181 `_) +* Reset global mask when `createGlobalMask` is created ((`164 `_) +* Correctly apply `mask_threshold` (`164 `_) +* Account for masked pixels in `createAveTemplate` and `simpleDifference` (`164 `_) +* Fix bugs in on-GPU stamp generation (`182 `_) +* Add bounds checking in `getResults` (`192 `_) +* Check for divide by zero in clustering function (`215 `_) +* Update pybind version (`217 `_) +* Ignore invalidly named files during file loading (`233 `_) diff --git a/docs/source/overview/results_filtering.rst b/docs/source/overview/results_filtering.rst index c7c08a33c..9be8eedb2 100644 --- a/docs/source/overview/results_filtering.rst +++ b/docs/source/overview/results_filtering.rst @@ -1,15 +1,15 @@ Results analysis ================ -The output files contain the set of all trajectories discovered by KBMoD. Many of these trajectories are false positive detections, some area already known objects and, because of the way KBMoD performs the search, some are duplicates. In the following sections we describe the various steps that remove unwanted trajectories from the set of results. +The output files contain the set of all trajectories discovered by KBMOD. Many of these trajectories are false positive detections, some area already known objects and, because of the way KBMOD performs the search, some are duplicates. In the following sections we describe the various steps that remove unwanted trajectories from the set of results. Filtering --------- -KBMoD uses two stages of filtering to reduce the number of candidate trajectories. The first stage uses the candidate trajectory's light curve and the second uses the coadded stamp generated from the trajectory's predicted positions. +KBMOD uses two stages of filtering to reduce the number of candidate trajectories. The first stage uses the candidate trajectory's light curve and the second uses the coadded stamp generated from the trajectory's predicted positions. -During the light curve filtering phase, KBMoD computes the predicted positions at each time steps, assembles a light curve, and looks for statistical outliers along this light curve using clipped-sigmaG filtering. This function identifies outlier points along the likelihood curve and marks them as invalid points. The candidate's overall likelihood is recomputed using only the valid points. The entire candidate trajectory is filtered if less than three valid points remain or the new likelihood is below the threshold defined by the `lh_level` parameter. Additional parameters, such as `sigmaG_lims` are used to control the light curve filtering. +During the light curve filtering phase, KBMOD computes the predicted positions at each time steps, assembles a light curve, and looks for statistical outliers along this light curve using clipped-sigmaG filtering. This function identifies outlier points along the likelihood curve and marks them as invalid points. The candidate's overall likelihood is recomputed using only the valid points. The entire candidate trajectory is filtered if less than three valid points remain or the new likelihood is below the threshold defined by the `lh_level` parameter. Additional parameters, such as `sigmaG_lims` are used to control the light curve filtering. Relevant light curve filtering parameters include: * `clip_negative` - Whether to remove all negative values during filtering. @@ -30,6 +30,7 @@ Relevant stamp filtering parameters include: * `peak_offset` - A length 2 list indicating how far the peak is from the center of the stamp in each of the x and y dimensions. * `mom_lims` - A length 5 list providing thresholds on the images moments. * `stamp_radius` - The radius of the stamps to use. + Note that stamps are only generated and output into files if `do_stamp_filter` is set to true. @@ -58,11 +59,11 @@ ________ Known Object Matching ----------------------- +--------------------- -Known object matching compares found trajectories against known objects from either SkyBot or the JPL Small Bodies API. Specifically, KBMoD uses the WCS in each FITS file to query the location on the sky that is covered by the image. The found trajectories are then compared against the known objects by checking their relative predicted positions in `(ra, dec)` at each timestep. Objects that are within the threshold for all timesteps are said to match. The number of known objects and matches are displayed. +Known object matching compares found trajectories against known objects from either SkyBot or the JPL Small Bodies API. Specifically, KBMOD uses the WCS in each FITS file to query the location on the sky that is covered by the image. The found trajectories are then compared against the known objects by checking their relative predicted positions in `(ra, dec)` at each timestep. Objects that are within the threshold for all timesteps are said to match. The number of known objects and matches are displayed. -Known object matching is included for debugging purposes to provide signals into whether there could be known objects in the images and KBMoD’s ability to extract them. All matching is approximate (e.g. KBMoD uses a linear trajectory model) and matching might not be comprehensive. All scientific studies should conduct their own matching analysis. +Known object matching is included for debugging purposes to provide signals into whether there could be known objects in the images and KBMOD’s ability to extract them. All matching is approximate (e.g. KBMOD uses a linear trajectory model) and matching might not be comprehensive. All scientific studies should conduct their own matching analysis. Relevant matching parameters include: @@ -70,6 +71,6 @@ Relevant matching parameters include: * `known_obj_jpl` - Use the JPL API instead of SkyBot. Acknowledgements -________________ +---------------- -The known object matching uses the IMCCE's SkyBoT VO tool (Berthier et. al. 2006) and JPL’s SSD (Solar System Dynamics) API service. +The known object matching uses the `IMCCE's SkyBoT VO tool `_ (Berthier et. al. 2006) and JPL’s SSD (Solar System Dynamics) `API service `_. If you use this functionality, please cite the above sources as appropriate. \ No newline at end of file diff --git a/docs/source/overview/search_params.rst b/docs/source/overview/search_params.rst index 630d118c6..28e31f10a 100644 --- a/docs/source/overview/search_params.rst +++ b/docs/source/overview/search_params.rst @@ -15,7 +15,7 @@ Search parameters are set extensively via the :py:attr:`kbmod.run_search.run_sea +--------------------+-----------------------------+----------------------------------------+ | bary_dist | None | The barycentric distance to use when | | | | correcting the predicted positions. | -| | | If set to None, kbmod will not use | +| | | If set to None, KBMOD will not use | | | | barycentric corrections. | +--------------------+-----------------------------+----------------------------------------+ | center_thresh | 0.00 | The minimum fraction of total flux | diff --git a/docs/source/overview/testing.rst b/docs/source/overview/testing.rst index eb74702f2..2c63ccbf6 100644 --- a/docs/source/overview/testing.rst +++ b/docs/source/overview/testing.rst @@ -1,13 +1,13 @@ Testing ======= -KBMoD provides multiple mechanisms for testing the code and confirming that code changes do not break the code or change the behavior. These include comprehensive unit tests, a regression test, and a diff test. +KBMOD provides multiple mechanisms for testing the code and confirming that code changes do not break the code or change the behavior. These include comprehensive unit tests, a regression test, and a diff test. Unit Tests ---------- -KBMoD’s unit tests are included in the `/test` directory and start with the prefix `test_`. The unit test suite can be run using: +KBMOD’s unit tests are included in the `/test` directory and start with the prefix `test_`. The unit test suite can be run using: ```python -m unittest``` @@ -17,7 +17,7 @@ from within that directory. Regression Test --------------- -The regression test generates fake data by creating images with Gaussian noise, inserting fake objects, writing those images to temporary files, calling KBMoD on the temporary files, and comparing the candidate trajectories with the fake objects inserted. The regression test passes if all inserted trajectories are recovered. +The regression test generates fake data by creating images with Gaussian noise, inserting fake objects, writing those images to temporary files, calling KBMOD on the temporary files, and comparing the candidate trajectories with the fake objects inserted. The regression test passes if all inserted trajectories are recovered. You can run the regression test from the `/tests` directory using: ```python regression_test.py``` @@ -31,11 +31,11 @@ Additional command line arguments allow you to change the behavior: Diff Test --------- -The diff test is provided to allow users to compare the results of KBMoD runs on real world data sets. This diff test requires that you have already downloaded or created a preprocessed data set (with warped images). The parameter `data_filepath` is used to point to a directory of these files. The diff test operates by running the full KBMoD algorithm and comparing the results of the output files. The diff test passes if the output files are identical within a very small noise threshold (to account for floating point errors). +The diff test is provided to allow users to compare the results of KBMOD runs on real world data sets. This diff test requires that you have already downloaded or created a preprocessed data set (with warped images). The parameter `data_filepath` is used to point to a directory of these files. The diff test operates by running the full KBMOD algorithm and comparing the results of the output files. The diff test passes if the output files are identical within a very small noise threshold (to account for floating point errors). Before running the diff test on new code, you must first generate golden files that will be used for comparisons. Build a clean version of the code and run the diff test using the `generate_goldens` flag. This will save the run’s output files to a `goldens` directory. -Once the golden files are generated, you can test new code by running the diff test without the `generate_goldens` flag. This will run KBMoD, save the results to a temporary directory, and compare the results and coadded stamp files to the ones in the goldens directory. +Once the golden files are generated, you can test new code by running the diff test without the `generate_goldens` flag. This will run KBMOD, save the results to a temporary directory, and compare the results and coadded stamp files to the ones in the goldens directory. The diff test takes a few parameters: * `data_filepath` - The location of the data files on which to test the code. diff --git a/docs/source/result_list.rst b/docs/source/result_list.rst new file mode 100644 index 000000000..93ef7e6a5 --- /dev/null +++ b/docs/source/result_list.rst @@ -0,0 +1,5 @@ +Module: result_list +=================== + +.. automodule:: kbmod.result_list + :members: diff --git a/setup.cfg b/setup.cfg index 96ed209cd..b632c117a 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,6 +1,6 @@ [metadata] name = kbmod -author = KBMoD Developers +author = KBMOD Developers description = Moving object detection library implemented on GPUs. long_description = file: README.md license = BSD 2_Clause Simplified License @@ -56,6 +56,6 @@ analysis = docs = sphinx numpydoc - pydata-sphinx-theme + sphinx_rtd_theme tests = black[jupyter] \ No newline at end of file