Merge pull request #239 from dirac-institute/release_notes

Update documentation

docs/source/conf.py

@@ -6,10 +6,10 @@
 # -- Project information -----------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
-project = 'KBMoD'
-copyright = '2022, KBMoD Developers'
-author = 'KBMoD Developers'
-release = '0.9.0'
+project = 'KBMOD'
+copyright = '2022, KBMOD Developers'
+author = 'KBMOD Developers'
+release = '1.0.0'
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
@@ -32,7 +32,7 @@
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
-html_theme = 'pydata_sphinx_theme'
+html_theme = 'sphinx_rtd_theme'
 html_static_path = ['_static',]
 html_logo = "_static/kbmod.svg"
 

docs/source/fake_data_creator.rst

@@ -0,0 +1,5 @@
+Module: fake_data_creator
+=========================
+
+.. automodule:: kbmod.fake_data_creator
+   :members:

docs/source/index.rst

@@ -5,11 +5,11 @@
 
 .. image:: _static/kbmod.svg
   :width: 400
-  :alt: KBMoD logo
+  :alt: KBMOD logo
 
 KBMOD (Kernel Based Moving Object Detection) is a GPU-accelerated framework for the detection of slowly moving asteroids within sequences of images. KBMOD enables users to detect moving objects that are too dim to be detected in a single image without requiring source detection in any individual image, nor at any fixed image cadence. KBMOD achieves this by “shift-and-stacking” images for a range of asteroid velocities and orbits without requiring the input images to be transformed to correct for the asteroid motion.
 
-.. Important:: If you use KBMoD for work presented in a publication or talk please help the project via proper citation or acknowledgement **what is proper citation or acknowledgement**
+.. Important:: If you use KBMOD for work presented in a publication or talk please help the project via proper citation or acknowledgement **what is proper citation or acknowledgement** In addition, if you use the known object matching functionality please cite either `IMCCE's SkyBoT VO tool <https://vo.imcce.fr/webservices/skybot/>`_ (Berthier et. al. 2006) or JPL’s SSD (Solar System Dynamics) `API service <https://ssd.jpl.nasa.gov/>`_ as appropriate.
 
 
 Getting Started
@@ -37,18 +37,21 @@ User Documentation
    run_search_referenceapi
    search_referenceapi
    analysis_utils
-   evaluate
    image_info
    jointfit_functions
    kbmod_info
    kbmodpy
-   known_objects
+   result_list
+   fake_data_creator
 
 
 For Developers
 ==============
 
+.. toctree::
+   :maxdepth: 1
 
+   overview/release_notes
 
 Indices and tables
 ==================

docs/source/overview/input_files.rst

@@ -1,13 +1,13 @@
 Input Files
 ===========
 
-KBMoD expects Vera C. Rubin Science Pipelines calexp-style FITS files. These are multi-extension fits files that contain:
+KBMOD expects Vera C. Rubin Science Pipelines calexp-style FITS files. These are multi-extension fits files that contain:
 
 * photometrically and astometrically calibrated single-CCD image, usually refered to as the "science image",
 * variance image, representing per-pixel noise levels, and a
 * pixel bitmask
 
-stored in 1st, 2nd and 3rd header extension/plane respectively. The zeroth header extension is expected to contain the image metadata. A collection of science images that overlap the same area on the sky at different times are expected to be grouped into directories, usually refered to as "pointing groups". The path to this directory is a required input to KBMoD, see :ref:`Search Parameters`.
+stored in 1st, 2nd and 3rd header extension/plane respectively. The zeroth header extension is expected to contain the image metadata. A collection of science images that overlap the same area on the sky at different times are expected to be grouped into directories, usually refered to as "pointing groups". The path to this directory is a required input to KBMOD, see :ref:`Search Parameters`.
 
 The images are expected to be warped, i.e. geometrically transformed to a set of images with a consistent and uniform relationship between sky coordinates and image pixels on a shared pixel grid. 
 
@@ -25,7 +25,7 @@ Time file
 There are two cases where you would want to use an external time file:
 
 * when the FITS files do not contain timestamp information
-      If no file is included, kbmod will attempt to extract the timestamp from the FITS file header (in the MJD field).
+      If no file is included, KBMOD will attempt to extract the timestamp from the FITS file header (in the MJD field).
 * when you want to prefilter the files based on the parameter `mjd_lims` (see :ref:`Search Parameters`) before loading the file.
       This reduces loading time when accessing a large directory.
 

docs/source/overview/masking.rst

@@ -1,19 +1,19 @@
 Masking
 =======
 
-The kbmod algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by:
+The KBMOD algorithm uses a data mask to represent invalid pixel values that should be ignored during the search. Masking is applied by:
 
 * applying the per-image mask
 * applying a computed a global mask
 * growing the computed mask footprint
 
-As stated previously (see :ref:`Input Files`), KBMoD expects  Vera C. Rubin Science Pipelines calexp-style FITS files. Therefore each science image has an associated mask. Per-image masks are applied by checking whether the associated mask image contains one or more of the specified flags, see :py:attr:`~kbmod.run_search.run_search.default_flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`.
+As stated previously (see :ref:`Input Files`), KBMOD expects  Vera C. Rubin Science Pipelines calexp-style FITS files. Therefore each science image has an associated mask. Per-image masks are applied by checking whether the associated mask image contains one or more of the specified flags, see :py:attr:`~kbmod.run_search.run_search.default_flag_keys` attribute of :py:class:`~kbmod.run_search.run_search`.
 These values can be overwritten by providing the :py:attr:`~kbmod.run_search.run_search.flag_keys` attribute  of :py:class:`~kbmod.run_search.run_search`. Futhermore, pixels with values that exceed a threshold value will be masked if a flux threshold is provided, see `mask_threshold` key in :ref:`Search Parameters`.
 
 Global mask is computed for all images by counting how many individual masks have flagged a pixel for **any** reason in the list of allowed global masking flags, and masking it out if that count surpasses a given threshold. The number of times a pixel has to be flagged to be masked can be set by `mask_num_images` parameter (see :ref:`Search Parameters`) and the list of flags to use in global masking can be modified via the `repeated_flag_keys` parameter and is by default set to :py:attr:`~kbmod.run_search.run_search.default_repeated_flag_keys`.
 Global masking is not applied when the list of allowed masking flags is empty.
 
-After the per-image and global masks are applied to every image, kbmod grows the mask to nearby pixels. The parameters `mask_grow` (see :ref:`Search Parameters`) determines the ammount of the growth.
+After the per-image and global masks are applied to every image, KBMOD grows the mask to nearby pixels. The parameters `mask_grow` (see :ref:`Search Parameters`) determines the ammount of the growth.
 
 The provided pixel bitmask uses the following mapping between flag and bitmask values, which corresponds to the Rubin Science Pipelines mask values:
 

docs/source/overview/output_files.rst

@@ -1,7 +1,7 @@
 Output Files
 ============
 
-KBMoD outputs a range of information about the discovered trajectories. Each filename includes a user defined suffix, allowing user to easily save and compare files from different runs. Below we use SUFFIX to indicate the user-defined suffix.
+KBMOD outputs a range of information about the discovered trajectories. Each filename includes a user defined suffix, allowing user to easily save and compare files from different runs. Below we use SUFFIX to indicate the user-defined suffix.
 
 The main file that most users will want to access is `results_SUFFIX.txt`. This file contains one line for each trajectory with the trajectory information (x pixel start, y pixel start, x velocity, y velocity), the number of observations seen, the estimated flux, and the estimated likelihood.
 

docs/source/overview/overview.rst

@@ -10,13 +10,14 @@ Overview
    output_files
    results_filtering
    testing
+   release_notes
 
 
 
-Running KBMoD
+Running KBMOD
 -------------
 
-To run a search, KBMoD must be provided with
+To run a search, KBMOD must be provided with
 
 * appropriately pre-processed input data (see :ref:`Input Files`)
 * appropriate search and filter parameters (see :ref:`Masking`)
@@ -35,7 +36,7 @@ The search is initiated via the :py:class:`kbmod.run_search.run_search` class an
 Data Model
 ----------
 
-Kbmod uses an hierarchy of three nested data structures to store the image data over which it searches.
+KBMOD uses an hierarchy of three nested data structures to store the image data over which it searches.
 
 .. image:: ../_static/datamodel.png
    :alt: schematic
@@ -67,4 +68,3 @@ Reference/API
    :maxdepth: 1
 
    ../run_search_referenceapi
-   ../search_module_referenceapi

docs/source/overview/release_notes.rst

@@ -0,0 +1,51 @@
+Release Notes
+=============
+
+Version 1.0.0 (2023-02-23)
+--------------------------
+
+Version 1.0.0 represents a full refactoring and redesign of KBMOD, including major changes to the features, functionality, and performance. Significant testing and documentation have also been added to facilitate future maintenance and collaboration. Below we detail many of the individual changes included since the v0.5.0 release.
+
+**New Features**
+
+* Add ability to compare results to known objects looked up from SkyBoT or JPL (`133 <https://github.com/dirac-institute/kbmod/pull/133>`_, `198 <https://github.com/dirac-institute/kbmod/pull/198>`_)
+* Add JointFit functions (`135 <https://github.com/dirac-institute/kbmod/pull/135>`_)
+* Add the ability to load per-image PSFs from an auxiliary file (`139 <https://github.com/dirac-institute/kbmod/pull/139>`_, `161 <https://github.com/dirac-institute/kbmod/pull/161>`_)
+* Add ability to pass 8-bit or 16-bit images to GPU functions to save memory (`152 <https://github.com/dirac-institute/kbmod/pull/152>`_)
+* Make external time file optional (`165 <https://github.com/dirac-institute/kbmod/pull/165>`_)
+* Rework build system to make package pip installable (`169 <https://github.com/dirac-institute/kbmod/pull/169>`_)
+* Add Sphinx documentation (`184 <https://github.com/dirac-institute/kbmod/pull/184>`_)
+* Remove unused filtering options to simplify configuration (`197 <https://github.com/dirac-institute/kbmod/pull/197>`_)
+
+**Testing Improvements**
+
+* Many unit tests added (Various PRs)
+* Add a diff test (`124 <https://github.com/dirac-institute/kbmod/pull/124>`_)
+* Add a regression test (`144 <https://github.com/dirac-institute/kbmod/pull/144>`_)
+* Add continuous integration tests (`201 <https://github.com/dirac-institute/kbmod/pull/201>`_)
+
+**Efficiency Improvements**
+
+* Reduce internal copies (`115 <https://github.com/dirac-institute/kbmod/pull/115>`_, `119 <https://github.com/dirac-institute/kbmod/pull/119>`_)
+* Speed up growMask function (`129 <https://github.com/dirac-institute/kbmod/pull/129>`_)
+* Move growMask to GPU (`153 <https://github.com/dirac-institute/kbmod/pull/153>`_)
+* Improve handling of single/multi-threading in post processing filtering (`155 <https://github.com/dirac-institute/kbmod/pull/155>`_)
+* Skip masking functions if there are no masking keys (`164 <https://github.com/dirac-institute/kbmod/pull/164>`_)
+* Move coadded stamp generation to GPU (`179 <https://github.com/dirac-institute/kbmod/pull/179>`_, `189 <https://github.com/dirac-institute/kbmod/pull/189>`_)
+
+**Bug Fixes**
+
+* Update DBSCAN parameters (`116 <https://github.com/dirac-institute/kbmod/pull/116>`_)
+* Fix initial value bug in GPU Max Pooling (`120 <https://github.com/dirac-institute/kbmod/pull/120>`_)
+* Correctly account for number of good images in median stamp creation (`123 <https://github.com/dirac-institute/kbmod/pull/123>`_)
+* Fix unwanted append when saving layered images (`136 <https://github.com/dirac-institute/kbmod/pull/136>`_)
+* Fix median computation with masked images (`137 <https://github.com/dirac-institute/kbmod/pull/137>`_)
+* Drop masked pixels before conducting sigmaG filtering on GPU (`142 <https://github.com/dirac-institute/kbmod/pull/142>`_, `180 <https://github.com/dirac-institute/kbmod/pull/180>`_, `181 <https://github.com/dirac-institute/kbmod/pull/181>`_)
+* Reset global mask when `createGlobalMask` is created ((`164 <https://github.com/dirac-institute/kbmod/pull/164>`_)
+* Correctly apply `mask_threshold` (`164 <https://github.com/dirac-institute/kbmod/pull/164>`_)
+* Account for masked pixels in `createAveTemplate` and `simpleDifference` (`164 <https://github.com/dirac-institute/kbmod/pull/164>`_)
+* Fix bugs in on-GPU stamp generation (`182 <https://github.com/dirac-institute/kbmod/pull/182>`_)
+* Add bounds checking in `getResults` (`192 <https://github.com/dirac-institute/kbmod/pull/192>`_)
+* Check for divide by zero in clustering function (`215 <https://github.com/dirac-institute/kbmod/pull/215>`_)
+* Update pybind version (`217 <https://github.com/dirac-institute/kbmod/pull/217>`_)
+* Ignore invalidly named files during file loading (`233 <https://github.com/dirac-institute/kbmod/pull/233>`_)

docs/source/overview/results_filtering.rst

@@ -1,15 +1,15 @@
 Results analysis
 ================
 
-The output files contain the set of all trajectories discovered by KBMoD. Many of these trajectories are false positive detections, some area already known objects and, because of the way KBMoD performs the search, some are duplicates. In the following sections we describe the various steps that remove unwanted trajectories from the set of results. 
+The output files contain the set of all trajectories discovered by KBMOD. Many of these trajectories are false positive detections, some area already known objects and, because of the way KBMOD performs the search, some are duplicates. In the following sections we describe the various steps that remove unwanted trajectories from the set of results. 
 
 
 Filtering
 ---------
 
-KBMoD uses two stages of filtering to reduce the number of candidate trajectories. The first stage uses the candidate trajectory's light curve and the second uses the coadded stamp generated from the trajectory's predicted positions.
+KBMOD uses two stages of filtering to reduce the number of candidate trajectories. The first stage uses the candidate trajectory's light curve and the second uses the coadded stamp generated from the trajectory's predicted positions.
 
-During the light curve filtering phase, KBMoD computes the predicted positions at each time steps, assembles a light curve, and looks for statistical outliers along this light curve using clipped-sigmaG filtering. This function identifies outlier points along the likelihood curve and marks them as invalid points. The candidate's overall likelihood is recomputed using only the valid points. The entire candidate trajectory is filtered if less than three valid points remain or the new likelihood is below the threshold defined by the `lh_level` parameter. Additional parameters, such as `sigmaG_lims` are used to control the light curve filtering.
+During the light curve filtering phase, KBMOD computes the predicted positions at each time steps, assembles a light curve, and looks for statistical outliers along this light curve using clipped-sigmaG filtering. This function identifies outlier points along the likelihood curve and marks them as invalid points. The candidate's overall likelihood is recomputed using only the valid points. The entire candidate trajectory is filtered if less than three valid points remain or the new likelihood is below the threshold defined by the `lh_level` parameter. Additional parameters, such as `sigmaG_lims` are used to control the light curve filtering.
 
 Relevant light curve filtering parameters include:
  * `clip_negative` - Whether to remove all negative values during filtering.
@@ -30,6 +30,7 @@ Relevant stamp filtering parameters include:
  * `peak_offset` - A length 2 list indicating how far the peak is from the center of the stamp in each of the x and y dimensions.
  * `mom_lims` -  A length 5 list providing thresholds on the images moments.
  * `stamp_radius` - The radius of the stamps to use.
+
 Note that stamps are only generated and output into files if `do_stamp_filter` is set to true.
 
 
@@ -58,18 +59,18 @@ ________
 
 
 Known Object Matching
-----------------------
+---------------------
 
-Known object matching compares found trajectories against known objects from either SkyBot or the JPL Small Bodies API. Specifically, KBMoD uses the WCS in each FITS file to query the location on the sky that is covered by the image. The found trajectories are then compared against the known objects by checking their relative predicted positions in `(ra, dec)` at each timestep. Objects that are within the threshold for all timesteps are said to match. The number of known objects and matches are displayed.
+Known object matching compares found trajectories against known objects from either SkyBot or the JPL Small Bodies API. Specifically, KBMOD uses the WCS in each FITS file to query the location on the sky that is covered by the image. The found trajectories are then compared against the known objects by checking their relative predicted positions in `(ra, dec)` at each timestep. Objects that are within the threshold for all timesteps are said to match. The number of known objects and matches are displayed.
 
-Known object matching is included for debugging purposes to provide signals into whether there could be known objects in the images and KBMoD’s ability to extract them. All matching is approximate (e.g. KBMoD uses a linear trajectory model) and matching might not be comprehensive. All scientific studies should conduct their own matching analysis.
+Known object matching is included for debugging purposes to provide signals into whether there could be known objects in the images and KBMOD’s ability to extract them. All matching is approximate (e.g. KBMOD uses a linear trajectory model) and matching might not be comprehensive. All scientific studies should conduct their own matching analysis.
 
 Relevant matching parameters include:
 
 * `known_obj_thresh` - The matching threshold (in arcseconds) to use. If no threshold is provided (known_obj_thresh = None) then no matching is performed.
 * `known_obj_jpl` - Use the JPL API instead of SkyBot.
 
 Acknowledgements
-________________
+----------------
 
-The known object matching uses the IMCCE's SkyBoT VO tool (Berthier et. al. 2006) and JPL’s SSD (Solar System Dynamics) API service.
+The known object matching uses the `IMCCE's SkyBoT VO tool <https://vo.imcce.fr/webservices/skybot/>`_ (Berthier et. al. 2006) and JPL’s SSD (Solar System Dynamics) `API service <https://ssd.jpl.nasa.gov/>`_. If you use this functionality, please cite the above sources as appropriate.

docs/source/overview/search_params.rst

@@ -15,7 +15,7 @@ Search parameters are set extensively via the :py:attr:`kbmod.run_search.run_sea
 +--------------------+-----------------------------+----------------------------------------+
 | bary_dist          | None                        | The barycentric distance to use when   |
 |                    |                             | correcting the predicted positions.    |
-|                    |                             | If set to None, kbmod will not use     |
+|                    |                             | If set to None, KBMOD will not use     |
 |                    |                             | barycentric corrections.               |
 +--------------------+-----------------------------+----------------------------------------+
 | center_thresh      | 0.00                        | The minimum fraction of total flux     |