Docstrings

docs/source/client.rst

@@ -0,0 +1,35 @@
+Client 
+========
+
+.. note:: 
+    When running the client, you can supply **either** the configuration yaml file, or the CLI arguments. 
+    You do not need to supply both. 
+
+Use the command `diagnose -h` to view all usage of the CLI helper at any time.  
+Specific argument descriptions and explanations can be found on the :ref:`configuration` page. 
+
+.. code-block:: bash
+
+    usage: diagnose [-h] [--config CONFIG] [--model_path MODEL_PATH] [--model_engine {SBIModel}] [--data_path DATA_PATH] [--data_engine {H5Data,PickleData}]
+                    [--simulator SIMULATOR] [--out_dir OUT_DIR] [--metrics [{CoverageFraction,AllSBC,LC2ST}]]
+                    [--plots [{CDFRanks,CoverageFraction,Ranks,TARP,LC2ST,PPC}]]
+
+    options:
+    -h, --help            show this help message and exit
+    --config CONFIG, -c CONFIG
+                            .yaml file with all arguments to run.
+    --model_path MODEL_PATH, -m MODEL_PATH
+                            String path to a model. Must be compatible with your model_engine choice.
+    --model_engine {SBIModel}, -e {SBIModel}
+                            Way to load your model. See each module's documentation page for requirements and specifications.
+    --data_path DATA_PATH, -d DATA_PATH
+                            String path to data. Must be compatible with data_engine choice.
+    --data_engine {H5Data,PickleData}, -g {H5Data,PickleData}
+                            Way to load your data. See each module's documentation page for requirements and specifications.
+    --simulator SIMULATOR, -s SIMULATOR
+                            String name of the simulator to use with generative metrics and plots. Must be pre-register with the `utils.register_simulator` method.
+    --out_dir OUT_DIR     Where the results will be saved. Path need not exist, it will be created.
+    --metrics [{CoverageFraction,AllSBC,LC2ST}]
+                            List of metrics to run. To not run any, supply `--metrics `
+    --plots [{CDFRanks,CoverageFraction,Ranks,TARP,LC2ST,PPC}]
+                            List of plots to run. To not run any, supply `--plots `

docs/source/conf.py

@@ -1,6 +1,6 @@
 import sys
 
-sys.path.append("../src")
+sys.path.append("../src/deepdiagnostics")
 
 # Configuration file for the Sphinx documentation builder.
 #
@@ -10,7 +10,7 @@
 # -- Project information -----------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
 
-project = "DeepDiagnostics"
+project = "deepdiagnostics"
 copyright = "2024, Becky Nevin, M Voetberg, Brian Nord"
 author = "Becky Nevin, M Voetberg, Brian Nord"
 release = "0.1.0"
@@ -23,17 +23,19 @@
     "sphinx.ext.autosummary",
     "sphinx.ext.napoleon",
     "sphinx_autodoc_typehints",
+    'sphinxcontrib.bibtex'
 ]
+bibtex_bibfiles = ['ref.bib']
 napoleon_use_param = True
 autodoc_default_options = {
     "members": True,
 }
 autodoc_typehints = "description"
-
+autoclass_content = "class"
 templates_path = ["_templates"]
 
 # -- Options for HTML output -------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
-html_theme = "alabaster"
+html_theme = "pyramid"
 html_static_path = ["_static"]

docs/source/configuration.rst

@@ -1,2 +1,167 @@
+.. _configuration:
+
 Configuration 
-===============
+===============
+
+The configuration file is a `.yaml` file that makes up the majority of the settings. 
+It is specified by the user, and if a field is not set, it falls back to a set of pre-defined defaults. 
+It is split into sections to easily organize different parameters.
+
+Specifying the Configuration 
+-----------------------------
+
+* Pipeline Mode 
+
+To run diagnostics via the command line, pass the path to a yaml file to the diagnostic command. 
+This will run the entire set of diagnostics according to the configuration file. 
+
+.. code-block:: bash 
+
+    diagnose --config path/to/your/config.yaml
+
+* Standalone Mode 
+The configuration file is not strictly required for running in standalone mode, 
+but it can be specified to quickly access variables to avoid re-writing initialization parameters or ensure repeatability. 
+
+.. code-block:: python 
+
+    from deepdiagnostics.utils.configuration import Config 
+
+
+    Config("path/to/your/config.yaml")
+
+
+Configuration Description 
+-----------------------
+
+.. attribute:: common
+
+    :param out_dir: Folder where the results of program are saved. The path need not exist, it will be created if it does not.
+
+    :param temp_config: Path to a yaml to store a temporary config. Used only if some arguments are specified outside the config (eg, if using both the --config and --model_path arguments)
+
+    :param sim_location: Path to store settings for simulations. When using the register_simulator method, this is where the registered simulations are catalogued. 
+
+    :param random_seed: Integer random seed to use. 
+
+.. code-block:: yaml 
+
+    common: 
+        out_dir: "./DeepDiagnosticsResources/results/"
+        temp_config: "./DeepDiagnosticsResources/temp/temp_config.yml"
+        sim_location: "DeepDiagnosticsResources/simulators"
+        random_seed: 42
+
+.. attribute:: model
+
+    :param model_path: Path to stored model. Required. 
+
+    :param model_engine: Loading method to use. Choose from methods listed in :ref:`models`.
+
+.. code-block:: yaml 
+
+    model: 
+        model_path: {No Default}
+        model_engine: "SBIModel"
+
+.. attribute:: data
+
+    :param data_path: Path to stored data. Required.
+
+    :param data_engine: Loading method to use. Choose from methods listed in :ref:`data`.
+
+    :param simulator: String name of the simulator. Must be pre-registered .
+
+    :param prior: Prior distribution used in training. Used if "prior" is not included in the passed data. 
+
+    :param prior_kwargs: kwargs to use with the initialization of the prior
+
+    :param simulator_kwargs: kwargs to use with the initialization of the simulation
+
+    :param simulator_dimensions: If the output of the simulation is 1D (non-image) or 2D (images.)
+
+.. code-block:: yaml 
+
+    data: 
+        data_path: {No Default}
+        data_engine: "H5Data"
+        prior: "normal"
+        prior_kwargs: {No Default}
+        simulator_kwargs: {No Default}
+        simulator_dimensions: 1
+
+.. attribute:: plots_common
+
+    :param axis_spines: Show axis ticks
+
+    :param tight_layout:  Minimize the space between axes and labels
+
+    :param default_colorway:  String colorway to use. Choose from `matplotlib's named colorways <https://matplotlib.org/stable/users/explain/colors/colormaps.html>`_.
+
+    :param plot_style: Style sheet. Choose form `matplotlib's style sheets <https://matplotlib.org/stable/gallery/style_sheets/style_sheets_reference.html>`_.
+
+    :param parameter_labels: Name of each theta parameter to use for titling and labels. Corresponding with the dim=1 axis of theta given by data. 
+
+    :param parameter_colors: Colors to use for each theta parameters when representing the parameters on the same plot. 
+
+    :param line_style_cycle: Line styles that can be used (besides for solid lines, which are always used.)
+
+    :param figure_size: Default size for square figures. Will be adapted (slightly expanded) for multi-plot figures.
+
+.. code-block:: yaml 
+
+    plots_common: 
+        axis_spines: False
+        tight_layout: True
+        default_colorway: "viridis"
+        plot_style: "fast"
+        parameter_labels: ["$m$", "$b$"]
+        parameter_colors: ["#9C92A3", "#0F5257"]
+        line_style_cycle: ["-", "-."]
+        figure_size: [6, 6]
+
+.. attribute:: metrics_common
+
+    These parameters are used for every metric calculated, and for plots that require new inference to be run. 
+
+    :param use_progress_bar: Show a progress bar when iteratively performing inference. 
+
+    :param samples_per_inference: Number of samples used in a single iteration of inference. 
+
+    :param percentiles: List of integer percentiles, for defining coverage regions. 
+
+    :param number_simulations: Number of different simulations to run. Often, this means that the number of inferences performed for a metric is samples_per_inference*number_simulations
+
+.. code-block:: yaml
+
+    metrics_common:
+        use_progress_bar: False
+        samples_per_inference: 1000
+        percentiles: [75, 85, 95]
+        number_simulations: 50
+
+
+.. attribute:: plots
+
+    A dictionary of different plots to generate and their arguments. 
+    Can be any of the implemented plots listed in :ref:`plots<plots>`
+    If the plots are specified with an empty dictionary, defaults from the class are used.
+    Defaults: ["CDFRanks", "Ranks", "CoverageFraction", "TARP", "LC2ST", "PPC"]
+
+.. code-block:: yaml
+
+    plots: 
+        TARP: {} 
+
+
+.. attribute:: metrics
+
+    A dictionary of different metrics to generate and their arguments. 
+    Can be any of the implemented plots listed in :ref:`metrics<metrics>`
+    If the metrics are specified with an empty dictionary, defaults from the class are used.
+    Defaults: ["AllSBC", "CoverageFraction", "LC2ST"]
+
+.. code-block:: yaml
+
+    metrics: 
+        LC2ST: {} 

docs/source/API/data.rst → docs/source/data.rst

@@ -1,3 +1,5 @@
+.. _data:
+
 Data 
 ======
 
@@ -8,4 +10,7 @@ Data
     :members:
 
 .. autoclass:: data.PickleData
-    :members:
+    :members:
+
+.. autoclass:: data.simulator.Simulator
+    :members: 

docs/source/index.rst

@@ -14,12 +14,9 @@ Welcome to DeepDiagnostics's documentation!
    configuration
    plots
    metrics
-   API/client
-   API/utils 
-   API/data
-   API/models 
-   API/plots 
-   API/metrics
+   client
+   data
+   models 
 
 Indices and tables
 ==================

docs/source/metrics.rst

@@ -1,4 +1,20 @@
 .. _metrics:
 
 Metrics
-=========
+=========
+
+.. autoclass:: metrics.metric.Metric
+    :members:
+
+.. autoclass:: metrics.AllSBC
+    :members: calculate
+
+.. autoclass:: metrics.LC2ST
+
+.. autoclass:: metrics.local_two_sample.LocalTwoSampleTest
+    :members: calculate 
+
+.. autoclass:: metrics.CoverageFraction
+    :members: calculate
+
+.. bibliography:: 

docs/source/API/models.rst → docs/source/models.rst

@@ -1,3 +1,5 @@
+.. _models:
+
 Models 
 ========
 

docs/source/plots.rst

@@ -1,4 +1,33 @@
 .. _plots:
 
 Plots 
-=======
+=======
+
+.. autoclass:: plots.plot.Display
+
+.. autoclass:: plots.CDFRanks
+    :members: plot
+
+.. autoclass:: plots.Ranks
+    :members: plot
+
+.. autoclass:: plots.CoverageFraction
+    :members: plot
+
+.. autoclass:: plots.TARP
+    :members: plot
+
+.. autoclass:: plots.LC2ST
+.. autoclass:: plots.local_two_sample.LocalTwoSampleTest
+    :members: plot
+
+.. autoclass:: plots.PPC
+    :members: plot
+
+.. autoclass:: plots.PriorPC
+    :members: plot
+
+.. autoclass:: plots.Parity
+    :members: plot
+
+.. bibliography::