Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docstrings #76

Merged
merged 11 commits into from
Jun 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions docs/source/API/client.rst

This file was deleted.

5 changes: 0 additions & 5 deletions docs/source/API/metrics.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/source/API/plots.rst

This file was deleted.

6 changes: 0 additions & 6 deletions docs/source/API/utils.rst

This file was deleted.

35 changes: 35 additions & 0 deletions docs/source/client.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Client
========

.. note::
When running the client, you can supply **either** the configuration yaml file, or the CLI arguments.
You do not need to supply both.

Use the command `diagnose -h` to view all usage of the CLI helper at any time.
Specific argument descriptions and explanations can be found on the :ref:`configuration` page.

.. code-block:: bash

usage: diagnose [-h] [--config CONFIG] [--model_path MODEL_PATH] [--model_engine {SBIModel}] [--data_path DATA_PATH] [--data_engine {H5Data,PickleData}]
[--simulator SIMULATOR] [--out_dir OUT_DIR] [--metrics [{CoverageFraction,AllSBC,LC2ST}]]
[--plots [{CDFRanks,CoverageFraction,Ranks,TARP,LC2ST,PPC}]]

options:
-h, --help show this help message and exit
--config CONFIG, -c CONFIG
.yaml file with all arguments to run.
--model_path MODEL_PATH, -m MODEL_PATH
String path to a model. Must be compatible with your model_engine choice.
--model_engine {SBIModel}, -e {SBIModel}
Way to load your model. See each module's documentation page for requirements and specifications.
--data_path DATA_PATH, -d DATA_PATH
String path to data. Must be compatible with data_engine choice.
--data_engine {H5Data,PickleData}, -g {H5Data,PickleData}
Way to load your data. See each module's documentation page for requirements and specifications.
--simulator SIMULATOR, -s SIMULATOR
String name of the simulator to use with generative metrics and plots. Must be pre-register with the `utils.register_simulator` method.
--out_dir OUT_DIR Where the results will be saved. Path need not exist, it will be created.
--metrics [{CoverageFraction,AllSBC,LC2ST}]
List of metrics to run. To not run any, supply `--metrics `
--plots [{CDFRanks,CoverageFraction,Ranks,TARP,LC2ST,PPC}]
List of plots to run. To not run any, supply `--plots `
10 changes: 6 additions & 4 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import sys

sys.path.append("../src")
sys.path.append("../src/deepdiagnostics")

# Configuration file for the Sphinx documentation builder.
#
Expand All @@ -10,7 +10,7 @@
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = "DeepDiagnostics"
project = "deepdiagnostics"
copyright = "2024, Becky Nevin, M Voetberg, Brian Nord"
author = "Becky Nevin, M Voetberg, Brian Nord"
release = "0.1.0"
Expand All @@ -23,17 +23,19 @@
"sphinx.ext.autosummary",
"sphinx.ext.napoleon",
"sphinx_autodoc_typehints",
'sphinxcontrib.bibtex'
]
bibtex_bibfiles = ['ref.bib']
napoleon_use_param = True
autodoc_default_options = {
"members": True,
}
autodoc_typehints = "description"

autoclass_content = "class"
templates_path = ["_templates"]

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = "alabaster"
html_theme = "pyramid"
html_static_path = ["_static"]
167 changes: 166 additions & 1 deletion docs/source/configuration.rst
Original file line number Diff line number Diff line change
@@ -1,2 +1,167 @@
.. _configuration:

Configuration
===============
===============

The configuration file is a `.yaml` file that makes up the majority of the settings.
It is specified by the user, and if a field is not set, it falls back to a set of pre-defined defaults.
It is split into sections to easily organize different parameters.

Specifying the Configuration
-----------------------------

* Pipeline Mode

To run diagnostics via the command line, pass the path to a yaml file to the diagnostic command.
This will run the entire set of diagnostics according to the configuration file.

.. code-block:: bash

diagnose --config path/to/your/config.yaml

* Standalone Mode
The configuration file is not strictly required for running in standalone mode,
but it can be specified to quickly access variables to avoid re-writing initialization parameters or ensure repeatability.

.. code-block:: python

from deepdiagnostics.utils.configuration import Config


Config("path/to/your/config.yaml")


Configuration Description
-----------------------

.. attribute:: common

:param out_dir: Folder where the results of program are saved. The path need not exist, it will be created if it does not.

:param temp_config: Path to a yaml to store a temporary config. Used only if some arguments are specified outside the config (eg, if using both the --config and --model_path arguments)

:param sim_location: Path to store settings for simulations. When using the register_simulator method, this is where the registered simulations are catalogued.

:param random_seed: Integer random seed to use.

.. code-block:: yaml

common:
out_dir: "./DeepDiagnosticsResources/results/"
temp_config: "./DeepDiagnosticsResources/temp/temp_config.yml"
sim_location: "DeepDiagnosticsResources/simulators"
random_seed: 42

.. attribute:: model

:param model_path: Path to stored model. Required.

:param model_engine: Loading method to use. Choose from methods listed in :ref:`models`.

.. code-block:: yaml

model:
model_path: {No Default}
model_engine: "SBIModel"

.. attribute:: data

:param data_path: Path to stored data. Required.

:param data_engine: Loading method to use. Choose from methods listed in :ref:`data`.

:param simulator: String name of the simulator. Must be pre-registered .

:param prior: Prior distribution used in training. Used if "prior" is not included in the passed data.

:param prior_kwargs: kwargs to use with the initialization of the prior

:param simulator_kwargs: kwargs to use with the initialization of the simulation

:param simulator_dimensions: If the output of the simulation is 1D (non-image) or 2D (images.)

.. code-block:: yaml

data:
data_path: {No Default}
data_engine: "H5Data"
prior: "normal"
prior_kwargs: {No Default}
simulator_kwargs: {No Default}
simulator_dimensions: 1

.. attribute:: plots_common

:param axis_spines: Show axis ticks

:param tight_layout: Minimize the space between axes and labels

:param default_colorway: String colorway to use. Choose from `matplotlib's named colorways <https://matplotlib.org/stable/users/explain/colors/colormaps.html>`_.

:param plot_style: Style sheet. Choose form `matplotlib's style sheets <https://matplotlib.org/stable/gallery/style_sheets/style_sheets_reference.html>`_.

:param parameter_labels: Name of each theta parameter to use for titling and labels. Corresponding with the dim=1 axis of theta given by data.

:param parameter_colors: Colors to use for each theta parameters when representing the parameters on the same plot.

:param line_style_cycle: Line styles that can be used (besides for solid lines, which are always used.)

:param figure_size: Default size for square figures. Will be adapted (slightly expanded) for multi-plot figures.

.. code-block:: yaml

plots_common:
axis_spines: False
tight_layout: True
default_colorway: "viridis"
plot_style: "fast"
parameter_labels: ["$m$", "$b$"]
parameter_colors: ["#9C92A3", "#0F5257"]
line_style_cycle: ["-", "-."]
figure_size: [6, 6]

.. attribute:: metrics_common

These parameters are used for every metric calculated, and for plots that require new inference to be run.

:param use_progress_bar: Show a progress bar when iteratively performing inference.

:param samples_per_inference: Number of samples used in a single iteration of inference.

:param percentiles: List of integer percentiles, for defining coverage regions.

:param number_simulations: Number of different simulations to run. Often, this means that the number of inferences performed for a metric is samples_per_inference*number_simulations

.. code-block:: yaml

metrics_common:
use_progress_bar: False
samples_per_inference: 1000
percentiles: [75, 85, 95]
number_simulations: 50


.. attribute:: plots

A dictionary of different plots to generate and their arguments.
Can be any of the implemented plots listed in :ref:`plots<plots>`
If the plots are specified with an empty dictionary, defaults from the class are used.
Defaults: ["CDFRanks", "Ranks", "CoverageFraction", "TARP", "LC2ST", "PPC"]

.. code-block:: yaml

plots:
TARP: {}


.. attribute:: metrics

A dictionary of different metrics to generate and their arguments.
Can be any of the implemented plots listed in :ref:`metrics<metrics>`
If the metrics are specified with an empty dictionary, defaults from the class are used.
Defaults: ["AllSBC", "CoverageFraction", "LC2ST"]

.. code-block:: yaml

metrics:
LC2ST: {}
7 changes: 6 additions & 1 deletion docs/source/API/data.rst → docs/source/data.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _data:

Data
======

Expand All @@ -8,4 +10,7 @@ Data
:members:

.. autoclass:: data.PickleData
:members:
:members:

.. autoclass:: data.simulator.Simulator
:members:
9 changes: 3 additions & 6 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,9 @@ Welcome to DeepDiagnostics's documentation!
configuration
plots
metrics
API/client
API/utils
API/data
API/models
API/plots
API/metrics
client
data
models

Indices and tables
==================
Expand Down
18 changes: 17 additions & 1 deletion docs/source/metrics.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,20 @@
.. _metrics:

Metrics
=========
=========

.. autoclass:: metrics.metric.Metric
:members:

.. autoclass:: metrics.AllSBC
:members: calculate

.. autoclass:: metrics.LC2ST

.. autoclass:: metrics.local_two_sample.LocalTwoSampleTest
:members: calculate

.. autoclass:: metrics.CoverageFraction
:members: calculate

.. bibliography::
2 changes: 2 additions & 0 deletions docs/source/API/models.rst → docs/source/models.rst
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
.. _models:

Models
========

Expand Down
31 changes: 30 additions & 1 deletion docs/source/plots.rst
Original file line number Diff line number Diff line change
@@ -1,4 +1,33 @@
.. _plots:

Plots
=======
=======

.. autoclass:: plots.plot.Display

.. autoclass:: plots.CDFRanks
:members: plot

.. autoclass:: plots.Ranks
:members: plot

.. autoclass:: plots.CoverageFraction
:members: plot

.. autoclass:: plots.TARP
:members: plot

.. autoclass:: plots.LC2ST
.. autoclass:: plots.local_two_sample.LocalTwoSampleTest
:members: plot

.. autoclass:: plots.PPC
:members: plot

.. autoclass:: plots.PriorPC
:members: plot

.. autoclass:: plots.Parity
:members: plot

.. bibliography::
Loading
Loading