Skip to content

Commit

Permalink
Warning in docs for metrics that cannot run without a simulator
Browse files Browse the repository at this point in the history
  • Loading branch information
voetberg committed Jul 22, 2024
1 parent 35a14ee commit 2bef6da
Show file tree
Hide file tree
Showing 5 changed files with 49 additions and 2 deletions.
35 changes: 33 additions & 2 deletions notebooks/example.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -196,15 +196,46 @@
"! diagnose -h"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Running a simple example\n",
"\n",
"Minimal examples can be run by supplying a model path (and a matching model engine to load it, if using a model format that is not SBI), and data. \n",
"Below is an example for both CLI and standalone execution.\n",
"When a simulator is not included, a warning will be shown but this does not impact non-generative metrics and plots."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Warning: Simulator not loaded. Can only run non-generative metrics.\n"
]
}
],
"source": [
"! diagnose --model_path ../resources/savedmodels/sbi/sbi_linear_from_data.pkl --data_path ../resources/saveddata/data_validation.h5 --plots CoverageFraction"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Using a simulator\n",
"\n",
"In order to run the models, you must supply a simulator. \n",
"Simulators are all subclasses of `data.Simulator`, and need to be registered with `register_simulator` to use during runtime. \n",
"In order to run some of the metrics and plots (including `LC2ST`, `PPC` and `PriorPC`), you must supply a simulator. \n",
"If you do not supply a simulator and try to run these options in CLI a warning will be raised and the other metrics/plots will run as expected. \n",
"In standalone mode, when trying to initialize metrics and plots that require a simulator, without a simulator, a `SimulatorNotFoundError` will be raised. \n",
"\n",
"\n",
"Simulators are all subclasses of `data.Simulator`, and need to be registered with `register_simulator` to use during runtime. \n",
"`data.Simulator` is an abstract class that requires a `generate_context` \n",
"(which takes a number of samples and returns a random sample of context the simulator uses to produce results. \n",
"This can either be loaded in from a specific file, or a random distribution.) \n",
Expand Down
4 changes: 4 additions & 0 deletions src/deepdiagnostics/metrics/local_two_sample.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@

class LocalTwoSampleTest(Metric):
"""
.. note::
A simulator is required to run this metric.
Adapted from :cite:p:`linhart2023lc2st`.
Train a classifier to verify the quality of the posterior via classifier accuracy.
Produces an array of inference accuracies for the trained classier, representing the cases of either denying the null hypothesis
Expand Down
4 changes: 4 additions & 0 deletions src/deepdiagnostics/plots/local_two_sample.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@

class LocalTwoSampleTest(Display):
"""
.. note::
A simulator is required to run this plot.
Produce plots showing the local evaluation of a posterior estimator for a given observation.
Adapted fom Linhart et. al. :cite:p:`linhart2023lc2st`.
Expand Down
4 changes: 4 additions & 0 deletions src/deepdiagnostics/plots/predictive_posterior_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@

class PPC(Display):
"""
.. note::
A simulator is required to run this plot.
Show the output of the model's generated posterior against the true values for the same context.
Can show either output vs input (in 1D) or examples of simulation output (in 2D).
Expand Down
4 changes: 4 additions & 0 deletions src/deepdiagnostics/plots/predictive_prior_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@

class PriorPC(Display):
"""
.. note::
A simulator is required to run this plot.
Plot random samples of the simulator's output from samples drawn from the prior
.. code-block:: python
Expand Down

0 comments on commit 2bef6da

Please sign in to comment.