diff --git a/notebooks/example.ipynb b/notebooks/example.ipynb index f2a5cfc..17d3c0f 100644 --- a/notebooks/example.ipynb +++ b/notebooks/example.ipynb @@ -196,15 +196,46 @@ "! diagnose -h" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Running a simple example\n", + "\n", + "Minimal examples can be run by supplying a model path (and a matching model engine to load it, if using a model format that is not SBI), and data. \n", + "Below is an example for both CLI and standalone execution.\n", + "When a simulator is not included, a warning will be shown but this does not impact non-generative metrics and plots." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Warning: Simulator not loaded. Can only run non-generative metrics.\n" + ] + } + ], + "source": [ + "! diagnose --model_path ../resources/savedmodels/sbi/sbi_linear_from_data.pkl --data_path ../resources/saveddata/data_validation.h5 --plots CoverageFraction" + ] + }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using a simulator\n", "\n", - "In order to run the models, you must supply a simulator. \n", - "Simulators are all subclasses of `data.Simulator`, and need to be registered with `register_simulator` to use during runtime. \n", + "In order to run some of the metrics and plots (including `LC2ST`, `PPC` and `PriorPC`), you must supply a simulator. \n", + "If you do not supply a simulator and try to run these options in CLI a warning will be raised and the other metrics/plots will run as expected. \n", + "In standalone mode, when trying to initialize metrics and plots that require a simulator, without a simulator, a `SimulatorNotFoundError` will be raised. \n", + "\n", "\n", + "Simulators are all subclasses of `data.Simulator`, and need to be registered with `register_simulator` to use during runtime. \n", "`data.Simulator` is an abstract class that requires a `generate_context` \n", "(which takes a number of samples and returns a random sample of context the simulator uses to produce results. \n", "This can either be loaded in from a specific file, or a random distribution.) \n", diff --git a/src/deepdiagnostics/metrics/local_two_sample.py b/src/deepdiagnostics/metrics/local_two_sample.py index c85c377..6047879 100644 --- a/src/deepdiagnostics/metrics/local_two_sample.py +++ b/src/deepdiagnostics/metrics/local_two_sample.py @@ -11,6 +11,10 @@ class LocalTwoSampleTest(Metric): """ + + .. note:: + A simulator is required to run this metric. + Adapted from :cite:p:`linhart2023lc2st`. Train a classifier to verify the quality of the posterior via classifier accuracy. Produces an array of inference accuracies for the trained classier, representing the cases of either denying the null hypothesis diff --git a/src/deepdiagnostics/plots/local_two_sample.py b/src/deepdiagnostics/plots/local_two_sample.py index 2a4234c..de602a0 100644 --- a/src/deepdiagnostics/plots/local_two_sample.py +++ b/src/deepdiagnostics/plots/local_two_sample.py @@ -11,6 +11,10 @@ class LocalTwoSampleTest(Display): """ + + .. note:: + A simulator is required to run this plot. + Produce plots showing the local evaluation of a posterior estimator for a given observation. Adapted fom Linhart et. al. :cite:p:`linhart2023lc2st`. diff --git a/src/deepdiagnostics/plots/predictive_posterior_check.py b/src/deepdiagnostics/plots/predictive_posterior_check.py index 0b6bcaf..52f1e00 100644 --- a/src/deepdiagnostics/plots/predictive_posterior_check.py +++ b/src/deepdiagnostics/plots/predictive_posterior_check.py @@ -8,6 +8,10 @@ class PPC(Display): """ + + .. note:: + A simulator is required to run this plot. + Show the output of the model's generated posterior against the true values for the same context. Can show either output vs input (in 1D) or examples of simulation output (in 2D). diff --git a/src/deepdiagnostics/plots/predictive_prior_check.py b/src/deepdiagnostics/plots/predictive_prior_check.py index 8051443..a0accee 100644 --- a/src/deepdiagnostics/plots/predictive_prior_check.py +++ b/src/deepdiagnostics/plots/predictive_prior_check.py @@ -7,6 +7,10 @@ class PriorPC(Display): """ + + .. note:: + A simulator is required to run this plot. + Plot random samples of the simulator's output from samples drawn from the prior .. code-block:: python