Merge pull request #85 from deepskies/sim_required_docs_84

Warning in docs for metrics that cannot run without a simulator

notebooks/example.ipynb

@@ -196,15 +196,46 @@
     "! diagnose -h"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Running a simple example\n",
+    "\n",
+    "Minimal examples can be run by supplying a model path (and a matching model engine to load it, if using a model format that is not SBI), and data. \n",
+    "Below is an example for both CLI and standalone execution.\n",
+    "When a simulator is not included, a warning will be shown but this does not impact non-generative metrics and plots."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Warning: Simulator not loaded. Can only run non-generative metrics.\n"
+     ]
+    }
+   ],
+   "source": [
+    "! diagnose --model_path ../resources/savedmodels/sbi/sbi_linear_from_data.pkl --data_path ../resources/saveddata/data_validation.h5  --plots CoverageFraction"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Using a simulator\n",
     "\n",
-    "In order to run the models, you must supply a simulator. \n",
-    "Simulators are all subclasses of `data.Simulator`, and need to be registered with `register_simulator` to use during runtime. \n",
+    "In order to run some of the metrics and plots (including `LC2ST`, `PPC` and `PriorPC`), you must supply a simulator. \n",
+    "If you do not supply a simulator and try to run these options in CLI a warning will be raised and the other metrics/plots will run as expected. \n",
+    "In standalone mode, when trying to initialize metrics and plots that require a simulator, without a simulator, a `SimulatorNotFoundError` will be raised. \n",
+    "\n",
     "\n",
+    "Simulators are all subclasses of `data.Simulator`, and need to be registered with `register_simulator` to use during runtime. \n",
     "`data.Simulator` is an abstract class that requires a `generate_context` \n",
     "(which takes a number of samples and returns a random sample of context the simulator uses to produce results. \n",
     "This can either be loaded in from a specific file, or a random distribution.) \n",

src/deepdiagnostics/metrics/local_two_sample.py

@@ -11,6 +11,10 @@
 
 class LocalTwoSampleTest(Metric): 
     """
+
+        .. note:: 
+            A simulator is required to run this metric.
+
         Adapted from :cite:p:`linhart2023lc2st`. 
         Train a classifier to verify the quality of the posterior via classifier accuracy. 
         Produces an array of inference accuracies for the trained classier, representing the cases of either denying the null hypothesis 

src/deepdiagnostics/plots/local_two_sample.py

@@ -11,6 +11,10 @@
 
 class LocalTwoSampleTest(Display): 
     """
+
+    .. note:: 
+        A simulator is required to run this plot.
+        
     Produce plots showing the local evaluation of a posterior estimator for a given observation. 
     Adapted fom Linhart et. al. :cite:p:`linhart2023lc2st`.
 

src/deepdiagnostics/plots/predictive_posterior_check.py

@@ -8,6 +8,10 @@
 
 class PPC(Display):
     """
+
+        .. note:: 
+            A simulator is required to run this plot.
+
         Show the output of the model's generated posterior against the true values for the same context. 
         Can show either output vs input (in 1D) or examples of simulation output (in 2D). 
 

src/deepdiagnostics/plots/predictive_prior_check.py

@@ -7,6 +7,10 @@
 
 class PriorPC(Display):
     """
+
+    .. note:: 
+        A simulator is required to run this plot.
+
     Plot random samples of the simulator's output from samples drawn from the prior
 
     .. code-block:: python