Refactor: PEtab code to sub-package (cont.) (#2211)

Continuation of #2205

.github/actions/setup-sonar-tools/action.yml

@@ -16,7 +16,8 @@ runs:
 
     - name: Install sonarcloud tools
       run: |
-        sudo apt-get install nodejs curl unzip \
+        sudo apt-get update \
+        && sudo apt-get install nodejs curl unzip \
         && curl --create-dirs -sSLo $HOME/.sonar/sonar-scanner.zip \
           https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-$SONAR_SCANNER_VERSION-linux.zip \
         && unzip -o $HOME/.sonar/sonar-scanner.zip -d $HOME/.sonar/ \

documentation/ExampleJax.ipynb

@@ -46,10 +46,10 @@
      "output_type": "stream",
      "text": [
       "Cloning into 'tmp/benchmark-models'...\n",
-      "remote: Enumerating objects: 336, done.\u001b[K\n",
-      "remote: Counting objects: 100% (336/336), done.\u001b[K\n",
-      "remote: Compressing objects: 100% (285/285), done.\u001b[K\n",
-      "remote: Total 336 (delta 88), reused 216 (delta 39), pack-reused 0\u001b[K\n",
+      "remote: Enumerating objects: 336, done.\u001B[K\n",
+      "remote: Counting objects: 100% (336/336), done.\u001B[K\n",
+      "remote: Compressing objects: 100% (285/285), done.\u001B[K\n",
+      "remote: Total 336 (delta 88), reused 216 (delta 39), pack-reused 0\u001B[K\n",
       "Receiving objects: 100% (336/336), 2.11 MiB | 7.48 MiB/s, done.\n",
       "Resolving deltas: 100% (88/88), done.\n"
      ]
@@ -557,8 +557,7 @@
       "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -I/Users/fabian/Documents/projects/AMICI/documentation/amici_models/Boehm_JProteomeRes2014 -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/include -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/ThirdParty/gsl -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/ThirdParty/sundials/include -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/ThirdParty/SuiteSparse/include -I/opt/homebrew/Cellar/hdf5/1.12.2_2/include -I/Users/fabian/Documents/projects/AMICI/build/venv/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c swig/Boehm_JProteomeRes2014_wrap.cpp -o build/temp.macosx-13-arm64-cpython-310/swig/Boehm_JProteomeRes2014_wrap.o -std=c++14\n",
       "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -I/Users/fabian/Documents/projects/AMICI/documentation/amici_models/Boehm_JProteomeRes2014 -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/include -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/ThirdParty/gsl -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/ThirdParty/sundials/include -I/Users/fabian/Documents/projects/AMICI/python/sdist/amici/ThirdParty/SuiteSparse/include -I/opt/homebrew/Cellar/hdf5/1.12.2_2/include -I/Users/fabian/Documents/projects/AMICI/build/venv/include -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.10/include/python3.10 -c wrapfunctions.cpp -o build/temp.macosx-13-arm64-cpython-310/wrapfunctions.o -std=c++14\n",
       "clang++ -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_Jy.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dJydsigma.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dJydy.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dJydy_colptrs.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dJydy_rowvals.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dsigmaydp.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdp.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdp_colptrs.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdp_rowvals.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdw.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdw_colptrs.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdw_rowvals.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdx.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdx_colptrs.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dwdx_rowvals.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dxdotdw.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dxdotdw_colptrs.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dxdotdw_rowvals.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_dydx.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_sigmay.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_sx0_fixedParameters.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_w.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_x0.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_x0_fixedParameters.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_x_rdata.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_x_solver.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_xdot.o build/temp.macosx-13-arm64-cpython-310/Boehm_JProteomeRes2014_y.o build/temp.macosx-13-arm64-cpython-310/swig/Boehm_JProteomeRes2014_wrap.o build/temp.macosx-13-arm64-cpython-310/wrapfunctions.o -L/opt/homebrew/Cellar/hdf5/1.12.2_2/lib -L/Users/fabian/Documents/projects/AMICI/python/sdist/amici/libs -lamici -lsundials -lsuitesparse -lcblas -lhdf5_hl_cpp -lhdf5_hl -lhdf5_cpp -lhdf5 -o /Users/fabian/Documents/projects/AMICI/documentation/amici_models/Boehm_JProteomeRes2014/Boehm_JProteomeRes2014/_Boehm_JProteomeRes2014.cpython-310-darwin.so\n",
-      "ld: warning: -undefined dynamic_lookup may not work with chained fixups\n",
-      "\n"
+      "ld: warning: -undefined dynamic_lookup may not work with chained fixups\n"
      ]
     },
     {
@@ -571,7 +570,7 @@
     }
    ],
    "source": [
-    "from amici.petab_import import import_petab_problem\n",
+    "from amici.petab.petab_import import import_petab_problem\n",
     "\n",
     "amici_model = import_petab_problem(petab_problem, force_compile=True)"
    ]
@@ -589,7 +588,7 @@
    "id": "e2ef051a",
    "metadata": {},
    "source": [
-    "For full jax support, we would have to implement a new [primitive](https://jax.readthedocs.io/en/latest/notebooks/How_JAX_primitives_work.html), which would require quite a bit of engineering, and in the end wouldn't add much benefit since AMICI can't run on GPUs. Instead will interface AMICI using the experimental jax module [`host_callback`](https://jax.readthedocs.io/en/latest/jax.experimental.host_callback.html). "
+    "For full jax support, we would have to implement a new [primitive](https://jax.readthedocs.io/en/latest/notebooks/How_JAX_primitives_work.html), which would require quite a bit of engineering, and in the end wouldn't add much benefit since AMICI can't run on GPUs. Instead, we will interface AMICI using the experimental jax module [`host_callback`](https://jax.readthedocs.io/en/latest/jax.experimental.host_callback.html). "
    ]
   },
   {
@@ -607,7 +606,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from amici.petab_objective import simulate_petab\n",
+    "from amici.petab.simulations import simulate_petab\n",
     "import amici\n",
     "\n",
     "amici_solver = amici_model.getSolver()\n",
@@ -655,7 +654,7 @@
    "id": "98e819bd",
    "metadata": {},
    "source": [
-    "Now we can finally define the JAX function that runs amici simulation using the host callback. We add a `custom_jvp` decorater so that we can define a custom jacobian vector product function in the next step. More details about custom jacobian vector product functions can be found in the [JAX documentation](https://jax.readthedocs.io/en/latest/notebooks/Custom_derivative_rules_for_Python_code.html)"
+    "Now we can finally define the JAX function that runs amici simulation using the host callback. We add a `custom_jvp` decorator so that we can define a custom jacobian vector product function in the next step. More details about custom jacobian vector product functions can be found in the [JAX documentation](https://jax.readthedocs.io/en/latest/notebooks/Custom_derivative_rules_for_Python_code.html)"
    ]
   },
   {
@@ -937,7 +936,7 @@
    "metadata": {},
    "source": [
     "We see quite some differences in the gradient calculation. The primary reason is that running JAX in default configuration will use float32 precision for the parameters that are passed to AMICI, which uses float64, and the derivative of the parameter transformation \n",
-    "As AMICI simulations that run on the CPU are the most expensive operation, there is barely any tradeoff for using float32 vs float64 in JAX. Therefore we configure JAX to use float64 instead and rerun simulations."
+    "As AMICI simulations that run on the CPU are the most expensive operation, there is barely any tradeoff for using float32 vs float64 in JAX. Therefore, we configure JAX to use float64 instead and rerun simulations."
    ]
   },
   {

documentation/GettingStarted.ipynb

@@ -14,7 +14,7 @@
    "metadata": {},
    "source": [
     "## Model Compilation\n",
-    "Before simulations can be run, the model must be imported and compiled. In this process, AMICI performs all symbolic manipulations that later enable scalable simulations and efficient sensitivity computation. The first step towards model compilation is the creation of an [SbmlImporter](https://amici.readthedocs.io/en/latest/generated/amici.sbml_import.SbmlImporter.html) instance, which requires an SBML Document that specifies the model using the [Systems Biology Markup Language (SBML)](http://sbml.org/Main_Page). \n",
+    "Before simulations can be run, the model must be imported and compiled. In this process, AMICI performs all symbolic manipulations that later enable scalable simulations and efficient sensitivity computation. The first step towards model compilation is the creation of an [SbmlImporter](https://amici.readthedocs.io/en/latest/generated/amici.sbml_import.SbmlImporter.html) instance, which requires an SBML Document that specifies the model using the [Systems Biology Markup Language (SBML)](https://sbml.org/). \n",
     "\n",
     "For the purpose of this tutorial, we will use `model_steadystate_scaled.xml`, which is contained in the same directory as this notebook."
    ]
@@ -113,7 +113,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Model simulations can be executed using the [amici.runAmiciSimulations](https://amici.readthedocs.io/en/latest/generated/amici.html#amici.runAmiciSimulation) routine. By default the model does not not contain any timepoints for which the model is to be simulated. Here we define a simulation timecourse with two timepoints at `0` and `1` and then run the simulation."
+    "Model simulations can be executed using the [amici.runAmiciSimulations](https://amici.readthedocs.io/en/latest/generated/amici.html#amici.runAmiciSimulation) routine. By default, the model does not contain any timepoints for which the model is to be simulated. Here we define a simulation timecourse with two timepoints at `0` and `1` and then run the simulation."
    ]
   },
   {

documentation/python_modules.rst

@@ -11,6 +11,14 @@ AMICI Python API
    amici.sbml_import
    amici.pysb_import
    amici.bngl_import
+   amici.petab
+   amici.petab.import_helpers
+   amici.petab.parameter_mapping
+   amici.petab.petab_import
+   amici.petab.pysb_import
+   amici.petab.sbml_import
+   amici.petab.simulations
+   amici.petab.simulator
    amici.petab_import
    amici.petab_import_pysb
    amici.petab_objective

python/examples/example_constant_species/ExampleEquilibrationLogic.ipynb

@@ -62,7 +62,7 @@
     "from IPython.display import Image\n",
     "\n",
     "fig = Image(\n",
-    "    filename=(\"../../../documentation/gfx/steadystate_solver_workflow.png\")\n",
+    "    filename=\"../../../documentation/gfx/steadystate_solver_workflow.png\"\n",
     ")\n",
     "fig"
    ]
@@ -97,12 +97,9 @@
    ],
    "source": [
     "import libsbml\n",
-    "import importlib\n",
     "import amici\n",
     "import os\n",
-    "import sys\n",
     "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
     "\n",
     "# SBML model we want to import\n",
     "sbml_file = \"model_constant_species.xml\"\n",
@@ -397,7 +394,7 @@
     "  * `-5`: Error: The model was simulated past the timepoint `t=1e100` without finding a steady state. Therefore, it is likely that the model has not steady state for the given parameter vector.\n",
     "\n",
     "Here, only the second entry of `posteq_status` contains a positive integer: The first run of Newton's method failed due to a Jacobian, which oculd not be factorized, but the second run (simulation) contains the entry 1 (success). The third entry is 0, thus Newton's method was not launched for a second time.\n",
-    "More information can be found in`posteq_numsteps`: Also here, only the second entry contains a positive integer, which is smaller than the maximum number of steps taken (<1000). Hence steady state was reached via simulation, which corresponds to the simulated time written to `posteq_time`.\n",
+    "More information can be found in`posteq_numsteps`: Also here, only the second entry contains a positive integer, which is smaller than the maximum number of steps taken (<1000). Hence, steady state was reached via simulation, which corresponds to the simulated time written to `posteq_time`.\n",
     "\n",
     "We want to demonstrate a complete failure if inferring the steady state by reducing the number of integration steps to a lower value:"
    ]
@@ -951,7 +948,7 @@
     }
    ],
    "source": [
-    "# Singluar Jacobian, use simulation\n",
+    "# Singular Jacobian, use simulation\n",
     "model.setSteadyStateSensitivityMode(\n",
     "    amici.SteadyStateSensitivityMode.integrateIfNewtonFails\n",
     ")\n",
@@ -1207,7 +1204,7 @@
     }
    ],
    "source": [
-    "# Non-singular Jacobian, use simulaiton\n",
+    "# Non-singular Jacobian, use simulation\n",
     "model_reduced.setSteadyStateSensitivityMode(\n",
     "    amici.SteadyStateSensitivityMode.integrateIfNewtonFails\n",
     ")\n",

python/examples/example_errors.ipynb

@@ -19,15 +19,16 @@
    "source": [
     "%matplotlib inline\n",
     "import os\n",
+    "from contextlib import suppress\n",
+    "from pathlib import Path\n",
+    "\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
     "import amici\n",
-    "from amici.petab_import import import_petab_problem\n",
-    "from amici.petab_objective import simulate_petab, RDATAS, EDATAS\n",
+    "from amici.petab.petab_import import import_petab_problem\n",
+    "from amici.petab.simulations import simulate_petab, RDATAS, EDATAS\n",
     "from amici.plotting import plot_state_trajectories, plot_jacobian\n",
-    "import petab\n",
-    "import numpy as np\n",
-    "import matplotlib.pyplot as plt\n",
-    "from pathlib import Path\n",
-    "from contextlib import suppress\n",
     "\n",
     "try:\n",
     "    import benchmark_models_petab\n",
@@ -153,7 +154,7 @@
     "    [amici.simulation_status_to_str(rdata.status) for rdata in res[RDATAS]],\n",
     ")\n",
     "assert all(rdata.status == amici.AMICI_SUCCESS for rdata in res[RDATAS])\n",
-    "print(\"Simulations finished succesfully.\")\n",
+    "print(\"Simulations finished successfully.\")\n",
     "print()\n",
     "\n",
     "\n",
@@ -174,7 +175,7 @@
     "    [amici.simulation_status_to_str(rdata.status) for rdata in res[RDATAS]],\n",
     ")\n",
     "assert all(rdata.status == amici.AMICI_SUCCESS for rdata in res[RDATAS])\n",
-    "print(\"Simulations finished succesfully.\")"
+    "print(\"Simulations finished successfully.\")"
    ]
   },
   {
@@ -339,7 +340,7 @@
     "**What happened?**\n",
     "\n",
     "AMICI failed to integrate the forward problem. The problem occurred for only one simulation condition, `condition_step_00_3`. The issue occurred at $t = 429.232$, where the error test failed.\n",
-    "This means, the solver is unable to take a step of non-zero size without violating the choosen error tolerances."
+    "This means, the solver is unable to take a step of non-zero size without violating the chosen error tolerances."
    ]
   },
   {
@@ -400,7 +401,7 @@
     "    [amici.simulation_status_to_str(rdata.status) for rdata in res[RDATAS]],\n",
     ")\n",
     "assert all(rdata.status == amici.AMICI_SUCCESS for rdata in res[RDATAS])\n",
-    "print(\"Simulations finished succesfully.\")"
+    "print(\"Simulations finished successfully.\")"
    ]
   },
   {
@@ -457,7 +458,7 @@
    "source": [
     "**What happened?**\n",
     "\n",
-    "The simulation failed because the initial step-size after an event or heaviside function was too small. The error occured during simulation of condition `model1_data1` after successful preequilibration (`model1_data2`)."
+    "The simulation failed because the initial step-size after an event or heaviside function was too small. The error occurred during simulation of condition `model1_data1` after successful preequilibration (`model1_data2`)."
    ]
   },
   {
@@ -646,7 +647,7 @@
    "id": "62d82971",
    "metadata": {},
    "source": [
-    "Considering that `n_par` occurrs as exponent, it's magnitude looks pretty high.\n",
+    "Considering that `n_par` occurs as exponent, it's magnitude looks pretty high.\n",
     "This term is very likely causing the problem - let's check:"
    ]
   },
@@ -909,7 +910,7 @@
    "source": [
     "**What happened?**\n",
     "\n",
-    "All given experimental conditions require pre-equilibration, i.e., finding a steady state. AMICI first tries to find a steady state using the Newton solver, if that fails, it tries simulating until steady state, if that also failes, it tries the Newton solver from the end of the simulation. In this case, all three failed. Neither Newton's method nor simulation yielded a steady state satisfying the required tolerances.\n",
+    "All given experimental conditions require pre-equilibration, i.e., finding a steady state. AMICI first tries to find a steady state using the Newton solver, if that fails, it tries simulating until steady state, if that also fails, it tries the Newton solver from the end of the simulation. In this case, all three failed. Neither Newton's method nor simulation yielded a steady state satisfying the required tolerances.\n",
     "\n",
     "This can also be seen in `ReturnDataView.preeq_status` (the three statuses corresponds to Newton \\#1, Simulation, Newton \\#2):"
    ]