New metabolic pathway

README.md

@@ -222,9 +222,9 @@ To use the cached sim data file, set the `CACHED_SIM_DATA` environment variable
 DESC="Example run with cached sim data." CACHED_SIM_DATA=1 python runscripts/fireworks/fw_queue.py
 ```
 
-### Using an interactive Sherlock node to run a Fireworks workflow
+### Using an interactive Sherlock connection to run a Fireworks workflow
 
-To run queued simulations on an interactive Sherlock node:
+To run queued simulations on an interactive connection to a Sherlock compute node:
 
 ```bash
 rlaunch rapidfire
@@ -237,7 +237,7 @@ Don't do this on a Sherlock login node.
 
 ### Using the SLURM scheduler on Linux to run a Fireworks workflow
 
-To run simulations on a Sherlock cluster (helpful when running more than one simulation):
+To run simulations on a Sherlock compute cluster (helpful when running more than one simulation):
 
 ```bash
 qlaunch -r rapidfire --nlaunches infinite --sleep 5
@@ -246,7 +246,7 @@ qlaunch -r rapidfire --nlaunches infinite --sleep 5
 The `qlaunch` command will run forever. Hit `Ctrl-C` to kill it once the console
 logs shows that all the simulation and analysis steps have finished.
 
-`qlaunch` is relatively lightweight, so it might work on a Sherlock login node.
+`qlaunch` is lightweight so it might be fine on a Sherlock login node.
 
 `qlaunch` will create block directories with stdout and stderr from each Firework.  To troubleshoot any errors or just to see the output you would normally see from an interactive session, use the following commands to search the block directories for your desired fw_id:
 ```bash

cloud/build-runtime.sh

@@ -1,25 +1,28 @@
 #!/bin/sh
 # Use Google Cloud Build servers to build a personalized "${ID}-wcm-runtime"
-# Docker Image and store it in the Google Container Registry.
+# Docker Image and store it in a Google Artifact Registry.
 #
 # COMMAND LINE ARGUMENTS:
 #   ARG1: Distinguishing ID prefix for the "${ID}-wcm-runtime" tag for the
-#     Docker Image to build in GCR; defaults to "${USER}". Must be in lower case.
+#     Docker Image to build; defaults to "${USER}". Must be in lower case.
 #
 # ASSUMES: The current working dir is the wcEcoli/ project root.
 
 set -eu
 
 ID="${1:-$USER}"
 
-PROJECT="$(gcloud config get-value core/project)"
+PROJECT="$(gcloud config get core/project)"
+REGION="$(gcloud config get compute/region)"
 WCM_RUNTIME="${ID}-wcm-runtime"
-TAG="gcr.io/${PROJECT}/${WCM_RUNTIME}"
+TAG="${REGION}-docker.pkg.dev/${PROJECT}/wcm/${WCM_RUNTIME}"
 
 echo "=== Cloud-building WCM runtime Docker Image: ${TAG} ==="
 
 # This needs only one payload file so copy it in rather than using a config at
 # the project root which would upload the entire project.
+# --region sets the Cloud Build region while TAG sets the artifact region et al.
 cp requirements.txt cloud/docker/runtime/
-gcloud builds submit --timeout=3h --tag "${TAG}" cloud/docker/runtime/
+gcloud builds submit --timeout=3h --region="${REGION}" \
+    --tag="${TAG}" cloud/docker/runtime/
 rm cloud/docker/runtime/requirements.txt

cloud/build-wcm.sh

@@ -1,20 +1,21 @@
 #!/bin/sh
 # Use Google Cloud Build servers to build a personalized "${ID}-wcm-code" Docker
-# Image and store it in the Google Container Registry.
+# Image and store it in a Google Artifact Registry.
 #
 # COMMAND LINE ARGUMENTS:
 #   ARG1: Distinguishing ID prefix for the "wcm-code" Docker Image tag;
 #     defaults to "${USER}".
-#   ARG2: Docker tag for the wcm-runtime Image in GCR to build FROM; defaults to
-#     "${ID}-wcm-runtime".
-#     The named Docker image must already exist in the GCR project.
+#   ARG2: Docker tag for the wcm-runtime Image in Artifact Registry to build FROM;
+#     defaults to "${ID}-wcm-runtime".
+#     The named Docker Image must already exist in the Artifact Registry.
 #
 # ASSUMES: The current working dir is the wcEcoli/ project root.
 
 set -eu
 
 ID="${1:-$USER}"
 
+REGION="$(gcloud config get compute/region)"
 WCM_RUNTIME="${2:-${ID}-wcm-runtime}"
 WCM_CODE="${ID}-wcm-code"
 GIT_HASH=$(git rev-parse HEAD)
@@ -29,7 +30,10 @@ echo "=== git hash ${GIT_HASH}, git branch ${GIT_BRANCH} ==="
 
 # This needs a config file to identify the project files to upload and the
 # Dockerfile to run.
+# --region sets the Cloud Build region.
 gcloud builds submit --timeout=15m --config config-build-2-wcm-code.json \
-    --substitutions="_WCM_RUNTIME=${WCM_RUNTIME},_WCM_CODE=${WCM_CODE},_GIT_HASH=${GIT_HASH},_GIT_BRANCH=${GIT_BRANCH},_TIMESTAMP=${TIMESTAMP}"
+    --region="${REGION}" \
+    --substitutions="_WCM_RUNTIME=${WCM_RUNTIME},_WCM_CODE=${WCM_CODE},\
+_GIT_HASH=${GIT_HASH},_GIT_BRANCH=${GIT_BRANCH},_TIMESTAMP=${TIMESTAMP}"
 
 rm source-info/git_diff.txt

cloud/build.sh

@@ -1,6 +1,6 @@
 #!/bin/sh
 # Use Google Cloud Build servers to build layered WCM Docker Images and store
-# them in the Google Container Registry.
+# them in a Google Artifact Registry in the current region.
 #
 # ASSUMES: The current working dir is the wcEcoli/ project root.
 

cloud/docker/runtime/Dockerfile

@@ -5,7 +5,7 @@
 #
 #     > docker build -f cloud/docker/runtime/Dockerfile -t ${USER}-wcm-runtime .
 #
-# (To build using the Cloud Build service and store in the Container Registry,
+# (To build using the Cloud Build service and store in an Artifact Registry,
 # run `cloud/build.sh`.)
 #
 # Add option `--build-arg from=ABC` to build from a different base image "ABC"
@@ -53,10 +53,10 @@ ARG NO_AVX2=0
 ENV NO_AVX2="$NO_AVX2"
 
 # Install OpenBLAS for numpy, scipy, and Aesara.
-ENV OPENBLAS_LABEL=v0.3.23
+ENV OPENBLAS_LABEL=v0.3.27
 RUN if [ -f ~/.numpy-site.cfg ]; then \
     (mkdir -p openblas && cd openblas \
-    && curl -SL https://github.com/xianyi/OpenBLAS/archive/${OPENBLAS_LABEL}.tar.gz | tar -xz \
+    && curl -SL https://github.com/OpenMathLib/OpenBLAS/archive/${OPENBLAS_LABEL}.tar.gz | tar -xz \
     && cd OpenBLAS* \
     && echo "Compiling OpenBLAS ${OPENBLAS_LABEL} with NO_AVX2=${NO_AVX2}" \
     && make "NO_AVX2=${NO_AVX2}" FC=gfortran \

cloud/docker/wholecell/Dockerfile

@@ -9,7 +9,7 @@
 # (If you want to build container image #3 on top of the locally-built base,
 # you'll have to edit that Dockerfile or upload this image.)
 #
-# (To build using the Cloud Build service and store in the Container Registry,
+# (To build using the Cloud Build service and store in an Artifact Registry,
 # run `cloud/build.sh`.)
 #
 # After building locally you can start up a new container from the image:
@@ -18,8 +18,9 @@
 #
 # or if you used build.sh or build-wcm.sh to build using Cloud Build:
 #
-#     > PROJECT="$(gcloud config get-value core/project)"
-#     > TAG="gcr.io/${PROJECT}/${USER}-wcm-code"
+#     > PROJECT="$(gcloud config get core/project)"
+#     > REGION="$(gcloud config get compute/region)"
+#     > TAG="${REGION}-docker.pkg.dev/${PROJECT}/wcm/${USER}-wcm-code"
 #     > docker run --name wholecelltest -it --rm "${TAG}"
 #
 # It will start a shell where you can execute commands:
@@ -46,7 +47,8 @@ LABEL application="Whole Cell Model of Escherichia coli" \
     email="[email protected]" \
     license="https://github.com/CovertLab/WholeCellEcoliRelease/blob/master/LICENSE.md" \
     organization="Covert Lab at Stanford" \
-    website="https://www.covert.stanford.edu/"
+    website="https://www.covert.stanford.edu/" \
+    repository="https://github.com/CovertLab/wcEcoli"
 
 COPY . /wcEcoli
 WORKDIR /wcEcoli

config-build-2-wcm-code.json

@@ -4,17 +4,17 @@
         "name": "gcr.io/cloud-builders/docker",
         "args": [
             "build",
-            "--build-arg", "from=gcr.io/${PROJECT_ID}/${_WCM_RUNTIME}",
+            "--build-arg", "from=${LOCATION}-docker.pkg.dev/${PROJECT_ID}/wcm/${_WCM_RUNTIME}",
             "--build-arg", "git_hash=${_GIT_HASH}",
             "--build-arg", "git_branch=${_GIT_BRANCH}",
             "--build-arg", "timestamp=${_TIMESTAMP}",
-            "-t", "gcr.io/${PROJECT_ID}/${_WCM_CODE}",
+            "-t", "${LOCATION}-docker.pkg.dev/${PROJECT_ID}/wcm/${_WCM_CODE}",
             "-f", "cloud/docker/wholecell/Dockerfile",
             "."
         ]
     }
     ],
     "images": [
-        "gcr.io/${PROJECT_ID}/${_WCM_CODE}"
+        "${LOCATION}-docker.pkg.dev/${PROJECT_ID}/wcm/${_WCM_CODE}"
     ]
 }

docs/README.md

@@ -43,16 +43,16 @@ Docker mechanics.
 
    ```shell script
    cloud/build.sh
-   docker pull gcr.io/${PROJECT}/${USER}-wcm-code
-   docker run --name=wcm -it --rm gcr.io/${PROJECT}/${USER}-wcm-code
+   docker pull ${REGION}-docker.pkg.dev/${PROJECT}/wcm/${USER}-wcm-code
+   docker run --name=wcm -it --rm ${REGION}-docker.pkg.dev/${PROJECT}/wcm/${USER}-wcm-code
    ```
 
    **NOTE:** It's better to build the Docker Image on Linux (e.g. in Google Cloud
    Build or WSL2) than on macOS where `build-containers-locally.sh` has to work around
    a Docker bug by disabling AVX2 instructions in OpenBLAS. That change somehow
    makes the computation produce different results and run a little slower.
    An Image built on Linux won't have the AVX2 workaround and yet it runs fine on macOS.
-   See [xianyi/OpenBLAS#2244](https://github.com/xianyi/OpenBLAS/issues/2244#issuecomment-696510557).
+   See [OpenMathLib/OpenBLAS#2244](https://github.com/OpenMathLib/OpenBLAS/issues/2244#issuecomment-696510557).
 
    Once you have a Docker Image, you can run the model's Python programs inside the Container.
    (PyCharm Pro should support debugging into a Docker Container but we haven't tested that.)
@@ -100,7 +100,7 @@ Docker mechanics.
    * [Required development tools](dev-tools.md) -- installation and tips
    * [Creating the "pyenv" runtime environment](create-pyenv.md)
    * [Setting up to run FireWorks](../wholecell/fireworks/README.md) -- needed only to run a FireWorks workflow of cell simulations and analysis plots
-   * [Set up Zookeeper and Kafka](../agent/README.md) -- needed only for multi-scale agents
+   * [Maintaining the shared Sherlock runtime environment](maintaining_the_sherlock_environment.md) -- how to update the shared virtualenv on Sherlock
 
 * **Also**
 
@@ -131,6 +131,7 @@ Docker mechanics.
 
 * [Simultaneous cross-evaluation of heterogeneous _E. coli_ datasets via mechanistic simulation](https://science.sciencemag.org/content/369/6502/eaav3751.full), _Science_, 24 July 2020
 * [A Whole-Cell Computational Model Predicts Phenotype from Genotype](https://www.cell.com/cell/abstract/S0092-8674(12)00776-3), _Cell_, July 20, 2012
+* TODO: Many more...
 
 ## dissertations
 * _Computational Simulations of Whole Cells: Strategies for Framework Design and Model Parameterization_, John Mason

docs/background.md

@@ -50,8 +50,8 @@ Scripts used as entry points for executing workflows or performing analysis.
     * `jenkins/`: scripts to test codebase through continuous integration (CI)
     * `manual/`: scripts for running portions of workflows interactively
     * `debug/`: scripts useful for debugging issues like output differences and inspecting `sim_data`
-    * `cloud/wcm.py`: used to make Docker fireworks workflows to run locally or on Google compute engine
-    * `fireworks/fw_queue.py`: used to make fireworks workflows to run locally or on Sherlock
+    * `cloud/wcm.py`: used to build Docker FireWorks workflows to run locally or on Google compute engine
+    * `fireworks/fw_queue.py`: used to build FireWorks workflows to run locally or on Sherlock
     * `tools/`: development tools
 
 ## Analysis

docs/create-pyenv.md

@@ -6,7 +6,7 @@ Before you begin all the steps below to install the Python runtime for the Whole
 
 The script `cloud/build-containers-locally.sh` will build a Docker Image on
 your computer if you have Docker Engine or Docker Desktop installed.
-The script `cloud/build.sh` will do the same using Google Container Registry if
+The script `cloud/build.sh` will do the same using Google Artifact Registry if
 you have a Google Cloud project set up.
 
 Either way, you can run the Whole Cell Model (WCM) in a Docker Container, isolated from your computer's operating system, Python versions, binary libraries, and everything else installed.
@@ -58,9 +58,12 @@ This page goes through the Python environment setup steps in more detail and wit
    **On macOS**
 
    ```bash
-   brew install glpk openssl readline swig suite-sparse xz
+   brew install cmake glpk openssl readline swig suite-sparse xz
    ```
 
+   Use the `cmake` native package rather than the `cmake` pip, which can't build
+   `osqp` and `qdldl`.
+
    **On Ubuntu**
 
    ```bash
@@ -97,7 +100,7 @@ This page goes through the Python environment setup steps in more detail and wit
        . "${PI_HOME}/etc/bash_profile"
    fi
 
-   module load git/2.27.0 git-lfs/2.11.
+   module load git/2.45.1 git-lfs/2.11.
    module load wcEcoli/python3
 
    export PYENV_ROOT="${PI_HOME}/pyenv"

docs/google-cloud.md

@@ -119,9 +119,10 @@ Also see [Borealis: How to run a workflow](https://github.com/CovertLab/borealis
    Cloud Firetasks that start running after you build an Image will "pull" it and use it,
    even Firetasks that were already queued to run. You can do this to update your code
    then use `lpad` commands to restart failed tasks and resume paused tasks.  
-   The full Docker Image path is `gcr.io/$PROJECT/$USER-wcm-code`, where
-   `$PROJECT` is the Google Cloud project name and `$USER` is your local username.
-   You can also "pull" this Image and run it locally.
+   The full Docker Image path is `${REGION}-docker.pkg.dev/${PROJECT}/wcm/${USER}-wcm-code`,
+   where `$REGION` is Artifact Registry region (we usually use `us-west1`),
+   `$PROJECT` is the Google Cloud project name, and `$USER` is your local username.
+   You can also `pull` this Image with Docker to run it locally.
 
 1. Open an ssh tunnel to the project's MongoDB "LaunchPad" server in Google Compute Engine: