Merge pull request #73 from nasaharvest/inference-tests

Enable Cloud APIs on deploy

README.md

@@ -25,26 +25,27 @@ Rapid map creation with machine learning and earth observation data.
 
 [cb]: https://colab.research.google.com/assets/colab-badge.svg
 
-**Examples:** [Cropland](https://github.com/nasaharvest/openmapflow/tree/main/crop-mask-example), [Buildings](https://github.com/nasaharvest/openmapflow/tree/main/buildings-example), [Maize](https://github.com/nasaharvest/openmapflow/tree/main/maize-example)
+**Example projects:** [Cropland](https://github.com/nasaharvest/openmapflow/tree/main/crop-mask-example), [Buildings](https://github.com/nasaharvest/openmapflow/tree/main/buildings-example), [Maize](https://github.com/nasaharvest/openmapflow/tree/main/maize-example)
 
 ![3maps-gif](assets/3maps.gif)
 
 * [Tutorial](#tutorial-)
-* [How it works](#how-it-works)
-* [Generating a project](#generating-a-project-)
-* [Adding data](#adding-data-)
-* [Training a model](#training-a-model-)
-* [Creating a map](#creating-a-map-)
-
-## Tutorial [![cb]](https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb)
+* [Creating a map from scratch](#creating-a-map-from-scratch)
+    * [Generating a project](#generating-a-project-)
+    * [Adding data](#adding-data-)
+    * [Training a model](#training-a-model-)
+    * [Creating a map](#creating-a-map-)
+* [Accessing existing datasets](#accessing-existing-datasets)
+
+# Tutorial [![cb]](https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/tutorial.ipynb)
 Colab notebook tutorial demonstrating data exploration, model training, and inference over small region. ([video](https://youtu.be/UHEUB4RSAi4))
 
 **Prerequisites:**
 - Github access token (obtained [here](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token))
 - [Forked OpenMapFlow repository](https://github.com/nasaharvest/openmapflow/fork)
 - Basic Python knowledge 
 
-## How it works 
+# Creating a map from scratch 
 
 To create your own maps with OpenMapFlow, you need to 
 1. [Generate your own OpenMapFlow project](#generating-a-project-), this will allow you to:
@@ -120,7 +121,7 @@ gsutil mb -l <YOUR_OPENMAPFLOW_YAML_GCLOUD_LOCATION> gs://<YOUR_OPENMAPFLOW_YAML
 
 ## Adding data
 
-#### Adding already existing data
+### Adding already existing data
 **Prerequisites:**
 - [ ] [Generated OpenMapFlow project](#generating-a-project-)
 
@@ -140,7 +141,7 @@ git commit -m'Created new dataset'
 git push
 ```
 
-#### Adding custom data [![cb]](https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb)
+### Adding custom data [![cb]](https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb)
 
 Data can be added by either following the below documentation OR running the above Colab notebook.
 
@@ -227,8 +228,10 @@ Now after merging the pull request, the model will be deployed to Google Cloud.
 
 Only available through above Colab notebook. Cloud Architecture must be deployed using the deploy.yaml Github Action.
 
-
-
-
-
-
+# Accessing existing datasets
+```python
+from openmapflow.datasets import togo_crop_2019
+df = togo_crop_2019.load_df()
+x = togo_crop_2019.iloc[0]["eo_data"]
+y = togo_crop_2019.iloc[0]["class_prob"]
+```

buildings-example/.gitignore

@@ -0,0 +1 @@
+__pycache__

buildings-example/README.md

@@ -0,0 +1,12 @@
+# OpenMapFlow Project
+
+[Detailed documentation](https://github.com/nasaharvest/openmapflow/blob/main/README.md)
+
+[cb]: https://colab.research.google.com/assets/colab-badge.svg
+[1]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb
+[2]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb
+[3]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb
+
+| Adding data  | Training a model | Creating a map |
+| ------------ | ---------------- | -------------- |
+| [![cb]][1]   | [![cb]][2]       | [![cb]][3]     |

buildings-example/openmapflow.yaml

@@ -1,4 +1,4 @@
-version: 0.1.1
+version: 0.1.2
 project: buildings-example
 description: OpenMapFlow buildings example
 gcloud:

crop-mask-example/.gitignore

@@ -0,0 +1 @@
+__pycache__

crop-mask-example/README.md

@@ -0,0 +1,12 @@
+# OpenMapFlow Project
+
+[Detailed documentation](https://github.com/nasaharvest/openmapflow/blob/main/README.md)
+
+[cb]: https://colab.research.google.com/assets/colab-badge.svg
+[1]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb
+[2]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb
+[3]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb
+
+| Adding data  | Training a model | Creating a map |
+| ------------ | ---------------- | -------------- |
+| [![cb]][1]   | [![cb]][2]       | [![cb]][3]     |

crop-mask-example/openmapflow.yaml

@@ -1,4 +1,4 @@
-version: 0.1.1
+version: 0.1.2
 project: crop-mask-example
 description: OpenMapFlow crop mask example
 gcloud:

maize-example/.gitignore

@@ -0,0 +1 @@
+__pycache__

maize-example/README.md

@@ -0,0 +1,12 @@
+# OpenMapFlow Project
+
+[Detailed documentation](https://github.com/nasaharvest/openmapflow/blob/main/README.md)
+
+[cb]: https://colab.research.google.com/assets/colab-badge.svg
+[1]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb
+[2]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb
+[3]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb
+
+| Adding data  | Training a model | Creating a map |
+| ------------ | ---------------- | -------------- |
+| [![cb]][1]   | [![cb]][2]       | [![cb]][3]     |

maize-example/openmapflow.yaml

@@ -1,4 +1,4 @@
-version: 0.1.1
+version: 0.1.2
 project: maize-example
 description: OpenMapFlow maize example
 gcloud:

openmapflow/constants.py

@@ -10,9 +10,11 @@
 TEMPLATE_TRAIN = TEMPLATES_DIR / "train.py"
 TEMPLATE_EVALUATE = TEMPLATES_DIR / "evaluate.py"
 TEMPLATE_REQUIREMENTS = TEMPLATES_DIR / "requirements.txt"
+TEMPLATE_GITIGNORE = TEMPLATES_DIR / ".gitignore"
+TEMPLATE_README = TEMPLATES_DIR / "README.md"
 TEMPLATE_DEPLOY_YML = TEMPLATES_DIR / "github-deploy.yaml"
 TEMPLATE_TEST_YML = TEMPLATES_DIR / "github-test.yaml"
-VERSION = "0.1.1"
+VERSION = "0.1.2"
 
 # -------------- Dataframe column names --------------------------------------
 SOURCE = "source"

openmapflow/datasets.py

@@ -25,3 +25,5 @@
         + "Knowledge Discovery Workshops, 2020."
     ),
 )
+
+datasets = [geowiki_landcover_2017, togo_crop_2019]

openmapflow/generate.py

@@ -9,6 +9,8 @@
     TEMPLATE_DATASETS,
     TEMPLATE_DEPLOY_YML,
     TEMPLATE_EVALUATE,
+    TEMPLATE_GITIGNORE,
+    TEMPLATE_README,
     TEMPLATE_REQUIREMENTS,
     TEMPLATE_TEST_YML,
     TEMPLATE_TRAIN,
@@ -70,6 +72,8 @@ def copy_template_files(PROJECT_ROOT: Path, overwrite: bool):
         TEMPLATE_TRAIN,
         TEMPLATE_EVALUATE,
         TEMPLATE_REQUIREMENTS,
+        TEMPLATE_GITIGNORE,
+        TEMPLATE_README,
     ]:
         if allow_write(PROJECT_ROOT / p.name, overwrite):
             shutil.copy(str(p), str(PROJECT_ROOT / p.name))

openmapflow/inference_utils.py

@@ -75,7 +75,7 @@ def get_ee_task_amount(prefix: Optional[str] = None):
     """
     ee_prefix = None
     if prefix is not None:
-        ee_prefix = prefix.replace("/", "-").replace("=", "-")
+        ee_prefix = prefix.replace("/", "-").replace("=", "-").replace(".", "-")
     amount = 0
     task_list = ee.data.getTaskList()
     for t in tqdm(task_list):

openmapflow/labeled_dataset.py

@@ -303,12 +303,10 @@ def summary(self, df: pd.DataFrame) -> str:
             + "\n"
         )
 
-    def load_df(self, skip_to_np=False) -> pd.DataFrame:
+    def load_df(self, skip_to_np: bool = False) -> pd.DataFrame:
         """Load dataset (labels + earth observation data) as a DataFrame"""
         if not self.df_path.exists():
-            raise FileNotFoundError(
-                f"{self.df_path} does not exist, run openmapflow create-datasets"
-            )
+            print(self.create_dataset())
         df = pd.read_csv(self.df_path)
         df = df[_clean_df_condition(df)].copy()
         if df[EO_DATA].isnull().any():

openmapflow/notebooks/train.ipynb

@@ -55,11 +55,7 @@
         "\n",
         "!git config --global user.email $username\n",
         "!git config --global user.name $email\n",
-        "!git clone {github_url.replace(\"https://\", f\"https://{username}:{token}@\")}\n",
-        "\n",
-        "# Temporarily install from Github\n",
-        "!pip install openmapflow\n",
-        "!pip install pyyaml==5.4.1 -q"
+        "!git clone {github_url.replace(\"https://\", f\"https://{username}:{token}@\")}"
       ]
     },
     {
@@ -73,7 +69,10 @@
         "from pathlib import Path\n",
         "import os\n",
         "openmapflow_yaml_path = input(\"Path to openmapflow.yaml: \")\n",
-        "%cd {Path(openmapflow_yaml_path).parent}"
+        "%cd {Path(openmapflow_yaml_path).parent}\n",
+        "\n",
+        "!pip install -r requirements.txt -q\n",
+        "!pip install pyyaml==5.4.1 -q"
       ]
     },
     {
@@ -89,22 +88,22 @@
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "bjBkmh1Xia4N"
+        "id": "xwb2HeCPRH8K"
       },
       "outputs": [],
       "source": [
-        "!pip install tsai -q"
+        "!dvc pull -q"
       ]
     },
     {
       "cell_type": "code",
       "execution_count": null,
       "metadata": {
-        "id": "oWoGz94avN0w"
+        "id": "Ex8RkDRiQaxo"
       },
       "outputs": [],
       "source": [
-        "!dvc pull -q"
+        "!openmapflow create-datasets"
       ]
     },
     {
@@ -116,17 +115,6 @@
         "# 3. Train model"
       ]
     },
-    {
-      "cell_type": "code",
-      "execution_count": null,
-      "metadata": {
-        "id": "Ex8RkDRiQaxo"
-      },
-      "outputs": [],
-      "source": [
-        "!openmapflow create-datasets"
-      ]
-    },
     {
       "cell_type": "code",
       "execution_count": null,
@@ -178,7 +166,7 @@
       },
       "outputs": [],
       "source": [
-        "!dvc commit $(openmapflow datapath MODELS) -q \n",
+        "!dvc commit -q\n",
         "!dvc push -q"
       ]
     },

openmapflow/scripts/deploy.sh

@@ -4,7 +4,7 @@
 
 set -e # Exit when any command fails
 
-echo "1/7 Setting OpenMapFlow environment variables"
+echo "1/8 Setting OpenMapFlow environment variables"
 export $(
         python -c \
         "from openmapflow.config import deploy_env_variables; \
@@ -14,7 +14,7 @@ export $(
 env | grep OPENMAPFLOW
 
 
-echo "2/7 Ensuring latest models are available for deployment"
+echo "2/8 Ensuring latest models are available for deployment"
 dvc pull "$OPENMAPFLOW_MODELS_DIR".dvc -f
 
 export OPENMAPFLOW_MODELS=$(
@@ -24,8 +24,15 @@ export OPENMAPFLOW_MODELS=$(
 )
 echo "MODELS: $OPENMAPFLOW_MODELS"
 
+echo "3/8 Enable Google Cloud APIs: Artifact Registry, Cloud Run, Cloud Functions"
+gcloud services enable artifactregistry.googleapis.com
+gcloud services enable cloudbuild.googleapis.com
+gcloud services enable cloudfunctions.googleapis.com
+gcloud services enable earthengine.googleapis.com
+gcloud services enable run.googleapis.com
+
 
-echo "3/7 Create Google Cloud Buckets if they don't exist"
+echo "4/8 Create Google Cloud Buckets if they don't exist"
 for BUCKET in $OPENMAPFLOW_GCLOUD_BUCKET_LABELED_EO \
         $OPENMAPFLOW_GCLOUD_BUCKET_INFERENCE_EO \
         $OPENMAPFLOW_GCLOUD_BUCKET_PREDS \
@@ -39,15 +46,14 @@ do
 done
 
 
-echo "4/7 Checking if Artifact Registry needs to be created for storing OpenMapFlow docker images"
-gcloud services enable artifactregistry.googleapis.com
+echo "5/8 Checking if Artifact Registry needs to be created for storing OpenMapFlow docker images"
 if [ -z "$(gcloud artifacts repositories list --format='get(name)' --filter "$OPENMAPFLOW_PROJECT")" ]; then
         gcloud artifacts repositories create "$OPENMAPFLOW_PROJECT" \
         --location "$OPENMAPFLOW_GCLOUD_LOCATION" \
         --repository-format docker
 fi
 
-echo "5/7 Build and push inference docker image to Google Cloud artifact registry"
+echo "6/8 Build and push inference docker image to Google Cloud artifact registry"
 gcloud auth configure-docker "${OPENMAPFLOW_GCLOUD_LOCATION}"-docker.pkg.dev
 docker build . \
         -f "$OPENMAPFLOW_LIBRARY_DIR"/Dockerfile \
@@ -59,7 +65,7 @@ docker build . \
 docker push "$OPENMAPFLOW_DOCKER_TAG"
 
 
-echo "6/7 Deploy inference docker image to Google Cloud Run"
+echo "7/8 Deploy inference docker image to Google Cloud Run"
 echo "Deploying prediction server on port 8080"
 gcloud run deploy "$OPENMAPFLOW_PROJECT" --image "$OPENMAPFLOW_DOCKER_TAG":latest \
         --cpu=4 \
@@ -81,7 +87,7 @@ gcloud run deploy "$OPENMAPFLOW_PROJECT"-management-api --image "$OPENMAPFLOW_DO
 
 
 
-echo "7. Deploy inference trigger as a Google Cloud Function"
+echo "8/8 Deploy inference trigger as a Google Cloud Function"
 export OPENMAPFLOW_URL=$(gcloud run services list --platform managed --filter $OPENMAPFLOW_PROJECT --limit 1 --format='get(URL)')
 
 gcloud functions deploy trigger-"$OPENMAPFLOW_PROJECT" \
@@ -90,5 +96,4 @@ gcloud functions deploy trigger-"$OPENMAPFLOW_PROJECT" \
     --allow-unauthenticated \
     --runtime=python39 \
     --entry-point=trigger \
-    --set-env-vars MODELS="$OPENMAPFLOW_MODELS",INFERENCE_HOST="$OPENMAPFLOW_URL" \
-    --timeout=300s
+    --set-env-vars MODELS="$OPENMAPFLOW_MODELS",INFERENCE_HOST="$OPENMAPFLOW_URL" 

openmapflow/templates/.gitignore

@@ -0,0 +1 @@
+__pycache__

openmapflow/templates/README.md

@@ -0,0 +1,12 @@
+# OpenMapFlow Project
+
+[Detailed documentation](https://github.com/nasaharvest/openmapflow/blob/main/README.md)
+
+[cb]: https://colab.research.google.com/assets/colab-badge.svg
+[1]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/new_data.ipynb
+[2]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/train.ipynb
+[3]: https://colab.research.google.com/github/nasaharvest/openmapflow/blob/main/openmapflow/notebooks/create_map.ipynb
+
+| Adding data  | Training a model | Creating a map |
+| ------------ | ---------------- | -------------- |
+| [![cb]][1]   | [![cb]][2]       | [![cb]][3]     |

openmapflow/templates/integration_test_project.py

@@ -30,7 +30,7 @@ def test_config(self):
             has_issues = True
             print(
                 f"\u2716 openmapflow.yaml version: {CONFIG_YML['version']} "
-                + "does not match package version: {VERSION}"
+                + f"does not match package version: {VERSION}"
             )
 
         if not path_exists(Path(dp.RAW_LABELS + ".dvc")):