Added documentations for how to run DAGs in development alongside how…

… to add new documentations. (#4388) * added documentations for how to run dags in development, and also how to add new documentation * Use MyST note callout * Clarify language around API keys --------- Co-authored-by: Madison Swain-Bowden <[email protected]>
WordPress · May 28, 2024 · a9b02c7 · a9b02c7
1 parent a504365
commit a9b02c7
Show file tree

Hide file tree

Showing 8 changed files with 93 additions and 0 deletions.
diff --git a/documentation/_static/airflow_home.png b/documentation/_static/airflow_home.png
diff --git a/documentation/_static/airflow_login.png b/documentation/_static/airflow_login.png
diff --git a/documentation/_static/dag_index.png b/documentation/_static/dag_index.png
diff --git a/documentation/_static/dag_unpaused.png b/documentation/_static/dag_unpaused.png
diff --git a/documentation/catalog/guides/dag_testing.md b/documentation/catalog/guides/dag_testing.md
@@ -0,0 +1,54 @@
+# Running DAGs locally
+
+[Apache Airflow](https://airflow.apache.org/) is used to manage our workflows
+(DAGs) in the catalog. For more information see
+[our quickstart guide](/catalog/guides/quickstart.md#api-data). This document
+describes how to run or test out these DAGs locally during development.
+
+Additionally, it is worth noting that not all DAGs can be run locally in
+development right away as some of them require API keys from the provider.
+However, some other DAGs like the ones for SMK or Finnish Museums can be run
+locally without any additional keys. In order to run some DAGs locally, you
+might need to get the API keys from the provider and add them to the
+`catalog/.env` file.
+
+## Getting started
+
+1. Refer to and follow the instructions at [Quickstart](./quickstart.md) to
+   setup and make sure the catalog service is up and running. If you have
+   successfully completed the general setup then this can be started by running
+   `just catalog/up`.
+
+2. Navigate to http://localhost:9090
+
+![Airflow Login Page](/_static/airflow_login.png)
+
+3. You should be met with an authentication page, use `airflow` as both the
+   username and password to log in.
+
+![Airflow Home Page](/_static/airflow_home.png)
+
+4. Search for or scroll down to any DAG of choice that does not require an API
+   key and click on it. We are using the `finnish_museums_workflow` for this
+   example.
+
+![Airflow DAG Index](/_static/dag_index.png)
+
+5. Click the toggle button labelled "DAG" at the top left, an alert box pops up
+   for you to confirm the action, click "OK" to continue. A run will get kicked
+   off.
+
+   DAGs which are run on a schedule (like this one) will usually kick off
+   immediately, though the page may need to be refreshed in order to view the
+   run. If you wish to kick off a new DAG run, you should click on the "Trigger
+   DAG" button represented by a "play" icon at the top right of the page.
+
+![Airflow DAG Unpaused](/_static/dag_unpaused.png)
+
+6. To get a summary of the DAG, click on the "DAG Docs" accordion and you should
+   see an overview of the DAG displayed.
+
+```{note}
+For more info about how Airflow works in general, check out their
+[documentation on the UI](https://airflow.apache.org/docs/apache-airflow/stable/ui.html).
+```
diff --git a/documentation/catalog/guides/index.md b/documentation/catalog/guides/index.md
@@ -7,4 +7,5 @@ quickstart
 deploy
 adding_a_new_provider
 deployment
+dag_testing
 ```
diff --git a/documentation/meta/documentation/adding_docs.md b/documentation/meta/documentation/adding_docs.md
@@ -0,0 +1,37 @@
+# Adding a new documentation page
+
+This is a quick guide for adding a new documentation page and running the
+documentation locally.
+
+## Prerequisites
+
+Refer to the [general setup guide](/general/general_setup.md) for setting up the
+prerequisites. Refer to the 'Docs' column in the
+[requirement matrix](/general/general_setup.md#requirement-matrix) to know what
+you need to run this.
+
+## Starting up
+
+The following demonstrates the process of adding a new documentation page.
+
+1. Create a markdown file at a place within the `documentation/` hierarchy that
+   makes the most sense for your case. Usually the parent folder for the new
+   file would already contain an `index.md` file, make sure that your new file
+   is located next to it. If a new directory is required, you will also need to
+   create an `index.md` file for it with a table of contents for the folder. See
+   other `index.md` files within the `documentation/` directory for examples.
+
+   For instance, adding a new documentation file `dag_testing.md` for describing
+   how to test/run the catalog DAGs would likely have you adding the file in
+   `documentation/catalog/guides/`.
+
+2. Make an entry with the name of the new document (excluding the `.md` file
+   extension) in the closest `index.md`. For our `dag_testing.md` instance, this
+   would mean adding `dag_testing` to `documentation/catalog/guides/index.md`.
+
+3. Write the document using [MyST flavored Markdown](https://mystmd.org/guide).
+
+4. Run `just lint` to properly format the document.
+
+5. Run `just documentation/live` locally (following the quick start guide) and
+   navigate to the new page to make sure it looks okay!
diff --git a/documentation/meta/documentation/index.md b/documentation/meta/documentation/index.md
@@ -5,6 +5,7 @@
 
 quickstart
 guidelines
+adding_docs
 extensions
 review
 ```