Add Airflow demo script for one click deployment (airbytehq#4451)

* Add Airflow Demo Example

* Finalize script instructions

* Add down.sh and run gradle format

* Note about down.sh

* Address review comments

Co-authored-by: Abhi Vaidyanatha <[email protected]>

.gitignore

@@ -32,3 +32,7 @@ crash.log
 *.tfstate
 *.tfstate.backup
 *.lock.hcl
+
+# Airflow Demo
+resources/examples/airflow/logs/*
+!resources/examples/airflow/logs/.gitkeep

resources/examples/airflow/.env

@@ -0,0 +1,2 @@
+AIRFLOW_UID=501
+AIRFLOW_GID=0

resources/examples/airflow/README.md

@@ -0,0 +1,17 @@
+# Airflow Airbyte Plugin Demo Example
+
+## Running the Script
+Simply run `up.sh` to bring up the Airbyte and Airflow containers. You don't need to worry about cleaning up the containers, as we run a `docker-compose down -v` before attempting to recreate them.
+
+After Airbyte starts running, head over to http://localhost:8000 and set up a connection. The script will then prompt you for an Airbyte connection id. You can find this in the URL after clicking on your connection.
+
+![](./assets/airbyte_connection_id.png)
+
+Simple enter that ID into your terminal and a connection will be set up in Airflow for you. Head over to http://localhost:8085 to access the Airflow UI. Enter `airflow` as your username and `airflow` as your password. You should see your DAG show up in the UI with no errors if your connection was configured correctly.
+
+![](./assets/airflow_start_dag.png)
+
+Trigger the DAG with the switch in the top right and you should be in business! If it doesn't automatically run, just hit the play button in the top right to kick off the sync.
+
+## Cleaning Up
+Run `down.sh` to clean up the containers. Or run `docker-compose down -v` here and in the root directory, your call.

resources/examples/airflow/airflow.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+#
+# Run airflow command in container
+#
+
+PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+set -euo pipefail
+
+export COMPOSE_FILE="${PROJECT_DIR}/docker-compose.yaml"
+exec docker-compose run --rm -e CONNECTION_CHECK_MAX_COUNT=0 airflow-worker "${@}"

resources/examples/airflow/dags/dag_airbyte_example.py

@@ -0,0 +1,21 @@
+from airflow import DAG
+from airflow.utils.dates import days_ago
+from airflow.providers.airbyte.operators.airbyte import AirbyteTriggerSyncOperator
+from airflow.models import Variable
+
+airbyte_connection_id = Variable.get("AIRBYTE_CONNECTION_ID")
+
+with DAG(dag_id='trigger_airbyte_job_example',
+         default_args={'owner': 'airflow'},
+         schedule_interval='@daily',
+         start_date=days_ago(1)
+         ) as dag:
+
+    example_sync = AirbyteTriggerSyncOperator(
+        task_id='airbyte_example',
+        airbyte_conn_id='airbyte_example',
+        connection_id=airbyte_connection_id,
+        asynchronous=False,
+        timeout=3600,
+        wait_seconds=3
+    )

resources/examples/airflow/docker-compose.yaml

@@ -0,0 +1,151 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Basic Airflow cluster configuration for CeleryExecutor with Redis and PostgreSQL.
+#
+# WARNING: This configuration is for local development. Do not use it in a production deployment.
+#
+# This configuration supports basic configuration using environment variables or an .env file
+# The following variables are supported:
+#
+# AIRFLOW_IMAGE_NAME         - Docker image name used to run Airflow.
+#                              Default: apache/airflow:master-python3.8
+# AIRFLOW_UID                - User ID in Airflow containers
+#                              Default: 50000
+# AIRFLOW_GID                - Group ID in Airflow containers
+#                              Default: 50000
+# _AIRFLOW_WWW_USER_USERNAME - Username for the administrator account.
+#                              Default: airflow
+# _AIRFLOW_WWW_USER_PASSWORD - Password for the administrator account.
+#                              Default: airflow
+#
+# Feel free to modify this file to suit your needs.
+---
+version: "3"
+x-airflow-common: &airflow-common
+  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.0}
+  environment: &airflow-common-env
+    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
+    AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
+    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
+    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
+    AIRFLOW__CORE__FERNET_KEY: ""
+    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
+    AIRFLOW__CORE__LOAD_EXAMPLES: "false"
+    AIRFLOW__API__AUTH_BACKEND: "airflow.api.auth.backend.basic_auth"
+  volumes:
+    - ./dags:/opt/airflow/dags
+    - ./logs:/opt/airflow/logs
+    - ./plugins:/opt/airflow/plugins
+    - ./requirements.txt:/opt/airflow/requirements.txt
+  user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
+  depends_on:
+    redis:
+      condition: service_healthy
+    postgres:
+      condition: service_healthy
+
+services:
+  postgres:
+    image: postgres:13
+    environment:
+      POSTGRES_USER: airflow
+      POSTGRES_PASSWORD: airflow
+      POSTGRES_DB: airflow
+    volumes:
+      - postgres-db-volume:/var/lib/postgresql/data
+    healthcheck:
+      test: ["CMD", "pg_isready", "-U", "airflow"]
+      interval: 5s
+      retries: 5
+    restart: always
+
+  redis:
+    image: redis:latest
+    ports:
+      - 6379:6379
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+      interval: 5s
+      timeout: 30s
+      retries: 50
+    restart: always
+
+  airflow-webserver:
+    <<: *airflow-common
+    container_name: "airflow_webserver"
+    command: 'bash -c "pip3 install -r requirements.txt && airflow webserver"'
+    ports:
+      - 8085:8080
+    healthcheck:
+      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
+      interval: 10s
+      timeout: 10s
+      retries: 5
+    restart: always
+
+  airflow-scheduler:
+    <<: *airflow-common
+    command: 'bash -c "pip3 install -r requirements.txt && airflow scheduler"'
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"',
+        ]
+      interval: 10s
+      timeout: 10s
+      retries: 5
+    restart: always
+
+  airflow-worker:
+    <<: *airflow-common
+    command: 'bash -c "pip3 install -r requirements.txt && airflow celery worker"'
+    healthcheck:
+      test:
+        - "CMD-SHELL"
+        - 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
+      interval: 10s
+      timeout: 10s
+      retries: 5
+    restart: always
+
+  airflow-init:
+    <<: *airflow-common
+    command: version
+    environment:
+      <<: *airflow-common-env
+      _AIRFLOW_DB_UPGRADE: "true"
+      _AIRFLOW_WWW_USER_CREATE: "true"
+      _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
+      _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
+
+  flower:
+    <<: *airflow-common
+    command: 'bash -c "pip3 install -r requirements.txt && airflow celery flower"'
+    ports:
+      - 5555:5555
+    healthcheck:
+      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
+      interval: 10s
+      timeout: 10s
+      retries: 5
+    restart: always
+
+volumes:
+  postgres-db-volume:

resources/examples/airflow/down.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+cd ../../..
+docker-compose down -v
+cd resources/examples/airflow || exit
+docker-compose down -v

resources/examples/airflow/requirements.txt

@@ -0,0 +1,2 @@
+apache-airflow-providers-http
+apache-airflow-providers-airbyte

resources/examples/airflow/up.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+cd ../../..
+docker-compose down -v
+docker-compose up -d
+cd resources/examples/airflow || exit
+docker-compose down -v
+docker-compose up -d
+echo "Access Airbyte at http://localhost:8000 and set up a connection."
+echo "Enter your Airbyte connection ID: "
+read connection_id
+docker exec -ti airflow_webserver airflow variables set 'AIRBYTE_CONNECTION_ID' "$connection_id"
+docker exec -ti airflow_webserver airflow connections add 'airbyte_example' --conn-uri 'airbyte://host.docker.internal:8000'
+echo "Access Airflow at http://localhost:8085 to kick off your Airbyte sync DAG."