Release version 1

.editorconfig

@@ -0,0 +1,12 @@
+root = true
+
+[*]
+indent_style = space
+indent_size = 4
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+
+[*.yml]
+indent_size = 2

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 Brooklyn Data Company LLC
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,66 @@
+# Meltano on Github Actions
+
+This template uses [cookiecutter](https://github.com/cookiecutter/cookiecutter) to generate a [GitHub Actions](https://github.com/features/actions) orchestrated [Meltano](https://meltano.com/) project.
+
+For a list of pre-made Singer taps and targets, see the [Meltano Hub](https://hub.meltano.com/singer/taps/).
+
+## Benefits to using GitHub Actions
+
+- Requires only a GitHub account to deploy a fully automated Meltano project with logging, alerting, and incremental state
+- Configuration is version controlled and maintained as code
+- Low barrier to entry, a working project can be deployed on a schedule in a couple of hours
+- Easy management of secrets
+- Low cost. See the free minutes for your plan [here](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes), and the incremental cost outside of your free minutes [here](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates).
+
+## Limitations of using GitHub Actions
+
+- There is a [6 hour maximum run time](https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits) for any isolated individual job in a workflow. This often means that backfills need to be performed outside of GitHub actions.
+- No way of exposing the Meltano UI
+
+## Usage prerequisites:
+- Python >= 3.7
+- pipx
+- meltano
+- cookiecutter
+
+Install pipx with:
+```bash
+pip install pipx
+pipx ensurepath
+```
+
+Install Meltano with:
+```bash
+pipx install meltano
+```
+
+Install Cookiecutter with:
+```bash
+pipx install cookiecutter
+```
+
+## Instructions
+
+1. In your terminal, navigate to the parent folder in which you'd like the project to be created.
+2. Run `cookiecutter https://github.com/brooklyn-data/meltano-on-github-actions` and follow the prompts.
+3. From inside the newly generated project, search for all 'TODO' strings, and complete any actions required.
+4. Once ready to publish, initialize Git with `git init`.
+5. Create an empty repository in GitHub.
+6. Take the `.git` URL of the newly created remote repository, and run `git remote add origin <.git url>`.
+7. Stage and commit the generated project files with `git add -A` and `git commit -m 'Initial commit'`.
+8. Make sure the branch is named `main` by running `git branch -M main`.
+9. Finally, push the created project to the remote repository with `git push -u origin main`.
+10. Configure any required secrets in the GitHub repo.
+
+## Slack alerts
+Slack alerts on failure are enabled [using the official Slack GitHub action](https://github.com/slackapi/slack-github-action), using '[Technique 2: Slack App](https://github.com/slackapi/slack-github-action#technique-2-slack-app)'. To configure:
+
+1. [Create a Slack App](https://api.slack.com/apps) for the workspace, with a suitable name (e.g. Meltano).
+2. Add the [chat.write](https://api.slack.com/scopes/chat:write) bot scope under OAuth & Permissions.
+3. Install the app to the workspace.
+4. Copy the app's Bot Token from the OAuth & Permissions page and add it as a secret in the repo settings named `SLACK_BOT_TOKEN`.
+5. Invite the bot user into the channel you wish to post messages to (/invite @bot_user_name).
+6. Copy the Slack channel's Channel ID (from the channel's About section, accessed by clicking the drop down arrow next to the channel's name) into another repository secret named `SLACK_CHANNEL_ID`.
+
+# About Brooklyn Data Co.
+We are a full-stack data and analytics team, focused on leadership, process improvement, implementation, and advanced analytics. [Read more about what we do](https://brooklyndata.co) and [check out our open roles!](https://brooklyndata.co/careers)

cookiecutter.json

@@ -0,0 +1,9 @@
+{
+    "org_name": "Correctly capitalized organization name",
+    "project_folder_name": "meltano-on-github-actions",
+    "add_target": [
+        "No",
+        "Snowflake",
+        "BigQuery"
+    ]
+}

{{ cookiecutter.project_folder_name }}/.editorconfig

@@ -0,0 +1,12 @@
+root = true
+
+[*]
+indent_style = space
+indent_size = 4
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true
+
+[*.yml]
+indent_size = 2

{{ cookiecutter.project_folder_name }}/.github/workflows/README.md

@@ -0,0 +1 @@
+Create new tap-target workflow files using the existing as a guide.

{{ cookiecutter.project_folder_name }}/.github/workflows/example-workflow.yml

@@ -0,0 +1,70 @@
+{%- raw %}
+on:
+  schedule:
+    - cron: '0 8 * * *' # Every day at 8AM UTC - TODO update as needed
+  workflow_dispatch: # workflow_dispatch enables the workflow to be triggered manually
+
+jobs:
+  meltano:
+    runs-on: ubuntu-latest
+    name: Meltano
+    steps:
+      # Checkout the repository so that a path to the action.yml can be used
+      - uses: actions/checkout@v2
+{% endraw %}{%- if cookiecutter.add_target == "BigQuery" %}{% raw %}
+      # Authenticate to Google SDK for target-bigquery
+      - name: Set up Cloud SDK
+        uses: google-github-actions/[email protected]
+        with:
+          # TODO add GCP_PROJECT_ID and GCP_SA_KEY as GitHub Secrets
+          project_id: ${{ secrets.GCP_PROJECT_ID }}
+          service_account_key: ${{ secrets.GCP_SA_KEY }}
+          export_default_credentials: true
+{% endraw %}{%- endif %}{% raw %}
+      # Get the database files from the previous run, if exists
+      - name: Download latest SQLite database artifact
+        uses: dawidd6/action-download-artifact@v2
+        continue-on-error: true
+        with:
+          workflow: ${{ github.workflow }}
+          branch: main
+          name: meltano.db
+          path: ${{ github.workspace }}
+          workflow_conclusion: completed # Download artifacts from the latest completed run, even if it failed. This enables Meltano to start off from where a tap failed.
+      - name: Meltano
+        if: always() # Always run this step even if the prior failed due to no artifact found (this should only take effect if first time running)
+        uses: ./
+        env:
+          # TODO add and modify as needed and add to the repo's secrets
+{% endraw %}{%- if cookiecutter.add_target == "Snowflake" %}{% raw %}
+          TARGET_SNOWFLAKE_ACCOUNT: ${{ secrets.TARGET_SNOWFLAKE_ACCOUNT }}
+          TARGET_SNOWFLAKE_USERNAME: MELTANO
+          TARGET_SNOWFLAKE_PASSWORD: ${{ secrets.TARGET_SNOWFLAKE_PASSWORD }}
+          TARGET_SNOWFLAKE_DBNAME: MELTANO
+          TARGET_SNOWFLAKE_WAREHOUSE: MELTANO
+          TARGET_SNOWFLAKE_FILE_FORMAT: meltano.public.csv
+{% endraw %}{%- elif cookiecutter.add_target == "BigQuery" %}{% raw %}
+          TARGET_BIGQUERY_PROJECT_ID: # TODO add the production BigQuery project ID here
+{% endraw %}{%- endif %}{% raw %}
+          MELTANO_DATABASE_URI: sqlite:////github/workspace/meltano.db
+        with:
+          # TODO set correct tap and target names
+          args: meltano elt tap-example {% endraw %}{% if cookiecutter.add_target == "Snowflake" %}target-snowflake{% elif cookiecutter.add_target == "BigQuery" %}target-bigquery{% else %}target-example{% endif %}{% raw %} --job_id=meltano # job_id required to use incremental state, can be anything
+      - name: Meltano Slack alert
+        # TODO configure SLACK_CHANNEL_ID and SLACK_BOT_TOKEN secrets in the repo
+        if: failure() # Only run if one of the previous steps fails
+        id: slack
+        uses: slackapi/[email protected]
+        with:
+          channel-id: ${{ secrets.SLACK_CHANNEL_ID }}
+          slack-message: "<https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|:warning: Workflow ${{ github.workflow }} has failed.>"
+        env:
+          SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
+      # Upload the sqlite database containing incremental, bookmark state
+      - name: Upload SQLite database artifact
+        if: always() # Always run this step even if the prior failed
+        uses: actions/upload-artifact@v2
+        with:
+          name: meltano.db
+          path: ${{ github.workspace }}/meltano.db*
+{%- endraw %}

{{ cookiecutter.project_folder_name }}/.gitignore

@@ -0,0 +1,137 @@
+# Secrets and internal config files
+**/.secrets/*
+gcp.json
+
+# Ignore meltano internal cache and sqlite systemdb
+
+.meltano/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

{{ cookiecutter.project_folder_name }}/Dockerfile

@@ -0,0 +1,10 @@
+FROM meltano/meltano:v1.96.0-python3.8
+
+# Meltano project and install
+COPY . /project/
+RUN meltano install
+
+# GitHub actions sets the working directory to the repo root when running.
+# The entrypoint script sets the current directory to the meltano project and
+# runs the container arguments as a command.
+ENTRYPOINT ["./entrypoint.sh"]