diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e454995..6d6f3ad 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,7 +4,7 @@ ### Personal access token (PAT) -Follow [these](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) instructions to configure a PAT. +Follow [these](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) instructions to configure a PAT in GitHub. Scopes: - repo (and all sub-items) @@ -17,7 +17,7 @@ Save the token to a secure location. Click the "Configure SSO" button and "Autho ```shell cp config.example.json config.json -# Add your GitHub username and token +# Add the relevant GitHub username, email address and token $EDITOR config.json # Export the JSON credentials into an environment variable @@ -44,29 +44,27 @@ PYTHONPATH=hubcap python -m pytest ## Run in test mode -```shell -export ENV=test -./cron.sh -``` - -### Optional configuration environment variables -```shell -# Default value is the `git-tmp` directory within the current working directory -# This directory will be deleted by default at the end of the run -export GIT_TMP=git-tmp -``` +- Set `"repo": "hub.getdbt.com-test"` within `config.json` (or specify some other non-production repository). +- Optional: set `"push_branches": false` within `config.json`. -### Optional parameters -Preserve commits/build artifacts within the `$GIT_TMP` directory +Run: ```shell -export ENV=test -./cron.sh --no-cleanup +python3 hubcap/hubcap.py ``` ## Run in production mode **WARNING:** Use with caution -- _will_ modify state. +- Set `"repo": "hub.getdbt.com"` within `config.json` (since [hub.getdbt.com](https://github.com/dbt-labs/hub.getdbt.com) is the production repository). +- Set `"push_branches": true` within `config.json`. + +Run: ```shell -export ENV=prod -./cron.sh +python3 hubcap/hubcap.py +``` + +### Optional configuration environment variables +```shell +# Default value is the `git-tmp` directory within the current working directory +export GIT_TMP=git-tmp ``` diff --git a/RELEASE.md b/RELEASE.md index 2106966..d8a2724 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -4,9 +4,8 @@ This application is hosted on [Heroku](https://www.heroku.com), but it can also ## Design overview It is designed to do the following: -1. Use a cron schedule to execute the `cron.sh` script at the beginning of every hour - 1. The `cron.sh` script creates a `git-tmp` directory that is destroyed upon completion -2. The `cron.sh` script defers most of the processing to `hubcap.py` +1. Use a cron schedule to execute the main `hubcap.py` script at the beginning of every hour +2. The `hubcap.py` script creates a `git-tmp` directory to hold cloned git repositories 3. `hubcap.py` creates a JSON spec file for each package+version combo within the `git-tmp/hub/data/packages/` directory 4. It opens pull requests against [dbt-labs/hub.getdbt.com](https://github.com/dbt-labs/hub.getdbt.com) for any new versions of dbt packages - [Example PR for first-time package](https://github.com/dbt-labs/hub.getdbt.com/pull/1681/files) @@ -17,12 +16,11 @@ It is designed to do the following: The commands below assume a production application named `dbt-hubcap`. Replace with `dbt-hubcap-staging` for the staging version of the application. 1. Use the [Heroku Scheduler](https://dashboard.heroku.com/apps/dbt-hubcap/scheduler) to set the following cron schedule: - - Job: `./cron.sh` + - Job: `python3 hubcap/hubcap.py` - Schedule: Every hour at :00 - Dyno size: Hobby / Standard-1X -1. Configure the `CONFIG` and `ENV` environment variables: [Settings > Config Vars > Reveal Config Vars](https://dashboard.heroku.com/apps/dbt-hubcap/settings) +1. Configure the `CONFIG` environment variable: [Settings > Config Vars > Reveal Config Vars](https://dashboard.heroku.com/apps/dbt-hubcap/settings) - `CONFIG`: copy format from `config.example.json` and adjust values as needed - - `ENV`: `prod` 1. (Re-)deploy the application using the instructions below. See [these](https://dashboard.heroku.com/apps/dbt-hubcap/deploy/heroku-git) instructions for context. @@ -55,6 +53,14 @@ heroku git:remote -a dbt-hubcap git push heroku main:main ``` +### Ad hoc executions of the script + +For off-schedule ad hoc executions, run the following from the deploy directory above: + +```shell +heroku run python3 hubcap/hubcap.py +``` + #### Explanation `heroku` is the remote that Heroku will use for deploys. `origin` is the source code hosted on GitHub. diff --git a/config.example.json b/config.example.json index 82abe48..e4501b9 100644 --- a/config.example.json +++ b/config.example.json @@ -5,7 +5,7 @@ "token": "pe4s0n@l-@cce$$-t0k3n" }, "org": "dbt-labs", - "repo": "hub.getdbt.com", + "repo": "hub.getdbt.com-test", "push_branches": true, "one_branch_per_repo": true } diff --git a/cron.sh b/cron.sh deleted file mode 100755 index 2830f2c..0000000 --- a/cron.sh +++ /dev/null @@ -1,91 +0,0 @@ -#!/bin/bash - -# == -# == Parse args -# == - -POSITIONAL=() -while [ "$#" != '0' ]; do - case "$1" in - # useful for wanting to save git-tmp and observe commits/build artifacts - -n|--no-cleanup) - CLEANUP='no' - shift - ;; - *) - POSITIONAL+=("$1") - shift - ;; - esac -done -set -- "${POSITIONAL[@]}" - -# == -# == Setup -# == - -# export to expose to python script later on -export RUN_DIR="$(pwd)" -export GIT_TMP="$RUN_DIR/git-tmp" - -exit_routine() { - local -r exit_status="${1}" - - # Remove temp files to avoid future conflicts - if [ "${CLEANUP:-yes}" = 'yes' ]; then - if [ -d "${GIT_TMP}" ]; then - rm -rf "${GIT_TMP}" - fi - fi - - # Reset local email and name git settings to wrote-author commits - if [ "$ENV" = 'prod' ] || [ "$ENV" = 'test' ]; then - function reset_git_params() { - # reset machine's git config state to pre-script invocation - if [ -z "${2}" ] && [ -z "$(git config --global user."${1}")" ]; then - git config --global --unset user."${1}" - else - git config --global user."${1}" "${2}" - fi - } - reset_git_params 'email' "$PRIOR_GIT_EMAIL" - reset_git_params 'name' "$PRIOR_GIT_NAME" - fi - - exit "${exit_status}" -} - -trap 'exit_routine $?' EXIT - -set -o errexit -set -o errtrace -set -o nounset -set -o pipefail - -# == -# == Main -# == - -# specify ENV in Heroku's config variables or override on CLI -export ENV="${ENV:-development}" - -if [ "$ENV" = 'prod' ] || [ "$ENV" = 'test' ]; then - # User's prior state saved to avoid corrupting local git config params - PRIOR_GIT_EMAIL="$(git config --global user.email || : )" - PRIOR_GIT_NAME="$(git config --global user.name || : )" - - # Setup git repo for automated commits during execution - git config --global user.email 'drew@fishtownanalytics.com' # TODO: make this a dedicated CI user - git config --global user.name 'dbt-hubcap' -fi - -# hubcap expects a fresh git-tmp, so deletion of git-tmp is forced before script is allowed to run in full -if ! [ -d "${GIT_TMP}" ]; then - mkdir "${GIT_TMP}" -else - >&2 printf "Error: ${GIT_TMP} already exists. Deleting and exiting.\n" - unset CLEANUP - exit 1 -fi - -python3 hubcap/hubcap.py