Skip to content

Commit

Permalink
Remove cron.sh script (#171)
Browse files Browse the repository at this point in the history
* Use a non-production repository in the example configuration file

* Update references to `cron.sh` or `ENV`

* Remove `cron.sh`
  • Loading branch information
dbeatty10 authored Nov 2, 2022
1 parent e5e4591 commit 8b6a48b
Show file tree
Hide file tree
Showing 4 changed files with 30 additions and 117 deletions.
36 changes: 17 additions & 19 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

### Personal access token (PAT)

Follow [these](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) instructions to configure a PAT.
Follow [these](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) instructions to configure a PAT in GitHub.

Scopes:
- repo (and all sub-items)
Expand All @@ -17,7 +17,7 @@ Save the token to a secure location. Click the "Configure SSO" button and "Autho
```shell
cp config.example.json config.json

# Add your GitHub username and token
# Add the relevant GitHub username, email address and token
$EDITOR config.json

# Export the JSON credentials into an environment variable
Expand All @@ -44,29 +44,27 @@ PYTHONPATH=hubcap python -m pytest

## Run in test mode

```shell
export ENV=test
./cron.sh
```

### Optional configuration environment variables
```shell
# Default value is the `git-tmp` directory within the current working directory
# This directory will be deleted by default at the end of the run
export GIT_TMP=git-tmp
```
- Set `"repo": "hub.getdbt.com-test"` within `config.json` (or specify some other non-production repository).
- Optional: set `"push_branches": false` within `config.json`.

### Optional parameters
Preserve commits/build artifacts within the `$GIT_TMP` directory
Run:
```shell
export ENV=test
./cron.sh --no-cleanup
python3 hubcap/hubcap.py
```

## Run in production mode

**WARNING:** Use with caution -- _will_ modify state.
- Set `"repo": "hub.getdbt.com"` within `config.json` (since [hub.getdbt.com](https://github.com/dbt-labs/hub.getdbt.com) is the production repository).
- Set `"push_branches": true` within `config.json`.

Run:
```shell
export ENV=prod
./cron.sh
python3 hubcap/hubcap.py
```

### Optional configuration environment variables
```shell
# Default value is the `git-tmp` directory within the current working directory
export GIT_TMP=git-tmp
```
18 changes: 12 additions & 6 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,8 @@ This application is hosted on [Heroku](https://www.heroku.com), but it can also

## Design overview
It is designed to do the following:
1. Use a cron schedule to execute the `cron.sh` script at the beginning of every hour
1. The `cron.sh` script creates a `git-tmp` directory that is destroyed upon completion
2. The `cron.sh` script defers most of the processing to `hubcap.py`
1. Use a cron schedule to execute the main `hubcap.py` script at the beginning of every hour
2. The `hubcap.py` script creates a `git-tmp` directory to hold cloned git repositories
3. `hubcap.py` creates a JSON spec file for each package+version combo within the `git-tmp/hub/data/packages/` directory
4. It opens pull requests against [dbt-labs/hub.getdbt.com](https://github.com/dbt-labs/hub.getdbt.com) for any new versions of dbt packages
- [Example PR for first-time package](https://github.com/dbt-labs/hub.getdbt.com/pull/1681/files)
Expand All @@ -17,12 +16,11 @@ It is designed to do the following:
The commands below assume a production application named `dbt-hubcap`. Replace with `dbt-hubcap-staging` for the staging version of the application.

1. Use the [Heroku Scheduler](https://dashboard.heroku.com/apps/dbt-hubcap/scheduler) to set the following cron schedule:
- Job: `./cron.sh`
- Job: `python3 hubcap/hubcap.py`
- Schedule: Every hour at :00
- Dyno size: Hobby / Standard-1X
1. Configure the `CONFIG` and `ENV` environment variables: [Settings > Config Vars > Reveal Config Vars](https://dashboard.heroku.com/apps/dbt-hubcap/settings)
1. Configure the `CONFIG` environment variable: [Settings > Config Vars > Reveal Config Vars](https://dashboard.heroku.com/apps/dbt-hubcap/settings)
- `CONFIG`: copy format from `config.example.json` and adjust values as needed
- `ENV`: `prod`
1. (Re-)deploy the application using the instructions below. See [these](https://dashboard.heroku.com/apps/dbt-hubcap/deploy/heroku-git) instructions for context.


Expand Down Expand Up @@ -55,6 +53,14 @@ heroku git:remote -a dbt-hubcap
git push heroku main:main
```

### Ad hoc executions of the script

For off-schedule ad hoc executions, run the following from the deploy directory above:

```shell
heroku run python3 hubcap/hubcap.py
```

#### Explanation

`heroku` is the remote that Heroku will use for deploys. `origin` is the source code hosted on GitHub.
Expand Down
2 changes: 1 addition & 1 deletion config.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"token": "pe4s0n@l-@cce$$-t0k3n"
},
"org": "dbt-labs",
"repo": "hub.getdbt.com",
"repo": "hub.getdbt.com-test",
"push_branches": true,
"one_branch_per_repo": true
}
91 changes: 0 additions & 91 deletions cron.sh

This file was deleted.

0 comments on commit 8b6a48b

Please sign in to comment.