Zoltar forecast archive project

This is a Django project to implement a repository of forecast challenges. See the internal Forecast repository notes document for a more detailed description. The internal reichlab Slack channel for this is #forecast-repository. The GitHub location is https://github.com/reichlab/forecast-repository .

Python version

Zoltar requires Python 3.6 or higher.

Postgres version

Zoltar has been tested against Postgres 9.6.

Email-based notification requirements

Zoltar uses Anymail to abstract access to the transactional email server that's used for notifications. (Currently we only have notifications about file uploads.) Anymail can be used for a number of services, as configured in settings. Currently we use SendinBlue.

Configuration: The environment variable SENDINBLUE_API_KEY must be set, e.g., for Heroku:

heroku config:set \
  SENDINBLUE_API_KEY=<YOUR_SENDINBLUE_API_KEY>

AWS S3 configuration

Zoltar uses S3 for temporary storage of uploaded files (forecasts, truth, and templates). You'll need to set three S3-related environment variables, either locally or, for Heroku:

heroku config:set \
  S3_BUCKET_PREFIX=<reichlab_bucket_prefix> \
  AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY> \
  AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_KEY>

These keys must enable read, write, and list operations on a bucket named S3_BUCKET_PREFIX + object type in that account. (See cloud_file.py for details re: our bucket naming convention.) For development that account was configured as follows. In addition, the Job buckets were configured to delete all files after one day.

• (IAM) Zoltar app user:
  • no groups
  • Permissions > Permissions policies > Attached directly:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::mc.zoltarapp.sandbox",
                "arn:aws:s3:::mc.zoltarapp.sandbox/*"
            ]
        }
    ]
}

• (S3) Zoltar upload bucket:
  • Permissions > Access control list: default (root)
  • Permissions > Bucket policy: none (controlled above at the user level)

Requirements (see Pipfile)

• Python 3
• pipenv
• for Heroku hosting:
  • Heroku Toolbelt
  • Postgres
• pymmwr

To install required packages:

$ pipenv --three
$ cd <readme.md's dir>/forecast-repository
$ pipenv install

Pipfile was created via:

pipenv install django
pipenv install click
pipenv install requests
pipenv install jsonfield
pipenv install psycopg2-binary
pipenv install dj-database-url
pipenv install gunicorn
pipenv install whitenoise
pipenv install djangorestframework
pipenv install pymmwr
pipenv install pyyaml
pipenv install djangorestframework-csv
pipenv install django-debug-toolbar
pipenv install rq
pipenv install django-rq
pipenv install boto3
pipenv install djangorestframework-jwt
pipenv install more-itertools
pipenv install django-anymail[sendgrid,sendinblue]

RQ infrastructure

Zoltar uses an asynchronous messaging queue to support executing long-running tasks outside the web dyno, which keeps the latter responsive and prevents Heroku's 30 second timeouts. We use RQ for this, which requires a Redis server along with one or more worker dynos. Currently we use the free 30MB rediscloud Heroku addon.

Here's the setup to run locally:

1. Start Redis:

redis-server

1. Start an rq worker:

cd ~/IdeaProjects/django-redis-play
pipenv shell
export PATH="/Applications/Postgres.app/Contents/Versions/9.6/bin:${PATH}" ; export DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3 ; export PYTHONPATH=.
python3 manage.py rqworker

1. Optionally start monitor (rq info or rqstats):

cd ~/IdeaProjects/django-redis-play
pipenv shell
rq info --interval 1

# alternatively:
export PATH="/Applications/Postgres.app/Contents/Versions/9.6/bin:${PATH}" ; export DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3 ; export PYTHONPATH=.
python3 manage.py rqstats --interval 1

1. Start the web app

cd ~/IdeaProjects/django-redis-play
pipenv shell
export PATH="/Applications/Postgres.app/Contents/Versions/9.6/bin:${PATH}" ; export DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3 ; export PYTHONPATH=.
python3 manage.py runserver --settings=forecast_repo.settings.local_sqlite3

1. Optionally monitor the progress in the web app

• http://127.0.0.1:8000/zadmin

Running the tests

Note that testing or running Zoltar on sqlite requires version 3.25.0 or higher.

$ cd <readme.md's dir>/forecast-repository
$ pipenv shell
$ cd forecast_app/tests
$ python3 ../../manage.py test --verbosity 2 --settings=forecast_repo.settings.local_sqlite3

Django project layout

This project's settings scheme follows the "split settings.py into separate files in their own 'settings' module" approach. Since we plan on deploying to Heroku, there is no production.py. Regardless, every app needs to set the DJANGO_SETTINGS_MODULE environment variable accordingly, e.g., one of the following:

$ export DJANGO_SETTINGS_MODULE="forecast_repo.settings.local_sqlite3"
$ ./manage.py migrate --settings=forecast_repo.settings.local_sqlite3
$ heroku config:set DJANGO_SETTINGS_MODULE=forecast_repo.settings.local_sqlite3
gunicorn -w 4 -b 127.0.0.1:8001 --settings=forecast_repo.settings.local_sqlite3

Heroku deployment

The site is currently hosted by Heroku at https://reichlab-forecast-repository.herokuapp.com/ . Follow these steps to update it:

login

$ cd ~/IdeaProjects/forecast-repository
$ pipenv shell
$ heroku login

optional: dump local db then copy to remote

$ PGPASSWORD=password
$ pg_dump -Fc --no-acl --no-owner -h localhost -U username forecast_repo > /tmp/mc-1219-forecast_repo.dump

• upload to somewhere publicly accessible, e.g., Amazon S3

$ heroku pg:backups:restore 'https://s3.us-east-2.amazonaws.com/yourbucket/yourdatabase.dump' DATABASE_URL
$ heroku run python manage.py createsuperuser --settings=forecast_repo.settings.heroku_production

push code

$ git push heroku master

start web dyno

$ heroku ps:scale web=1
$ heroku open

static file configuration

This project's bin/ directory is via heroku-django-cookbook.

Following is from Static Files Made Easy — Django Compressor + Whitenoise + AWS CloudFront + Heroku:

$ heroku config:set DISABLE_COLLECTSTATIC=1