Skip to content

Latest commit

 

History

History
283 lines (180 loc) · 10.7 KB

README.rst

File metadata and controls

283 lines (180 loc) · 10.7 KB

influencetx

influencetx: An ATX Hack for Change project for accessing Texas campaign finance and voting records.

Built with Cookiecutter Django

Setup

To get started on this project for the first time, you can follow these simple steps.

  • Install Docker CE

  • Clone code:

    cd your/code/directory
git clone https://github.com/open-austin/influence-texas.git
cd influence-texas

  • Define environment variables (see below) and export those variables:

    source env.sh

  • Start up docker container:

    docker-compose up -d

The first time it's run, docker-compose will pull down generic python and postgres images. After that, it will install dependendencies specific to the app and start up a server for the influencetx app at http://localhost:5120/.

Define environment variables

Credentials are stored as environment variables that are not committed to source control. To make your environment reproducible, you'll add these environment variables to a script named env.sh with the following values:

export OPENSTATES_API_KEY=YOUR-API-KEY
export TPJ_DB_USER=YOUR-USERNAME
export TPJ_DB_PASSWORD=YOUR-PASSWORD

The TPJ variables require credentials from Texans for Public Justice. Currently, there's no established process for acquiring those credentials. See the following section to acquire an OpenStates key.

When you start up a new shell, you should run the following to setup environment variables:

source env.sh

Note that changes to env.sh should never be committed.

Add Open States API Key

If you want to use portions of the site that rely on the Open States API, you'll need to add an API key to the secrets file.

  • Register for an Open States API key
    • Use your own name and email
    • Website: https://www.open-austin.org/
    • Organization: Open Austin
    • Intended Usage: Local development of influencetx app
  • You should receive an email with your new API key. Follow the activation link.
  • Copy key to env.sh.

Syncing data from Open States API

Custom django-admin commands are used to sync data from Open States API. To pull data for legislators and bills from Open States, run the following in order:

./djadmin.sh sync_legislators_from_openstate
./djadmin.sh sync_bills_from_openstate

Note that the order matters because bills have voting data which requires legislators to be in the database for correct attribution.:

./djadmin.sh sync_legislators_from_openstate --session 85

The number of bills in the database is quite large. For testing purposes, you can grab a subset of the data by replacing the second command above with:

./djadmin.sh sync_bills_from_openstate --max 100

Note: By default, sync_bills_from_openstate syncs data from the most recent session, but the most recent special session currently has some data missing, so you may want to use data from the 85th session instead:

./djadmin.sh sync_bills_from_openstate --max 100 --session 85

Basic Commands

During everyday development, there are a few commands that you'll need to execute to debug, update the database, etc. All of the basic commands are based off of the following commands for interacting with the docker container:

  • docker-compose: Run generic docker commands in docker containers.
    • Run docker-compose -h to see a full list of commands.
    • Run docker-compose help <COMMAND> to see help on a command.
  • ./pyinvoke.sh: A shortcut for running invoke commands in docker containers.
    • Run ./pyinvoke.sh -l to see a full list of commands.
    • Run ./pyinvoke.sh -h <COMMAND> to see help on a command.
  • ./djadmin.sh: A shortcut for running django admin commands in docker containers.
    • Run ./djadmin.sh help to see a full list of commands.
    • Run ./djadmin.sh help <COMMAND> to see help on a command.

These instructions assume you're executing the command from the parent directory of this file. You can find details of any commands using the commands above. A few commonly used commands are

Maintenance commands

The commands commonly used for maintenance of this project are described below.

  • docker-compose -f docker-compose.dev up -d: Start up docker container in detached mode (background task). You can keep a docker container running continuously, so you may only need to run this after restarting your machine.
  • ./djadmin.sh makemigrations: Make schema migrations to reflect your changes to Django models. Any migrations that you make should be committed and pushed with your model changes.
  • ./djadmin.sh migrate: Migrate database to the current schema. You'll need to run this after running ./djadmin.sh makemigrations to actually apply migrations. If you pull code from github that includes migrations, you should run this to sync your database.
  • ./pyinvoke.sh test: Execute tests using pytest. At minimum, run this before committing code.
  • ./pyinvoke.sh check: Check project for problems. At minimum, run this before committing code.
  • ./pyinvoke.sh create-app: Create Django app. Django apps are small collections of functionality for your web application.

Debugging commands

  • docker-compose logs -f --tail=5: Watch output of containers. (Alias: -f = --follow.)
  • docker-compose -f docker-compose.dev logs: Display bash output for all containers.
  • docker-compose -f docker-compose.dev exec web bash: Run bash shell within web container.
  • ./djadmin.sh shell: Start IPython shell.
  • ./djadmin.sh dbshell: Start Postgres shell.

Debugging Python code

You can't use the output window from a docker-compose logs --f call to debug, since it actually interacts with multiple containers. Instead, run the following in a terminal:

docker attach `docker-compose -f docker-compose.dev ps -q web`

The docker-compose-part of the command simply returns the id of the web container for the app. You can replace the above with:

docker attach influencetexas_web_1

This will attach the terminal to the web container and allow you to interact with the running process. Now you can add a break point somewhere in your python code:

import ipdb; ipdb.set_trace()

Settings

Moved to settings.

Vagrant

A Vagrant based deployment method is also available, which mirrors the configurations of the live integration/production server. It provides a virtual machine for running the postgresql database, and is configured as a docker host. The benefits to using an isolated VM for development is that your work is encapsulated within the VM, thereby allowing you to work on more than one project. Another benefit is that by developing in an environment that is the same as the integration/production servers, then a CI/CD pipeline can be setup. The primary reason for the vagrant environment was to provide a development environment for ansible development.

Pre-requisites

You must first install the following software to utilize the Vagrant development environment:

Usage

To start the virtual machine:

vagrant up

To stop the virtual machine:

vagrant halt

To open a terminal on the virtual machine:

vagrant ssh

To rebuild and deploy the application:

vagrant provision

Devlopment Workflow

There are two uses of the Vagrant environment for testing production deployments, from inside the VM or from outside the VM.

The Vagrant VM is run by default with the settings: ```

vb.memory = "2048" vb.cpus = "2"

```

Reduce these numbers for running on smaller hardware.

Internal

To perform development from inside the VM, perform the vagrant ssh command, then change directory to "/vagrant". The git repository is mounted automatically inside the VM at the "/vagrant" directory. The docker-compose files will be accessible, as well as the code which is built using the docker-compose files. You can perform updates to the code, then perform sudo docker-compose build command to generate the docker image inside the VM, then re-deploy the containers with the new images by issuing a sudo docker-compose up -d command. You can also make git commits and push, but you will need to use an HTTPS checkout and supply your github credentials with each git push

External

You can also perform development outside the VM by making code updates, then issuing a vagrant provision command. This method allows SSH based checkouts of the git repository.

Production Build and Deployment

This requires root privileges on the deployment server:

ssh [email protected]
cd influence-texas
git pull
docker-compose build
docker-compose up -d --force-recreate

The first docker-compose command builds the docker container with the influencetx codebase, and the second starts the web application and associated services.

You can access the logs on the production server using:

docker logs web