Upgrade to 1.4.4 (#21)

* Use sentry in production w/o heroku * Update legislators-current.yaml * update six version, per #18 * Normalize district padding before redis lookup * Use 0 for at large districts * Fix ternary syntax * unicode formatted string names * Hot fix for districts without current representatives, for OpenSourceActivismTech#39 * Compare strings by value, not identity * Check against None for district (can be 0) * ActionKit sync WIP * Update CSP * Site credit to OpenSourceActivism.tech * Add EU countries and languages * Update site credit * remove links to sunlight * Allow campaigns to make international calls * EUDataProvider loads no stored data * Return targeting in initial call/create json response * allow_intl_calls display in campaign form * Limit call/create target_response to campaign.call_maximum * Direct integration readme * explicitly allow wss in CSP * Re-shuffle matched targets if campaign.include_special == only * move us open data sources to OpenSourceActivismTech ownership * TargetList collection sort numerically * Blocked phone numbers display as e164 * Limit call/create requests to 1 / hour per phone number. Logged-in admins are exempt. * Exempt admins from filter even if not logged in, by checking User models * Ensure rate-limit responses are proper JSON * increase rate-limit to 2/hr (in case first call gets disconnected) * Statistics display error message if target_calls cannot load * Bump twilio client for safari 11 webrtc support * AudioRecording version selection also sets parent formGroup valid * Bump version 1.3.8 * Improve campaign copy experience: enforce unique name (OpenSourceActivismTech#63), and link existing audio recordings (OpenSourceActivismTech#79) * Store optional referral code on call session, expose via API * Update governor data, match new column heading for state_abbr and name * Update default CSP to include additional cloudflare, twilio, new relic domains * Limit referral_code to 64 characters * Display scheduled call stats on launch page * set uwsgi http-timeout, to handle slow clients before heroku does * readme update * Bump version to 1.3.9 * document ref parameter on call /create * Political data search ignore accented characters, which users may not know are included in source * Scheduled call check subscription status in create_call job, in case database and rq are no longer in sync * Improve target_calls performance * Further simplify target_calls queries * Campaign target_calls data fallback for custom targets * Campaign statistics secondary sort on last name * bump version 1.3.10 * Campaign phone number set field description * crm sync use rq scheduler, to perform jobs by crontab * Manual sync job schedule * Update CRM campaign meta with completed call count * Bump version number 1.4.beta-1 * Admin ability to list scheduled calls, and delete them on request * schedule routes login required * Campaign audio validate presence of text-to-speech before save * Manager command to bulk cancel scheduled calls before a specific date * gevent-psycopg deprecated, won’t install any longer. switch to psycogreen * CRM sync meta to actionkit requires use of XMLRPC api * Cross-origin headers for external blueprints with flask-cors * Update political data (Chaffetz -> Curtis, Becerra -> Gomez) * update us_districts * Fix date comparison in dashboard * Simplify call-in zipcode validation * Retry zipcode->latlon geocode if local cache lookup didn’t have it * Catch empty string before sending to geocoder * CSP img-src, not image-src * Political data update: Trent Franks and John Conyers resigned * Pass scheduled param in rq.job to create call * New AL, MN senators * District office data cleanup by @arrighik * Run loadpoliticaldata before bringing up production server. * Add location field for target, so we can read it in msg_target_intro * Shuffle campaign targets within chamber, defaults to True for US Senate * Normalize state name/abbr for US Governors * Remove complex handlebars syntax from msg defaults * More compelling schedule glyphicons * Update python runtime, psycopg2 * Launch campaign order more descriptive * Campaign target_display include location, if available * Fast-forward CRMSync alembic migration * CRM Sync overview doc * Release 1.4.0 * readme link * Custom target hide null location * Wrap target.location display in parentheses * Trigger focusout to fix placeholder display * set target default location to DC for US Congress campaigns for 202 numbers * Fix default location for search results * US governors update * Fix typo * Adjust special_targets matching to use string startswith, to include district offices * ditto * Same, but maintain ordering by using lists instead of sets * US Executive connect to Whitehouse switchboard instead of comment line * Historical political data update * US political data, add testing zips in AZ NY * District office phone number check more robust * Add note about ADMIN_API_KEY environment variable * Improve local template debugging by removing jinja cache when using runserver * Improve overlay modal, OpenSourceActivismTech#82 * Embed 429 handler, for OpenSourceActivismTech#93 * Embed integration options * For OpenSourceActivismTech#93 use alert instead of onError, because we actually want to inform the person. * Enable optional frontend error tracking with Sentry * Update US political data (Sen Tiberi resigned) * Improve handling of locally cached zipcodes to geocoder * Campaign target ordering place Special first if matches user location Rename Campaign.constants, use in tests, migrate data * Fix broken OpenStates test * remove missing zipcode tests, now that geocoder is hooked up it may provide a response * Bump version * Doug Jones offices, for unitedstates/congress-legislators#548 * Improve and document rate limit * US State governor data adapter fixes * Validate mime type of audio upload * Fix whitespace * Save audio files w/ correct extesnion * Use libmagic to detect audio file type * Migrate from gevent-psycopg to psycogreen pip now requires that packages be downloaded over HTTPS. Installing gevent-psycopg triggers a download over HTTP. The recommended solution to the HTTP problem is to upgrade to the newest version of the offending package. gevent-psycopg is deprecated in favor or psycogreen, so I'm migrating to that instead. I'm not sure what we're using this for :( * Remove unneeded imports * Use FileStorage.mimetype consistently * Keep original extension of uploaded file * Target data adapters fallback, display openstates legislator district number * US political data update * Include office_type as ‘district’ or ‘constituency’ * disable app-wide ADMIN_PHONES_LIST when testing * Update political data * improve ScheduledCall searching, sending, blocking * Inline schedule edit * Scheduled call blocklist fixes * Enable alter prompt for scheduled calls, Skip scheduling if we create new subscription * ScheduleCall logging * resolve flask_login warning * release 1.4.2 * Fix TargetOffice name/location attributes * improve Target and TargetOffice database caching semantics * For OpenStates legislator offices, remove “office” and “#1” from name * Ensure custom targets have phone number before we try to connect with them * Location repr and attr improvements * If one call fails, move on to next instead of hanging up * Explicitly skip duplicating scheduled call subscriptions when copying existing campaigns * scheduleSkip param now must be truthy, to avoid loop * Simple approach to OpenSourceActivismTech#95, adds party prioritization to campaign target_order choices * Manager command to reset scheduled jobs, in case redis is cleared. * limit scheduled_call reset and outgoing calls to live campaigns, unset jobs when pausing or archiving * Bump version number * Fix str comparison, some offices in political_data have unicode values * Fix data_cache logging * Allow audio/mpeg file mimetype, not just audio/mp3 * bump version
colorofchange · Aug 24, 2018 · 05349c3 · 05349c3
1 parent 3e5bdc6
commit 05349c3
Show file tree

Hide file tree

Showing 96 changed files with 16,419 additions and 14,695 deletions.
diff --git a/INSTALLATION.md b/INSTALLATION.md
@@ -33,7 +33,8 @@ For production, you will also need to set:
 * DATABASE_URI, a sqlalchemy [connection string](https://pythonhosted.org/Flask-SQLAlchemy/config.html#connection-uri-format) for a postgres or mysql database addresses
 * REDIS_URL, a URI for the Redis server
 * APPLICATION_ROOT to the path where the application will live. If you are using a whole domain or subdomain, this *SHOULD NOT* be defined. Otherwise, it will mess up cookie handling and cause CSRF 400 errors on login.
-* SERVER_NAME to the domain or subdomain on which the application will live
+* SERVER_NAME to the domain or subdomain on which the application will live (if this is not set, external urls will default to localhost)
+* CALL_RATE_LIMIT, the maximum number of allowed calls to a phone number for each campaign, to limit abuse potential. Admin phone numbers and logged in users are exempt. Defaults to "2 / hour", and must be specified in [flask-limit notation](https://flask-limiter.readthedocs.io/en/stable/#rate-limit-string-notation).
 
 If you are storing assets on Amazon S3, or another [Flask-Store provider](http://flask-store.soon.build)
 

diff --git a/INTEGRATION_API.md b/INTEGRATION_API.md
@@ -2,7 +2,7 @@
 REST API
 ===========
 
-A read-only REST API is available for integrating with external services. Access is allowed by passing an api_key parameter, or through flask user authentication. Results can be filtered by any field, using a [flask-restless](http://flask-restless.readthedocs.org/en/latest/searchformat.html) formatted parameter.
+A read-only REST API is available for integrating with external services. Access is allowed by passing an api_key parameter that is equal to the `ADMIN_API_KEY` environment variable, or through flask user authentication. Results can be filtered by any field, using a [flask-restless](http://flask-restless.readthedocs.org/en/latest/searchformat.html) formatted parameter.
 
 `GET /api/calls` returns a paginated list of Call objects. Fields include:
 
@@ -51,3 +51,13 @@ A read-only REST API is available for integrating with external services. Access
 
 `GET /api/campaign/ID/target_calls.json` returns a list of campaign calls by target and status. Results can be further limited by passing start or end parameters in ISO format.
 
+## Public read-only routes
+
+For display on external sites, these routes do not require authentication. Results are cached for 10 minutes.
+
+`GET /api/campaign/ID/count.json` returns calls aggregate statistics for a campaign
+
+* completed: total calls completed
+* last_24h: calls completed in last 24 hours
+* last_week: calls completed in the last 7 days
+* referral_codes: a hash of referral codes which have generated at least two completed calls
diff --git a/INTEGRATION_CRM.md b/INTEGRATION_CRM.md
@@ -0,0 +1,18 @@
+CRM Integrations
+===========
+
+CallPower is designed to keep as little data as required to place calls connecting their members to targets. However, it can be useful for some organizations to track the duration of individual calls, to identify conversations that may be useful for staff to follow-up.
+
+An out-of-band sync system copies call information to a CRM, joining user records on their phone number or other unique ID, and storing the duration of the call, which target(s) they were connected to, and the overall status of their call session.
+
+This is run nightly with rq jobs, which requires running a scheduler and at least one worker.
+
+Configuration
+----
+
+ActionKit
+
+- `CRM_INTEGRATION=ActionKit`
+- `ACTIONKIT_DOMAIN=client.actionkit.com`
+- `ACTIONKIT_USER`
+- `ACTIONKIT_PASSWORD` or `ACTIONKIT_API_KEY`
diff --git a/INTEGRATION_JS.md b/INTEGRATION_JS.md
@@ -10,6 +10,10 @@ Additional parameters can be specified on the `CallPowerOptions` object before i
 * form: form id to attach, defaults to 'call_form'
 * phoneField: input id with the user phone number, defaults to 'phone_id'
 * locationField: input id with the user location, defaults to 'location_id'
+* scriptDisplay: one of 'replace', 'redirect', or 'overlay'
+* if scriptDisplay is 'redirect', redirectAfter will set the new URL
+* if scriptDisplay is 'overlay', overlayCloseText: will add a link below the responseText
+* if errorTracking is defined, errors will be reported to the SENTRY_DSN public endpoint via Raven.js
 
 To render a complete form, include `<iframe src="/api/campaign/ID/embed_iframe.html"></iframe>">` to create a form ready to be filled out.
 
@@ -19,3 +23,36 @@ Custom Embeds
 The embed script should interoperate with existing forms, but if you have a form with a submit callback already defined, you may want to write your own integration. You can add javascript that will be run after the success callback in the Custom JS field. For example, if you are using the overlay script display, you might want to manually trigger an action after the overlay is closed, like `$('.overlay').on('hide', function() { actionkit.form.submit(); });`
 
 If your validation needs are more complex, you can include just CallPowerForm.js from `/api/campaign/ID/CallPowerForm.js` and define your own functions for location, phone, onSuccess or onError.
+
+Calling Endpoints Directly
+--------------------------
+
+You can also trigger a phone call by hitting the `/call/create` endpoint directly with either a GET or a POST. URL parameters should include:
+
+* userPhone (required)
+* campaignId (required)
+* userLocation (optional)
+* userCountry (optional)
+* ref (optional, limited to 64 characters)
+
+The response will be a JSON object with the following fields:
+
+```
+{
+  "call": "queued", 
+  "campaign": "live", "paused", or "archived", 
+  "fromNumber": "+18005551212", 
+  "redirect": null or a URL to redirect to on success, 
+  "script": "" or an HTML blob to display to the user, 
+  "targets": {
+    "segment": "custom",
+    "objects": [
+      {
+        "name": "", 
+        "phone": "", 
+        "title": ""
+      }, 
+    ]
+    or "display": "Congress" (a string to display for campaigns which do not have pre-set custom targets)
+}
+```
diff --git a/OPEN_DATA_SOURCES.md b/OPEN_DATA_SOURCES.md
@@ -9,7 +9,7 @@ Data is provided under the original license of the data provider.
 
 * US Congress contact data from [UnitedStates/congress](https://github.com/unitedstates/congress), public domain
 * US Governors contact data scraped from [National Governors Association](https://github.com/spacedogXYZ/us_governors_contact), all rights reserved
-* US States legislature contact data from [OpenStates API](http://sunlightlabs.github.io/openstates-api/)
+* US States legislature contact data from [OpenStates API](http://docs.openstates.org/en/latest/api/)
 
 * Canadan Member of Parliament contact data from [Represent](http://represent.opennorth.ca), licenses available from [OpenNorth](https://github.com/opennorth/represent-canada-data)
 

diff --git a/README.md b/README.md
@@ -26,7 +26,7 @@ Campaign Configuration
 1) Create a campaign with one of several types, to determine how callers are matched to targets.
 
 * _Executive_ connects callers to the Whitehouse Switchboard, or to a specific office if known
-* _Congress_ connects callers to their Senators, Representative, or both. Uses data from the Sunlight Foundation.
+* _Congress_ connects callers to their Senators, Representative, or both. Uses data from the TheUnitedStates.io.
 * _State_ connects callers to their Governor or State Legislators. Uses the OpenStates API.
 * _Local_ connects callers to a local official. Campaigners must enter these numbers in advance.
 * _Custom_ can connect callers to corporate offices, local officals, or any other phone number entered in advance.
@@ -44,6 +44,8 @@ For most uses, you can just place the `<script>` tag provided in the launch page
 
 For more complex integrations, Call Power provides [javascript embeds](INTEGRATION_JS.md) and a full [json API](INTEGRATION_API.md).
 
+Individual actions can also be synced to CRMs, via a [batch process](INTEGRATION_CRM.md).
+
 Installation Instructions
 -------------------
 This application should be easy to host on Heroku, with Docker, or directly on any WSGI-compatible server. Requires Python, flask, a SQL database (we recommend Postgres, but Mysql should work), Redis or Memcache, and an SMTP server.

diff --git a/alembic/versions/3254c70a6f37_campaign_target_shuffle_chamber.py b/alembic/versions/3254c70a6f37_campaign_target_shuffle_chamber.py
@@ -0,0 +1,30 @@
+"""campaign target shuffle chamber
+
+Revision ID: 3254c70a6f37
+Revises: f8e4f3786603
+Create Date: 2018-01-11 14:53:00.218981
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '3254c70a6f37'
+down_revision = 'f8e4f3786603'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+
+def upgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('campaign_campaign', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('target_shuffle_chamber', sa.Boolean(), nullable=True, server_default='1'))
+    ### end Alembic commands ###
+
+
+def downgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('campaign_campaign', schema=None) as batch_op:
+        batch_op.drop_column('target_shuffle_chamber')
+    ### end Alembic commands ###
diff --git a/alembic/versions/4f2d9aae4765_target_office_location_type.py b/alembic/versions/4f2d9aae4765_target_office_location_type.py
@@ -0,0 +1,33 @@
+"""target_office location and type
+
+Revision ID: 4f2d9aae4765
+Revises: d024eda790a3
+Create Date: 2018-04-26 10:42:20.052516
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '4f2d9aae4765'
+down_revision = 'd024eda790a3'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('campaign_target_office', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('latlon', sa.String(length=100), nullable=True))
+        batch_op.add_column(sa.Column('type', sa.String(length=100), nullable=True))
+        batch_op.drop_column('location')
+    ### end Alembic commands ###
+
+def downgrade():
+    with op.batch_alter_table('campaign_target_office', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('location', sa.VARCHAR(length=100), autoincrement=False, nullable=True))
+        batch_op.drop_column('type')
+        batch_op.drop_column('latlon')
+
+    ### end Alembic commands ###
diff --git a/alembic/versions/7f361698d677_call_session_referral_code.py b/alembic/versions/7f361698d677_call_session_referral_code.py
@@ -0,0 +1,32 @@
+"""call_session_referral_code
+
+Revision ID: 7f361698d677
+Revises: ae2d9ad59424
+Create Date: 2017-10-12 19:50:43.278732
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '7f361698d677'
+down_revision = 'ae2d9ad59424'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('calls_session', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('referral_code', sa.String(length=64), nullable=True))
+
+    ### end Alembic commands ###
+
+
+def downgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('calls_session', schema=None) as batch_op:
+        batch_op.drop_column('referral_code')
+
+    ### end Alembic commands ###
diff --git a/alembic/versions/817e83870d89_crmsync.py b/alembic/versions/817e83870d89_crmsync.py
@@ -0,0 +1,46 @@
+"""CRMSync
+
+Revision ID: 817e83870d89
+Revises: 3254c70a6f37
+Create Date: 2017-04-22 14:04:23.304971
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = '817e83870d89'
+down_revision = '3254c70a6f37'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    op.create_table('sync_campaign',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('job_id', sa.String(length=36), nullable=False),
+    sa.Column('created_time', sa.DateTime(), nullable=True),
+    sa.Column('last_sync_time', sa.DateTime(), nullable=True),
+    sa.Column('campaign_id', sa.Integer(), nullable=True),
+    sa.Column('crm_id', sa.String(length=40), nullable=True),
+    sa.ForeignKeyConstraint(['campaign_id'], ['campaign_campaign.id'], ),
+    sa.PrimaryKeyConstraint('id')
+    )
+    op.create_table('sync_call',
+    sa.Column('id', sa.Integer(), nullable=False),
+    sa.Column('created_time', sa.DateTime(), nullable=True),
+    sa.Column('call_id', sa.Integer(), nullable=True),
+    sa.Column('saved', sa.Boolean(), nullable=True),
+    sa.ForeignKeyConstraint(['call_id'], ['calls.id'], ),
+    sa.PrimaryKeyConstraint('id')
+    )
+    ### end Alembic commands ###
+
+
+def downgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table('sync_call')
+    op.drop_table('sync_campaign')
+    ### end Alembic commands ###
diff --git a/alembic/versions/ae2d9ad59424_campaign_allow_international_calls.py b/alembic/versions/ae2d9ad59424_campaign_allow_international_calls.py
@@ -0,0 +1,32 @@
+"""campaign allow international calls
+
+Revision ID: ae2d9ad59424
+Revises: a47e5ebb013e
+Create Date: 2017-08-30 15:21:06.406736
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'ae2d9ad59424'
+down_revision = 'a47e5ebb013e'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+
+
+def upgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('campaign_campaign', schema=None) as batch_op:
+        batch_op.add_column(sa.Column('allow_intl_calls', sa.Boolean(), nullable=True))
+
+    ### end Alembic commands ###
+
+
+def downgrade():
+    ### commands auto generated by Alembic - please adjust! ###
+    with op.batch_alter_table('campaign_campaign', schema=None) as batch_op:
+        batch_op.drop_column('allow_intl_calls')
+
+    ### end Alembic commands ###
diff --git a/alembic/versions/d024eda790a3_campaign_target_special_rename.py b/alembic/versions/d024eda790a3_campaign_target_special_rename.py
@@ -0,0 +1,63 @@
+"""campaign_target_special_rename
+
+Revision ID: d024eda790a3
+Revises: 817e83870d89
+Create Date: 2018-02-12 16:01:25.519837
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = 'd024eda790a3'
+down_revision = '817e83870d89'
+branch_labels = None
+depends_on = None
+
+from alembic import op
+import sqlalchemy as sa
+
+
+campaigns_helper = sa.Table(
+        'campaign_campaign',
+        sa.MetaData(),
+        sa.Column('id', sa.Integer, primary_key=True),
+        sa.Column('include_special', sa.String(length=100)),
+    )
+
+def upgrade():
+    connection = op.get_bind()
+
+    # rename include_special constants from first/last to before/after
+    connection.execute(
+        campaigns_helper.update().where(
+            campaigns_helper.c.include_special == 'first'
+        ).values(
+            include_special='before'
+        )
+    )
+    connection.execute(
+            campaigns_helper.update().where(
+                campaigns_helper.c.include_special == 'last'
+            ).values(
+                include_special='after'
+            )
+        )
+
+
+def downgrade():
+    connection = op.get_bind()
+
+    # reverse
+    connection.execute(
+        campaigns_helper.update().where(
+            campaigns_helper.c.include_special == 'before'
+        ).values(
+            include_special='first'
+        )
+    )
+    connection.execute(
+            campaigns_helper.update().where(
+                campaigns_helper.c.include_special == 'after'
+            ).values(
+                include_special='last'
+            )
+        )