weekly opsbase update (#45)

Author: none

**Files changed (97):**
base/ops/config/modules/pillar/new_services_pillar.py (+0 -24 ~24)
base/ops/config/modules/pillar/v2_services.py (+0 -118 ~118)
base/ops/config/modules/pillar/workflow_repositories.py (+0 -16 ~16)
base/ops/config/modules/states/boto_dynamodb.py (+23 -6 ~29)
base/ops/config/pillar/deleted_iam_objects.sls (+1 -0 ~1)
base/ops/config/pillar/github_teams.sls (+18 -0 ~18)
base/ops/config/pillar/groups.sls (+81 -46 ~127)
base/ops/config/pillar/lyft_users.sls (+92 -55 ~147)
base/ops/config/pillar/nutcracker.sls (+25 -12 ~37)
base/ops/config/pillar/policies.sls (+40 -0 ~40)
base/ops/config/pillar/roles.sls (+21 -0 ~21)
base/ops/config/pillar/services.sls (+20 -44 ~64)
base/ops/config/states/base/logagent/hekad.toml (+1 -1 ~2)
base/ops/config/states/base/manage_install.sls (+0 -19 ~19)
base/ops/config/states/base/pulldeploy/check.py (+7 -4 ~11)
base/ops/config/states/base/pulldeploy/pulldeploy.py (+24 -17 ~41)
base/ops/config/states/base/salt/new_services_pillar.conf (+0 -2 ~2)
base/ops/config/states/base/services.json (+0 -1 ~1)
base/ops/config/states/envoy2/init.sls (+3 -3 ~6)
base/ops/config/states/lyft-python/gunicorn/gunicorn.conf (+3 -2 ~5)
base/ops/config/states/lyft-python/init.sls (+1 -1 ~2)
base/ops/orca/iam_roles.sls (+6 -0 ~6)
base/ops/orca/iam_users.sls (+0 -2 ~2)
base/tests/test_new_services_pillar.py (+0 -66 ~66)
base/tests/test_pulldeploy.py (+184 -0 ~184)
base/tests/test_pulldeploy_check.py (+103 -6 ~109)
bootartifact/Dockerfile (+1 -1 ~2)
buildnotify/Dockerfile (+1 -1 ~2)
buildnotify/Makefile (+1 -1 ~2)
buildnotify/buildnotify/ciexec.py (+9 -5 ~14)
buildnotify/buildnotify/cli.py (+1 -0 ~1)
buildnotify/buildnotify/delete_old_branches.py (+1 -0 ~1)
buildnotify/buildnotify/github.py (+1 -1 ~2)
buildnotify/buildnotify/routes.py (+8 -2 ~10)
buildnotify/buildnotify/services.py (+23 -16 ~39)
buildnotify/buildnotify/settings.py (+7 -0 ~7)
buildnotify/buildnotify/submit_queue.py (+28 -14 ~42)
buildnotify/buildnotify/tests/test_ciexec.py (+16 -11 ~27)
buildnotify/buildnotify/tests/test_jobs.py (+2 -2 ~4)
buildnotify/buildnotify/tests/test_routes.py (+10 -2 ~12)
buildnotify/buildnotify/tests/test_services.py (+565 -473 ~1038)
buildnotify/buildnotify/tests/test_submit_queue.py (+1 -0 ~1)
buildnotify/buildnotify/tests/test_v3_callbacks.py (+7 -3 ~10)
buildnotify/buildnotify/tests/test_v3_jenkins_jobs.py (+4 -1 ~5)
buildnotify/buildnotify/tests/test_v3_models.py (+3 -3 ~6)
buildnotify/buildnotify/v2_services.py (+27 -24 ~51)
buildnotify/buildnotify/v3_callbacks.py (+4 -2 ~6)
buildnotify/buildnotify/v3_jenkins.py (+4 -4 ~8)
buildnotify/buildnotify/v3_jenkins_jobs.py (+2 -1 ~3)
buildnotify/buildnotify/v3_manifests.py (+3 -6 ~9)
buildnotify/buildnotify/v3_models.py (+2 -2 ~4)
buildnotify/ops/config/pillar/buildnotify.sls (+2 -0 ~2)
buildnotify/requirements.txt (+25 -24 ~49)
buildnotify/scripts/prfairy.py (+1 -0 ~1)
buildslave/Makefile (+3 -0 ~3)
buildslave/deploy (+1 -1 ~2)
buildslave/ops/orca/buildslave.sls (+1 -1 ~2)
docker/onebox/starter (+2 -2 ~4)
docs/datastores-network/oncall-runbook.md (+2 -0 ~2)
docs/mongodb/runbook.md (+58 -0 ~58)
docs/redis/overview.md (+5 -9 ~14)
gateway/Dockerfile (+1 -1 ~2)
gateway/manifest.yaml (+1 -1 ~2)
gateway/ops/config/states/gateway/attach-elastic-ip.sh (+41 -0 ~41)
gateway/ops/config/states/gateway/init.sls (+16 -0 ~16)
gateway/ops/orca/gateway.sls (+12 -0 ~12)
grafana/Dockerfile (+1 -1 ~2)
hacktools/fabfile.py (+1 -1 ~2)
jenkins/apitestrevisions/force_update_last_good_revision.py (+6 -0 ~6)
jenkins/apitestrevisions/get_last_good_revision.py (+5 -1 ~6)
jenkins/apitestrevisions/send_failure_report.py (+17 -7 ~24)
jenkins/apitestrevisions/store_test_failure_report.py (+6 -1 ~7)
jenkins/apitestrevisions/tests/test_send_failure_report.py (+9 -0 ~9)
jenkins/docs/runbook.md (+9 -8 ~17)
jenkins/ops/config/states/jenkins/init.sls (+16 -0 ~16)
jenkins/ops/config/states/jenkins/templates/test-dummy.xml (+71 -0 ~71)
jenkins/ops/config/states/jenkins/templates/test_image.xml (+30 -16 ~46)
jenkins/requirements.txt (+1 -1 ~2)
opstools/Dockerfile (+1 -1 ~2)
opstools/cloudwatch2wavefront/cloudwatch.py (+6 -0 ~6)
opstools/dynamodb_restore/streams_to_dynamodb_restore_tool/streams_dynamodb_processor.py (+1 -0 ~1)
orca/Dockerfile (+1 -1 ~2)
orca/pillar/aws.sls (+5 -4 ~9)
redis/ops/orca/redis.sls (+1 -1 ~2)
statsd/Dockerfile (+1 -1 ~2)
statsd/docs/runbook.md (+3 -3 ~6)
statsd/ops/config/pillar/env.sls (+4 -0 ~4)
statsd/ops/config/pillar/top.sls (+5 -0 ~5)
statsd/ops/config/states/statsdbatch/init.sls (+3 -0 ~3)
statsd/ops/config/states/top.sls (+8 -0 ~8)
statsd/ops/orca/dynamo.sls (+16 -0 ~16)
statsd/ops/orca/statsd.sls (+68 -4 ~72)
statsd/requirements.txt (+31 -0 ~31)
statsd/scripts/__init__.py (+0 -0 ~0)
statsd/scripts/config-generator.py (+173 -0 ~173)
vpcproxy/Dockerfile (+1 -1 ~2)
vpn/requirements.txt (+0 -1 ~1)

**104 commits:**
[Fix failure report script (#9486)](lyft/ops_DEPRECATED@f5249ea)
[weekly opsbase update (#9489)](lyft/ops_DEPRECATED@4143c02)
[weekly opsbase update (#9490)](lyft/ops_DEPRECATED@29e9f6b)
[weekly opsbase update (#9491)](lyft/ops_DEPRECATED@88885fd)
[weekly opsbase update (#9492)](lyft/ops_DEPRECATED@e911c09)
[weekly opsbase update (#9493)](lyft/ops_DEPRECATED@4a1dd0b)
[weekly opsbase update (#9494)](lyft/ops_DEPRECATED@6349f05)
[weekly opsbase update (#9495)](lyft/ops_DEPRECATED@127d2f6)
[weekly opsbase update (#9496)](lyft/ops_DEPRECATED@16bc29f)
[PROV-796: Check.py fails with shadow traffic clusters (#9487)](lyft/ops_DEPRECATED@b503614)
[updated some destructive permissions (#9484)](lyft/ops_DEPRECATED@aa87494)
[DS-550: scale eventdedupe redis (#9499)](lyft/ops_DEPRECATED@876a387)
[add ridesapi slack channel (#9483)](lyft/ops_DEPRECATED@a0972c6)
[Backport features (#9500)](lyft/ops_DEPRECATED@a076e87)
[redis/docs: misc improvements (#9505)](lyft/ops_DEPRECATED@cfcf7eb)
[hacktools: fix nutcracker_sls typo (#9507)](lyft/ops_DEPRECATED@6e77e1c)
[Use a version of py-googleauth that does not pin deps (#9503)](lyft/ops_DEPRECATED@aaf0016)
[[SECHELP-160] Add mpaine to AWS (#9502)](lyft/ops_DEPRECATED@6c2d40a)
[updaing manifest to be in parity with services.sls (#9510)](lyft/ops_DEPRECATED@856d1d1)
[nutcracker: update eventdedupe config (#9504)](lyft/ops_DEPRECATED@d5c19be)
[Give myself keys to the kingdom (#9508)](lyft/ops_DEPRECATED@c6f3014)
[[SECHELP-135] Add brady to ats/images admin (#9509)](lyft/ops_DEPRECATED@1238961)
[Update runbook for buildnotify localdev to v3 devbox instructions. (#9512)](lyft/ops_DEPRECATED@4b7695d)
[[SEC-1149] Add lyft-data-sience-s3-access policy to Developers and Data_Platform. (#9498)](lyft/ops_DEPRECATED@c237301)
[Use manifest to determine upstream tests. (#9513)](lyft/ops_DEPRECATED@47a4c8c)
[Add ci_version to Dynamo tables (#9515)](lyft/ops_DEPRECATED@6b19a3c)
[TSDB archive service: just lots of S3 (#9517)](lyft/ops_DEPRECATED@59d8160)
[jenkins: fix curl to upload console to s3 (#9519)](lyft/ops_DEPRECATED@95e7eac)
[Fix vincentchang's unix name. (#9514)](lyft/ops_DEPRECATED@872203e)
[Fix log rotation in brochure service (#9518)](lyft/ops_DEPRECATED@ccd380b)
[envoy prod 773ff47816c63327896c4eea8fefce78e206b2b5 (#9524)](lyft/ops_DEPRECATED@7896212)
[Revert new services pillar (#9521)](lyft/ops_DEPRECATED@e67dd98)
[[SECHELP-163] Add bnorquist to snitch ro group (#9511)](lyft/ops_DEPRECATED@88d7dac)
[Update lyft_users.sls](lyft/ops_DEPRECATED@f03691b)
[Removed inactive developers (#9506)](lyft/ops_DEPRECATED@c117f88)
[Include timeout in GeventWorkerTimeoutException message (#9501)](lyft/ops_DEPRECATED@da7fe71)
[[SEC-727] add attach elastic ip script (#9456)](lyft/ops_DEPRECATED@2a1d83c)
[buildnotify: post to ciexec with json body (#9527)](lyft/ops_DEPRECATED@ac4fb35)
[fixed typo (#9529)](lyft/ops_DEPRECATED@2f36c75)
[Call build with parameters for dev (#9530)](lyft/ops_DEPRECATED@16ecfed)
[link to streaming schedule (#9531)](lyft/ops_DEPRECATED@ef2e7db)
[DS-438 - mongodb runbook - document maxconns problem (#9522)](lyft/ops_DEPRECATED@0b2554f)
[move elastic ip file location (#9533)](lyft/ops_DEPRECATED@18b8f92)
[jenkins: upgrade testrunner to include links (#9534)](lyft/ops_DEPRECATED@1f8711b)
[Remove upstream_tests definition for supplyalerts from services.sls (#9520)](lyft/ops_DEPRECATED@2d22abd)
[jenkins: put console output at the correct place for archival (#9536)](lyft/ops_DEPRECATED@79ac04a)
[jenkins: pick up the correct test runner :( (#9537)](lyft/ops_DEPRECATED@ddae473)
[opstools: dynamo streams restore add finish message (#9539)](lyft/ops_DEPRECATED@4ef0b70)
[Added Raman Malik (Marketing) (#9525)](lyft/ops_DEPRECATED@3e7a400)
[Trigger V3 dev jobs with parameters (#9532)](lyft/ops_DEPRECATED@3c7b60a)
[log whether or not we are using manifest for tests - because I am at my wits' end (#9541)](lyft/ops_DEPRECATED@b350805)
[Remove upstream definitions for api from services.sls (#9544)](lyft/ops_DEPRECATED@a2969d7)
[Revert "log whether or not we are using manifest for tests - because (#9541)" (#9546)](lyft/ops_DEPRECATED@03c2ce8)
[Remove "set -e" (#9538)](lyft/ops_DEPRECATED@daac6a8)
[fixed typo (#9548)](lyft/ops_DEPRECATED@b8f0b13)
[Fixed another problem with checking exit codes with set -e (#9549)](lyft/ops_DEPRECATED@5b04e3c)
[Allow runtime override to skip running V2 PR tests. (#9458)](lyft/ops_DEPRECATED@6c3dbe0)
[envoy deploy (#9551)](lyft/ops_DEPRECATED@2b4c444)
[Automation 1435 incentives float dep (#9526)](lyft/ops_DEPRECATED@8de4ed9)
[analyticsingestloadtester cannot deploy with new instances (#9555)](lyft/ops_DEPRECATED@87ccc1a)
[Update statsd dashboard links (#9556)](lyft/ops_DEPRECATED@35aa09c)
[Added sfo eips (#9554)](lyft/ops_DEPRECATED@82ffcd9)
[buildnotify: stringify all values for ciexec (#9559)](lyft/ops_DEPRECATED@6019e69)
[fixed a typo (#9561)](lyft/ops_DEPRECATED@9430a90)
[Give redshift-production-iad access to lyft-analytics and lyft-analytics-tmp S3 buckets. (#9552)](lyft/ops_DEPRECATED@b83b50f)
[get region from az (#9562)](lyft/ops_DEPRECATED@c40aaea)
[Fix high CPU alarms on buildnotify (#9563)](lyft/ops_DEPRECATED@30bd682)
[jenkins: factor in lyft/testrunner_DEPRECATED#328 (#9567)](lyft/ops_DEPRECATED@175ea96)
[Add moar types to Buildnotify. (#9566)](lyft/ops_DEPRECATED@f82feae)
[Tweak link description (#9557)](lyft/ops_DEPRECATED@0e37810)
[Give redshift-production-iad read access to the eta-models buckets. (#9568)](lyft/ops_DEPRECATED@2f2274b)
[Report status on V3 Dev jobs to Buildnotify (#9570)](lyft/ops_DEPRECATED@4e5d048)
[Remove references to devbox-default (#9571)](lyft/ops_DEPRECATED@fdba6a2)
[envoy 551b8dc38ec0ed31e9eb555792da73da4672e747 deploy. (#9574)](lyft/ops_DEPRECATED@ecc8c03)
[Update requirements for buildnotify (#9573)](lyft/ops_DEPRECATED@c56e8a8)
[Envoy 37e6e5b5224e18f27755f4751440e6879536a7ac (#9576)](lyft/ops_DEPRECATED@d2bcc79)
[Don't run Skipped V2 tests when pushing to master. (#9569)](lyft/ops_DEPRECATED@4c0fef6)
[Give S3 access back to Analytics group. (#9578)](lyft/ops_DEPRECATED@f752b2a)
[Fix a hole in our MFA device + AccessKey self-service policy. (#9575)](lyft/ops_DEPRECATED@2107678)
[[SECHELP-174] Erin Gray to github (#9565)](lyft/ops_DEPRECATED@92cfc98)
[[SECHELP-180] Add Jerrison Li to analyst db (#9580)](lyft/ops_DEPRECATED@58800d5)
[Factor out the v2_services logic that updates the upstream_tests map. (#9579)](lyft/ops_DEPRECATED@97a86a6)
[[SECHELP-166] Add Scott Owades to guardian, ridehistory, runtime ro r (#9582)](lyft/ops_DEPRECATED@02baa05)
[Get off slackclient v1.0.4 - I think it kills deploy_bot.](lyft/ops_DEPRECATED@b25e39f)
[Always hit v3_manifests when finding upstream tests. (#9585)](lyft/ops_DEPRECATED@617589a)
[Fix the handling of limited_test_set in v2_services.py (#9586)](lyft/ops_DEPRECATED@73c557e)
[offboard jthai (#9587)](lyft/ops_DEPRECATED@48291b4)
[statsd batch box (#9528)](lyft/ops_DEPRECATED@f1a06c2)
[grant security + data privacy ssh access to team (#9588)](lyft/ops_DEPRECATED@066ed9c)
[[SECHELP-167] Give amartine read access to requested repos (#9589)](lyft/ops_DEPRECATED@ac49691)
[Move Lift*MFADevices out of MFA-protected action, as we use them to bootstrap MFA in blessclient and awsaccess. (#9591)](lyft/ops_DEPRECATED@866b541)
[Update envoy sha (#9592)](lyft/ops_DEPRECATED@fcffea6)
[Add dynamo table where we store failure reports (#9595)](lyft/ops_DEPRECATED@e0d2d37)
[Default commits to an empty map (#9596)](lyft/ops_DEPRECATED@d3cbeca)
[[SECHELP-179] Add Jordeen Chang to financial repo (#9564)](lyft/ops_DEPRECATED@d523bb0)
[jenkins: add a test dummy job (#9597)](lyft/ops_DEPRECATED@4086573)
[Update envoy sha (#9598)](lyft/ops_DEPRECATED@995d819)
[Block mongo logs from Envoy from entering heka (#9599)](lyft/ops_DEPRECATED@ac4c1f6)
[use v3 tests for datatoolkit (#9581)](lyft/ops_DEPRECATED@091621d)
[Simplifying the upstream test process](lyft/ops_DEPRECATED@08b6c30)
[updated iam groups to expand desctructive actions (#9603)](lyft/ops_DEPRECATED@a53ceb2)
[Add TCS dependency to integration tests (#9594)](lyft/ops_DEPRECATED@1b47618)
[Add ability to override region name while creating alarms (#9560)](lyft/ops_DEPRECATED@4bd8046)
[Add clientbuilds development deploy step (#9606)](lyft/ops_DEPRECATED@d4a47bd)
NOT PASSING TESTS

Dockerfile

@@ -1,4 +1,4 @@
-FROM lyft/opsbase:d883c38f9da3a29967651e683cb97518c2c484f8
+FROM lyft/opsbase:d4a47bd56ccb8034b729de3ac4c1532cd1295700
 COPY . /code/statsrelay-private
 RUN apt-get update -y && \
     apt-get install -y software-properties-common python-software-properties && \