Re-submitting PR singer-io#140 (Add additional base test suite tests) (…

…singer-io#145)

* add mac to gitignore

* change testing repo

* adjust `start_date_2` for new repo

* add tap-tester automatic fields

* add tap-tester all fields

* add all expected streams to all fields test

* set specific bookmark for test-repo

* fix `collaborators` stream bookmark spelling  for tap-tester

* add more streams to automatic fields test

* add tap-tester bookmarks

* updates to automatic fields test:
* add check for unique primary keys in replicated records
* replace explicit set of expected streams with `expected_check_streams` from base
* update `test_run` doc string

* omit `team_memberships` stream from `expected_check_streams`

* build expected_check_streams() using expected_streams()

* add bug id and description

* pylint fixes:
* adjust imports
* use specific error class
* set encoding

* adjust circle config:
* use latest image
* trim pylint disable options
* add unit tests step
* make sure integration tests run always

* Re-submitting PR singer-io#141 (All repos for an organization) (singer-io#146)

* add parsing of "org/*" wildcard to retreive all repos for an org

* add unit test cases for extract_repos_from_config()

* add requests-mock for dev requirements

* add unit test for get_all_repos()

* Re-submitting PR singer-io#142 (Improve Rate Limiting and Retry Logic) (singer-io#143)

* add basic backoff retry

* add backoff to setup.py

* replace deprecated assertEquals method with assertEqual

* add MAX_SLEEP_SECONDS parsing from config and DEFAULT_MAX_SECONDS for rate limiting

* add comments for changes and use DEFAULT_SLEEP_SECONDS

* add pylint ignore for global-statement

* README updates: (singer-io#148)

- add full list of replicated streams
- update GitHub docs links

* add streams to excluded_streams that aren't respecting automatic fields

* add NotFoundException handling to collaborators stream

* add bug info

* adjust test for test-repo

* pylint fixes

* run unit and integations steps always

* don't raise a NotFoundException to deal with access issues to resources

* pylint fix and return empty response body for 404

* add collaborators stream to excluded set

* fixes to tap-tester tests

* adjust 2nd sync dates for data

* deal with None from get method

* adjust start date tap-tester dates

* actually deal with NoneType in get method

* FIX: sub_streams sync functions passed parent metadata

* remove expected_check_streams after bug identified and addressed

* use expected_streams after bug identified and addressed

* don't write a bookmark for FULL_TABLE streams

* update unit test expectations to recent changes

* updates to bookmarks tap-tester:
* adjust test expectatons for streams
* create simulated_states based on test data and tap behavior
* adjust tests based on commits and pr_commits schema

* update base based on tap behavior and test data

* adjust test expectations for team_members stream

* Exclude collaborators stream due to access issues in circle

* update circle config to include slack orb and tap-tester-user context

* add bug info

* add tap-tester-user to daily build context

.circleci/config.yml

@@ -1,8 +1,11 @@
-version: 2
+version: 2.1
+orbs:
+  slack: circleci/[email protected]
+
 jobs:
   build:
     docker:
-      - image: 218546966473.dkr.ecr.us-east-1.amazonaws.com/circle-ci:tap-tester-v4
+      - image: 218546966473.dkr.ecr.us-east-1.amazonaws.com/circle-ci:stitch-tap-tester
     steps:
       - checkout
       - run:
@@ -21,14 +24,15 @@ jobs:
           name: 'pylint'
           command: |
             source /usr/local/share/virtualenvs/tap-github/bin/activate
-            pylint tap_github --disable 'broad-except,chained-comparison,empty-docstring,fixme,invalid-name,line-too-long,missing-class-docstring,missing-function-docstring,missing-module-docstring,no-else-raise,no-else-return,too-few-public-methods,too-many-arguments,too-many-branches,too-many-lines,too-many-locals,ungrouped-imports,wrong-spelling-in-comment,wrong-spelling-in-docstring,bad-whitespace'
+            pylint tap_github --disable 'missing-module-docstring,missing-function-docstring,missing-class-docstring,line-too-long,invalid-name,too-many-lines,consider-using-f-string,too-many-arguments,too-many-locals'
       - run:
           name: 'Unit Tests'
           command: |
             source /usr/local/share/virtualenvs/tap-github/bin/activate
             pip install nose coverage
             nosetests --with-coverage --cover-erase --cover-package=tap_github --cover-html-dir=htmlcov tests/unittests
             coverage html
+          when: always
       - store_test_results:
           path: test_output/report.xml
       - store_artifacts:
@@ -39,20 +43,19 @@ jobs:
             aws s3 cp s3://com-stitchdata-dev-deployment-assets/environments/tap-tester/tap_tester_sandbox dev_env.sh
             source dev_env.sh
             source /usr/local/share/virtualenvs/tap-tester/bin/activate
-            run-test --tap=tap-github \
-                     --target=target-stitch \
-                     --orchestrator=stitch-orchestrator \
-                     [email protected] \
-                     --password=$SANDBOX_PASSWORD \
-                     --client-id=50 \
-                     --token=$STITCH_API_TOKEN \
-                     tests
+            run-test --tap=tap-github tests
+          when: always
+      - slack/notify-on-failure:
+          only_for_branches: master
+
 workflows:
   version: 2
   commit:
     jobs:
       - build:
-          context: circleci-user
+          context:
+            - circleci-user
+            - tap-tester-user
   build_daily:
     triggers:
       - schedule:
@@ -63,4 +66,6 @@ workflows:
                 - master
     jobs:
       - build:
-          context: circleci-user
+          context:
+            - circleci-user
+            - tap-tester-user

.gitignore

@@ -97,3 +97,8 @@ properties.json
 
 # Jetbrains IDE
 .idea
+
+# macOS
+*.DS_Store
+.AppleDouble
+.LSOverride

README.md

@@ -2,20 +2,32 @@
 
 This is a [Singer](https://singer.io) tap that produces JSON-formatted
 data from the GitHub API following the [Singer
-spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
+spec](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md).
 
 This tap:
 - Pulls raw data from the [GitHub REST API](https://developer.github.com/v3/)
 - Extracts the following resources from GitHub for a single repository:
-  - [Assignees](https://developer.github.com/v3/issues/assignees/#list-assignees)
-  - [Collaborators](https://developer.github.com/v3/repos/collaborators/#list-collaborators)
-  - [Commits](https://developer.github.com/v3/repos/commits/#list-commits-on-a-repository)
-  - [Issues](https://developer.github.com/v3/issues/#list-issues-for-a-repository)
-  - [Pull Requests](https://developer.github.com/v3/pulls/#list-pull-requests)
-  - [Comments](https://developer.github.com/v3/issues/comments/#list-comments-in-a-repository)
-  - [Reviews](https://developer.github.com/v3/pulls/reviews/#list-reviews-on-a-pull-request)
-  - [Review Comments](https://developer.github.com/v3/pulls/comments/)
-  - [Stargazers](https://developer.github.com/v3/activity/starring/#list-stargazers)
+  - [Assignees](https://docs.github.com/en/rest/reference/issues#list-assigneess)
+  - [Collaborators](https://docs.github.com/en/rest/reference/repos#list-repository-collaborators)
+  - [Commits](https://docs.github.com/en/rest/reference/repos#list-commits)
+  - [Commit Comments](https://docs.github.com/en/rest/reference/repos#list-commit-comments-for-a-repository)
+  - [Events](https://docs.github.com/en/rest/reference/issues#events)
+  - [Issues](https://docs.github.com/en/rest/reference/issues#list-repository-issues)
+  - [Issue Events](https://docs.github.com/en/rest/reference/issues#list-issue-events-for-a-repository)
+  - [Issue Milestones](https://docs.github.com/en/rest/reference/issues#list-milestones)
+  - [Projects](https://docs.github.com/en/rest/reference/projects#list-repository-projects)
+  - [Project Cards](https://docs.github.com/en/rest/reference/projects#list-project-cards)
+  - [Project Columns](https://docs.github.com/en/rest/reference/projects#list-project-columns)
+  - [Pull Requests](https://docs.github.com/en/rest/reference/pulls#list-pull-requests)
+  - [PR Commits](https://docs.github.com/en/rest/reference/pulls#list-commits-on-a-pull-request)
+  - [Releases](https://docs.github.com/en/rest/reference/repos#list-releases)
+  - [Comments](https://docs.github.com/en/rest/reference/issues#list-issue-comments-for-a-repository)
+  - [Reviews](https://docs.github.com/en/rest/reference/pulls#list-reviews-for-a-pull-request)
+  - [Review Comments](https://docs.github.com/en/rest/reference/pulls#list-review-comments-in-a-repository)
+  - [Stargazers](https://docs.github.com/en/rest/reference/activity#list-stargazers)
+  - [Teams](https://docs.github.com/en/rest/reference/teams#list-teams)
+  - [Team Members](https://docs.github.com/en/rest/reference/teams#list-team-members)
+  - [Team Memberships](https://docs.github.com/en/rest/reference/teams#get-team-membership-for-a-user)
 - Outputs the schema for each resource
 - Incrementally pulls data based on the input state
 

setup.py

@@ -11,13 +11,15 @@
       py_modules=['tap_github'],
       install_requires=[
           'singer-python==5.12.1',
-          'requests==2.20.0'
+          'requests==2.20.0',
+          'backoff==1.8.0'
       ],
       extras_require={
           'dev': [
               'pylint==2.6.2',
               'ipdb',
               'nose',
+              'requests-mock==1.9.3'
           ]
       },
       entry_points='''