This repository has been archived by the owner on Jan 25, 2022. It is now read-only.
Fetch singer-io tap codebase #4
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update PK for `ads_insights_country` * Update test expectations
* Bump to v1.15.0, update changelog * Fix changelog headers
* Update streams list in README; fix typo * Remove hourly audience breakdown Co-authored-by: Andy Lu <[email protected]>
* Bump facebook_business to 11.0.0 * Get on the latest sdk version
* Reorganize config file * Fix missing key * Make ensure_env run first * Run unit tests and pylint in parallel * Add checkouts to every job * Name steps, add parallelism to Unit Tests * Only run if you have tests * Parallelize integration tests; serialize unit tests * Acutally run integration tests * Fix copy pasta * Add logging around what tests are running * Fix typo * Remove unused code * Add slack notifications for failures * Reorder work to shrink the diff * Add build_daily back in * Add the version line back in * Update docker image * Fix run command for the newer docker image * Decouple tests without parallelism * Fix build_daily too * Fix path to tests * fail if we are missing a tap-tester test in the circle-config * reuse commit workflow as build-daily * fix typo * Add build job for github status * Add missing context * Refer to names of other jobs instead of the template? * Remove yaml reuse * Add deploy step to commit workflow, list jobs for build_daily workflow * Add filter to only deploy from master * Fix copy pasta * Add notifications on success and on failure Co-authored-by: kspeer <[email protected]>
mavimo
approved these changes
Oct 6, 2021
* Add sleep and retry for insights job polling * correct dylans typo * added tests, got tests passing * Makes api_get_with_retry function that uses retry wrapper * Style changes * Style changes 2 * Version Bump * removed count
* TDL-9728: Stream `ads_insights_age_gender` has unexpected datatype for replication key field `date_start` (#172) * added format as date-time in schema file * added code coverage * added check for date format in the bookmark test * added the check for first sync messages Co-authored-by: namrata270998 <[email protected]> * TDL-9809: `forced-replication-method` missing from metadata for some streams and TDL-9872: replication keys are not specified as expected in discoverable metadata (#167) * added valid replication keys in catalog * modified the code * TDL-9809: Added replication keys in metadata * adde code coverage * Resolved review comments Co-authored-by: harshpatel4_crest <[email protected]> Co-authored-by: namrata270998 <[email protected]> * TDL-7455: Add tap-tester test to verify replication of deleted records (#168) * TDL-7455: Added archived data integration test * TDL-7455: Updated integration test * added code coverage * Resolved review comment Co-authored-by: namrata270998 <[email protected]> * TDL-7596: Add tap-tester test for attribution window (#169) * added tap tester test for attribution window * updated the code * added code coverage * updated the code according to the comments * updated code to raise error when attribution window is not 1, 7, 28 * test: run invalid attribution window intergation test * updated test case * test: updated test case code * test: test invalid attribution window * test: test invalid attribution window * test: test invalid attribution window * test: test invalid attribution window * test: run invalid attribution window test case * added intergation test for invalid sttribution window Co-authored-by: namrata270998 <[email protected]> Co-authored-by: namrata270998 <[email protected]> Co-authored-by: savan-chovatiya <[email protected]>
Co-authored-by: KrishnanG <[email protected]>
…ve requests that are not retrying (#171) * TDL-6148: Added retry for Attribute error of sync batches * TDL-6148: Removed unused imports from unit tests * TDL-13267: Added retry for 500 error of AdCreatives * TDL-6148: Add AttributeError backoff for all sync functions * added code coverage * Resolved review comment * Resolved review comments * Added code comments * Resolved review comment * TDL-9728: Stream `ads_insights_age_gender` has unexpected datatype for replication key field `date_start` (#172) * added format as date-time in schema file * added code coverage * added check for date format in the bookmark test * added the check for first sync messages Co-authored-by: namrata270998 <[email protected]> * TDL-9809: `forced-replication-method` missing from metadata for some streams and TDL-9872: replication keys are not specified as expected in discoverable metadata (#167) * added valid replication keys in catalog * modified the code * TDL-9809: Added replication keys in metadata * adde code coverage * Resolved review comments Co-authored-by: harshpatel4_crest <[email protected]> Co-authored-by: namrata270998 <[email protected]> * TDL-7455: Add tap-tester test to verify replication of deleted records (#168) * TDL-7455: Added archived data integration test * TDL-7455: Updated integration test * added code coverage * Resolved review comment Co-authored-by: namrata270998 <[email protected]> Co-authored-by: namrata270998 <[email protected]> Co-authored-by: Harsh <[email protected]> Co-authored-by: harshpatel4_crest <[email protected]> Co-authored-by: KrisPersonal <[email protected]>
* Bump to version 1.18.1 * Wrong Syntax Co-authored-by: KrishnanG <[email protected]>
* TDL-15863: Added request timeout and unit tests * TDL-15863: Added unit tests * TDL-15863: Removed commented code * TDL-15863: Added request_timeout lookup from config also * added code coverage * TDL-15863: Added float type cast for timeout * Updated comment * added code change for empty string timeout value from config * Added empty string handling in request_timeout param * Resolved review comment * Added if else for request timeout * Added backoff for ConnectionError * TDL-9728: Stream `ads_insights_age_gender` has unexpected datatype for replication key field `date_start` (#172) * added format as date-time in schema file * added code coverage * added check for date format in the bookmark test * added the check for first sync messages Co-authored-by: namrata270998 <[email protected]> * TDL-9809: `forced-replication-method` missing from metadata for some streams and TDL-9872: replication keys are not specified as expected in discoverable metadata (#167) * added valid replication keys in catalog * modified the code * TDL-9809: Added replication keys in metadata * adde code coverage * Resolved review comments Co-authored-by: harshpatel4_crest <[email protected]> Co-authored-by: namrata270998 <[email protected]> * TDL-7455: Add tap-tester test to verify replication of deleted records (#168) * TDL-7455: Added archived data integration test * TDL-7455: Updated integration test * added code coverage * Resolved review comment Co-authored-by: namrata270998 <[email protected]> Co-authored-by: namrata270998 <[email protected]> Co-authored-by: harshpatel4_crest <[email protected]> Co-authored-by: Harsh <[email protected]> Co-authored-by: KrisPersonal <[email protected]>
Co-authored-by: KrishnanG <[email protected]>
* move build time and notifications * parallel integration tests with 'glob', break out deploy to multiple jobs' * parallelism = 9 * remove unnecessary check file * push_pypi waits only on git_tag
* remove setuptools and pip upgrade * pin deps Co-authored-by: kspeer <[email protected]>
* Added test cases or attribution window * added a decorator to skip the test
*using tap-tester logger * remove singer import * appease Dylan (PR feedback) * fix attr window test - broken with previous change * update field selection test * fix invalid attr window and start date tests Co-authored-by: kspeer <[email protected]>
* bump to v13 * Change CHANGElog and setup.py Co-authored-by: KrisPersonal <[email protected]> Co-authored-by: KrishnanG <[email protected]>
* save test logs, use tt logger * fix logger usage Co-authored-by: kspeer <[email protected]>
* use test data from 2021 * update bookmarks test for insights latest data * fix start date test, and timdelta_formatted implementation * account for conversion window in start date choice * fix timedelta formatted usage in attr window test Co-authored-by: kspeer <[email protected]>
* Bump facebook API to v14.0 which is used in sdk facebook_business version 14.0.0 * New tap-facebook bump version 1.18.4
* upgrade SDK to v16.0.2 * version bump and CHANGELOG
Co-authored-by: Ubuntu <[email protected]>
* conversions * feature - add conversion to all streams
* TDl-19503 * new base file
* add retry logic for status code - 503 * handle the attribute error by checking the error body instance is dictionary * Update error body in test case and indent the code * setup and changelog update
…d JIRA card is done (#228)
* All fields Test using new framework * Add method to exclude the non-upsert fields * Implemented PR Review comments * excluded the problematic streams
…g fields, document in jira (#233)
* Facebook test client WIP, campaigns stream only * Add pagination test for campaigns stream, update new base file and client * First review comments, add adsets stream, start work on ads * Support for adcreative, ads, adsets, and campaigns, get only for ads_insights * Clean up for next review round * PR review, clean up for further review * PR review comments round 2, pass limit and date range to test client get request * Convert times to utc to compare all datetime objects as tz naive * Use base_suite parse_date * Final review comments, add TODO to update get_account_objects(), update BaseCase page_size(), update self.start_date pattern
* test tap-facebook on python 3.11 * version bump and changelog update * standard image base -----------------------------
* upgrade the facebook sdk version * place the facebook_business sdk locally and add retry for the summary param error * Monkey patch the original function(call) of facebook business sdk * update logger statement
* update facebook sdk * update changelog * Exclude failing streams * Attach jira ticket * Fail and update the all fields test if outstanding JIRA cards are moved to status of done * exclude adinsights streams from the tests * exclude streams from the testing * update reset test * exclude insights stream from the other tests as well * skip the attribution window test * * refactor integration tests * * fix integration test * * Fix review comments --------- Co-authored-by: btowles <[email protected]> Co-authored-by: RushiT0122 <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.