[TDL-24367] Fix streams implementation

CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 3.0.0
+  * Bump to API version `202309`
+  * Removed `FIELDS_UNACCEPTED_BY_API` beta fields & updated integration tests.
+  * Updated schema, replication key & primary key for "video_ads" stream
+  * Updated path for streams "campaign_groups", "campaigns", "creatives" & "video_ads"
+  * Unit & integration tests updated for version bump
+  * [#64](https://github.com/singer-io/tap-linkedin-ads/pull/64)
+  * [#65](https://github.com/singer-io/tap-linkedin-ads/pull/65)
+
 ## 2.1.0
   * Bump to API version `202302`
   * Move and update `FIELDS_UNACCEPTED_BY_API`

README.md

@@ -6,7 +6,7 @@ spec](https://github.com/singer-io/getting-started/blob/master/SPEC.md).
 
 This tap:
 
-- Pulls raw data from the [LinkedIn Marketing Ads July 2022](https://docs.microsoft.com/en-us/linkedin/marketing/)
+- Pulls raw data from the [LinkedIn Marketing Ads October 2023](https://docs.microsoft.com/en-us/linkedin/marketing/)
 - Extracts the following resources:
   - [Ad Accounts](https://docs.microsoft.com/en-us/linkedin/marketing/integrations/ads/account-structure/create-and-manage-accounts#search-for-accounts)
     - [Video Ads](https://docs.microsoft.com/en-us/linkedin/marketing/integrations/ads/advertising-targeting/create-and-manage-video#finders)
@@ -31,14 +31,14 @@ This tap:
 - Transformations: Fields camelCase to snake_case. URNs to ids. Unix epoch millisecond integers to date-times. Audit date-times created_at and last_modified_at de-nested. String to decimal for total_budget field.
 - Children: video_ads
 
-[**video_ads**](https://docs.microsoft.com/en-us/linkedin/marketing/integrations/ads/advertising-targeting/create-and-manage-video#finders)
-- Endpoint: https://api.linkedin.com/rest/adDirectSponsoredContents
-- Primary key field: content_reference
+[**video_ads**](https://learn.microsoft.com/en-us/linkedin/marketing/integrations/community-management/shares/posts-api?view=li-lms-2023-10&tabs=http#find-posts-by-account-1)
+- Endpoint: https://api.linkedin.com/rest/posts
+- Primary key field: id
 - Foreign keys: account_id (accounts), owner_organization_id (organizations)
 - Replication strategy: Incremental (query all, filter results)
   - Filter: account (from parent account) and owner (from parent account) (see NOTE below)
-  - Bookmark: last_modified_time (date-time)
-- Transformations: Fields camelCase to snake_case. URNs to ids. Unix epoch millisecond integers to date-times. Audit date-times created_at and last_modified_at de-nested.
+  - Bookmark: last_modified_at (date-time)
+- Transformations: Fields camelCase to snake_case. URNs to ids. Unix epoch millisecond integers to date-times. Ad context fields dsc_status, dsc_name, dsc_ad_type & dsc_ad_account de-nested.
 - Parent: account
 **NOTE**: The parent Account **MUST** reference and **Organization** (not a Person)
 - [Campaign Manager User Roles for Video Ads](https://www.linkedin.com/help/lms/answer/90733/campaign-manager-user-roles-for-video-ads?lang=en)
@@ -53,7 +53,7 @@ This tap:
 - Transformations: Fields camelCase to snake_case. URNs to ids. Unix epoch millisecond integers to date-times. Audit date-times created_at and last_modified_at de-nested.
 
 [**campaign_groups**](https://docs.microsoft.com/en-us/linkedin/marketing/integrations/ads/account-structure/create-and-manage-campaign-groups#search-for-campaign-groups)
-- Endpoint: https://api.linkedin.com/rest/adCampaignGroups
+- Endpoint: https://api.linkedin.com/rest/adAccounts/{account-id}/adCampaignGroups
 - Primary key field: id
 - Foreign keys: account_id (accounts)
 - Replication strategy: Incremental (query all, filter results)
@@ -63,7 +63,7 @@ This tap:
 - Transformations: Fields camelCase to snake_case. URNs to ids. Unix epoch millisecond integers to date-times. Audit date-times created_at and last_modified_at de-nested.
 
 [**campaigns**](https://docs.microsoft.com/en-us/linkedin/marketing/integrations/ads/account-structure/create-and-manage-campaigns#search-for-campaigns)
-- Endpoint: https://api.linkedin.com/rest/adCampaigns
+- Endpoint: https://api.linkedin.com/rest/adAccounts/{account-id}/adCampaigns
 - Primary key field: id
 - Foreign keys: account_id (accounts)
 - Replication strategy: Incremental (query all, filter results)
@@ -74,7 +74,7 @@ This tap:
 - Children: creatives, ad_analytics_by_campaign, ad_analytics_by_creative
 
 [**creatives**](https://learn.microsoft.com/en-us/linkedin/marketing/integrations/ads/account-structure/create-and-manage-creatives?view=li-lms-2023-01&tabs=http#search-for-creatives)
-- Endpoint: https://api.linkedin.com/rest/creatives
+- Endpoint: https://api.linkedin.com/rest/adAccounts/{account-id}/creatives
 - Primary key field: id
 - Foreign keys: campaign_id (campaigns)
 - Replication strategy: Incremental (query all, filter results)
@@ -118,11 +118,13 @@ The API user account should be assigned one of the following roles:
 - **VIEWER** (Recommended)
 
 The API user account should be assigned the following **permissions** for the API endpoints:
-- accounts, account_users, video_ads, campaign_groups, campaigns, creatives:
+- accounts, account_users, campaign_groups, campaigns, creatives:
     - r_ads: read ads (Recommended)
     - rw_ads: read-write ads
 - ad_analytics_by_campaign, ad_analytics_by_creative:
     - r_ads_reporting: read ads reporting
+- video_ads:
+    - r_organization_social: read video ads
 
 **NOTE**: Legacy permissions (r_ad_campaigns) have been migrated to the new permissions (r_ads and r_ads_reporting) based on this [permissions mapping](https://docs.microsoft.com/en-us/linkedin/shared/references/migrations/marketing-permissions-migration?context=linkedin/marketing/context). 
 

setup.py

@@ -3,7 +3,7 @@
 from setuptools import setup, find_packages
 
 setup(name='tap-linkedin-ads',
-      version='2.1.0',
+      version='3.0.0',
       description='Singer.io tap for extracting data from the LinkedIn Marketing Ads API API 2.0',
       author='[email protected]',
       classifiers=['Programming Language :: Python :: 3 :: Only'],

tap_linkedin_ads/__init__.py

@@ -19,9 +19,9 @@
 ]
 
 
-def do_discover(client, config):
+def do_discover(client):
     LOGGER.info('Starting discover')
-    client.check_accounts(config)
+    client.check_accounts()
     catalog = _discover()
     json.dump(catalog.to_dict(), sys.stdout, indent=2)
     LOGGER.info('Finished discover')
@@ -30,13 +30,13 @@ def do_discover(client, config):
 @singer.utils.handle_top_exception(LOGGER)
 def main():
     parsed_args = singer.utils.parse_args(REQUIRED_CONFIG_KEYS)
-    config = parsed_args.config
 
     with LinkedinClient(parsed_args.config.get('client_id', None),
                         parsed_args.config.get('client_secret', None),
                         parsed_args.config.get('refresh_token', None),
                         parsed_args.config.get('access_token'),
                         parsed_args.config_path,
+                        parsed_args.config,
                         REQUEST_TIMEOUT,
                         parsed_args.config['user_agent']
                         ) as client:
@@ -45,10 +45,9 @@ def main():
         if parsed_args.state:
             state = parsed_args.state
         if parsed_args.discover:
-            do_discover(client, config)
+            do_discover(client)
         elif parsed_args.catalog:
             _sync(client=client,
-                  config=config,
                   catalog=parsed_args.catalog,
                   state=state)
 

tap_linkedin_ads/client.py

@@ -11,7 +11,7 @@
 BASE_URL = 'https://api.linkedin.com/rest'
 LINKEDIN_TOKEN_URI = 'https://www.linkedin.com/oauth/v2/accessToken'
 INTROSPECTION_URI = 'https://www.linkedin.com/oauth/v2/introspectToken'
-LINKEDIN_VERSION = '202302'
+LINKEDIN_VERSION = '202309'
 
 # set default timeout of 300 seconds
 REQUEST_TIMEOUT = 300
@@ -128,12 +128,14 @@ def __init__(self, # pylint: disable=too-many-arguments
                  refresh_token,
                  access_token,
                  config_path,
+                 config,
                  request_timeout=REQUEST_TIMEOUT,
                  user_agent=None):
         self.__client_id = client_id
         self.__client_secret = client_secret
         self.__refresh_token = refresh_token
         self.__config_path = config_path
+        self.config = config
         self.__user_agent = user_agent
         self.__access_token = access_token
         self.__expires = None
@@ -175,7 +177,6 @@ def set_mock_expires_for_test(self, mock_expire):
         self.__expires = mock_expire
         return self.__expires
 
-
     def write_access_token_to_config(self):
         """
         Write an updated access token in the config to reuse in the next sync.
@@ -189,6 +190,7 @@ def write_access_token_to_config(self):
             config = json.load(file)
         # Set new access_token
         config['access_token'] = self.__access_token
+        self.config = config
 
         with open(self.__config_path, 'w') as file:
             json.dump(config, file, indent=2)
@@ -288,7 +290,8 @@ def fetch_and_set_access_token(self):
                           (Server5xxError, requests.exceptions.ConnectionError, requests.exceptions.Timeout),
                           max_tries=5,
                           factor=2)
-    def check_accounts(self, config):
+    def check_accounts(self):
+        config = self.config
         headers = {}
         if self.__user_agent:
             headers['User-Agent'] = self.__user_agent

tap_linkedin_ads/schema.py

@@ -6,14 +6,6 @@
 # Reference:
 #   https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#Metadata
 
-# The following fields of ads_analytics (...by_campaign and ...by_creative) were previously in beta and are not available on
-# API version 202302. Requesting them results in a 403.
-# https://docs.microsoft.com/en-us/linkedin/marketing/integrations/ads-reporting/ads-reporting?view=li-lms-2023-02&tabs=http#accuracy
-FIELDS_UNACCEPTED_BY_API = {
-    "average_daily_reach_metrics",
-    "average_previous_seven_day_reach_metrics",
-    "average_previous_thirty_day_reach_metrics",
-}
 
 def get_abs_path(path):
     return os.path.join(os.path.dirname(os.path.realpath(__file__)), path)
@@ -27,10 +19,6 @@ def get_schemas():
         with open(schema_path, encoding='utf-8') as file:
             schema = json.load(file)
 
-            if stream_name in ('ad_analytics_by_campaign', 'ad_analytics_by_creative'):
-                for field in FIELDS_UNACCEPTED_BY_API:
-                    metadata.delete(schema, 'properties', field)
-
         schemas[stream_name] = schema
         mdata = metadata.new()
 

tap_linkedin_ads/schemas/video_ads.json

@@ -5,110 +5,90 @@
   ],
   "additionalProperties": false,
   "properties": {
-    "account": {
+    "account_id": {
+      "type": [
+        "null",
+        "integer"
+      ]
+    },
+    "status": {
       "type": [
         "null",
         "string"
       ]
     },
-    "account_id": {
+    "name": {
       "type": [
         "null",
-        "integer"
+        "string"
+      ]
+    },
+    "type": {
+      "type": [
+        "null",
+        "string"
+      ]
+    },
+    "account": {
+      "type": [
+        "null",
+        "string"
       ]
     },
-    "change_audit_stamps": {
+    "ad_context": {
       "type": [
         "null",
         "object"
       ],
       "additionalProperties": false,
       "properties": {
-        "created": {
+        "dsc_status": {
+          "type": [
+            "null",
+            "string"
+          ]
+        },
+        "dsc_name": {
+          "type": [
+            "null",
+            "string"
+          ]
+        },
+        "dsc_ad_type": {
           "type": [
             "null",
-            "object"
-          ],
-          "additionalProperties": false,
-          "properties": {
-            "time": {
-              "type": [
-                "null",
-                "string"
-              ],
-              "format": "date-time"
-            }
-          }
+            "string"
+          ]
         },
-        "last_modified": {
+        "dsc_ad_account": {
           "type": [
             "null",
-            "object"
-          ],
-          "additionalProperties": false,
-          "properties": {
-            "time": {
-              "type": [
-                "null",
-                "string"
-              ],
-              "format": "date-time"
-            }
-          }
+            "string"
+          ]
         }
       }
     },
-    "created_time": {
+    "created_at": {
       "type": [
         "null",
         "string"
       ],
       "format": "date-time"
     },
-    "last_modified_time": {
+    "last_modified_at": {
       "type": [
         "null",
         "string"
       ],
       "format": "date-time"
     },
-    "content_reference": {
-      "type": [
-        "null",
-        "string"
-      ]
-    },
-    "content_reference_ucg_post_id": {
-      "type": [
-        "null",
-        "integer"
-      ]
-    },
-    "content_reference_share_id": {
-      "type": [
-        "null",
-        "integer"
-      ]
-    },
-    "name": {
-      "type": [
-        "null",
-        "string"
-      ]
-    },
-    "owner": {
+    "id": {
       "type": [
         "null",
         "string"
       ]
     },
-    "owner_organization_id": {
-      "type": [
-        "null",
-        "integer"
-      ]
-    },
-    "type": {
+    "author": {
       "type": [
         "null",
         "string"