Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨Source Snapchat Marketing: Add ability to break-down the Stats streams by report_dimensions query parameter #30761

Closed
wants to merge 4 commits into from
Closed

✨Source Snapchat Marketing: Add ability to break-down the Stats streams by report_dimensions query parameter #30761

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d763f62
Add ability to break-down the Stats streams by the report_dimensions …
pabloescoder Sep 26, 2023
0a1d4f5
Update PR number and link in docs
pabloescoder Sep 26, 2023
a0c0633
Fix merge conflict
pabloescoder Feb 12, 2024
f7026ab
Make report_dimension an enum
pabloescoder Feb 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-snapchat-marketing/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ COPY source_snapchat_marketing ./source_snapchat_marketing
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.3.0
LABEL io.airbyte.version=0.4.0
LABEL io.airbyte.name=airbyte/source-snapchat-marketing
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-snapchat-marketing/metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 200330b2-ea62-4d11-ac6d-cfe3e3f8ab2b
dockerImageTag: 0.3.0
dockerImageTag: 0.4.0
dockerRepository: airbyte/source-snapchat-marketing
githubIssueLabel: source-snapchat-marketing
icon: snapchat.svg
Expand Down
27 changes: 27 additions & 0 deletions ...s/connectors/source-snapchat-marketing/source_snapchat_marketing/schemas/basic_stats.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,33 @@
},
"story_completes": {
"type": ["null", "number"]
},
"country": {
"type": ["null", "string"]
},
"region": {
"type": ["null", "string"]
},
"dma": {
"type": ["null", "string"]
},
"gender": {
"type": ["null", "string"]
},
"age_bucket": {
"type": ["null", "string"]
},
"operating_system": {
"type": ["null", "string"]
},
"make": {
"type": ["null", "string"]
},
"interest_category_id": {
"type": ["null", "number"]
},
"interest_category_name": {
"type": ["null", "string"]
}
}
}
111 changes: 106 additions & 5 deletions ...yte-integrations/connectors/source-snapchat-marketing/source_snapchat_marketing/source.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,82 @@
"earned_reach",
]

# Maps which type of metric is supported by which dimension
# Refer: https://marketingapi.snapchat.com/docs/#reporting-insights-and-dimensions
DIMENSION_METRIC_DICT = {
"country": "Delivery/Conversion",
"country,os": "Delivery",
"os,country": "Delivery",
"region": "Delivery",
"dma": "Delivery",
"gender": "Delivery/Conversion",
"age": "Delivery/Conversion",
"age,gender": "Delivery/Conversion",
"gender,age": "Delivery/Conversion",
"os": "Delivery/Conversion",
"make": "Delivery",
"lifestyle_category": "Delivery"
}

# Refer: https://marketingapi.snapchat.com/docs/#reporting-insights-and-dimensions
DELIVERY_METRICS = [
"impressions",
"swipes",
"quartile_1",
"quartile_2",
"quartile_3",
"view_completion",
"spend",
"video_views",
"attachment_quartile_1",
"attachment_quartile_2",
"attachment_quartile_3",
"attachment_view_completion",
"attachment_total_view_time_millis",
"attachment_video_views",
"story_opens",
"story_completes",
"shares",
"saves"
]

# Refer: https://marketingapi.snapchat.com/docs/#reporting-insights-and-dimensions
CONVERSION_METRICS = [
"total_installs",
"conversion_purchases",
"conversion_purchases_value",
"conversion_save",
"conversion_start_checkout",
"conversion_add_cart",
"conversion_view_content",
"conversion_add_billing",
"conversion_sign_ups",
"conversion_searches",
"conversion_level_completes",
"conversion_app_opens",
"conversion_page_views",
"conversion_subscribe",
"conversion_ad_click",
"conversion_ad_view",
"conversion_complete_tutorial",
"conversion_invite",
"conversion_login",
"conversion_share",
"conversion_reserve",
"conversion_achievement_unlocked",
"conversion_add_to_wishlist",
"conversion_spend_credits",
"conversion_rate",
"conversion_start_trial",
"conversion_list_view",
"conversion_visit",
"custom_event_1",
"custom_event_2",
"custom_event_3",
"custom_event_4",
"custom_event_5"
]

logger = logging.getLogger("airbyte")


Expand Down Expand Up @@ -165,10 +241,11 @@ class SnapchatMarketingStream(HttpStream, ABC):
primary_key = "id"
raise_on_http_errors = True

def __init__(self, start_date, end_date, **kwargs):
def __init__(self, start_date, end_date, report_dimension="", **kwargs):
super().__init__(**kwargs)
self.start_date = start_date
self.end_date = end_date
self.report_dimension = report_dimension

def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, Any]]:
next_page_cursor = response.json().get("paging", False)
Expand Down Expand Up @@ -246,7 +323,7 @@ def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, Any]]]:
self.initial_state = stream_state.get(self.cursor_field) if stream_state else self.start_date
self.max_state = self.initial_state

parent_stream = self.parent(authenticator=self.authenticator, start_date=self.start_date, end_date=self.end_date)
parent_stream = self.parent(authenticator=self.authenticator, start_date=self.start_date, end_date=self.end_date, report_dimension=self.report_dimension)
stream_slices = get_parent_ids(parent_stream)

if stream_slices:
Expand Down Expand Up @@ -368,7 +445,7 @@ def parent(self) -> SnapchatMarketingStream:
def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, Any]]]:
"""Each stream slice represents each entity id from parent stream"""

parent_stream = self.parent(authenticator=self.authenticator, start_date=self.start_date, end_date=self.end_date)
parent_stream = self.parent(authenticator=self.authenticator, start_date=self.start_date, end_date=self.end_date, report_dimension=self.report_dimension)
self.parent_name = parent_stream.name
stream_slices = get_parent_ids(parent_stream)

Expand All @@ -390,6 +467,19 @@ def request_params(
params["granularity"] = self.granularity.value
if self.metrics:
params["fields"] = ",".join(self.metrics)
if self.report_dimension:
params["report_dimension"] = self.report_dimension
# Filter metrics based on the type supported by the provided report_dimension
new_metrics = []
supported_type = DIMENSION_METRIC_DICT.get(self.report_dimension)
for metric in self.metrics:
if supported_type == "Delivery/Conversion":
if metric in DELIVERY_METRICS or metric in CONVERSION_METRICS:
new_metrics.append(metric)
else:
if metric in DELIVERY_METRICS:
new_metrics.append(metric)
params["fields"] = ",".join(new_metrics)

return params

Expand All @@ -407,7 +497,12 @@ def parse_response(
response=response, stream_state=stream_state, stream_slice=stream_slice, next_page_token=next_page_token
):
# move all 'stats' metrics to root level
record.update(record.pop("stats", {}))
# if report dimension is specified, they are found under 'dimension_stats' instead of 'stats'
# Example for report dimension output here: https://marketingapi.snapchat.com/docs/#insights-by-demo
if self.report_dimension:
record.update(record.pop("dimension_stats", {}))
else:
record.update(record.pop("stats", {}))

yield record

Expand Down Expand Up @@ -524,7 +619,12 @@ def parse_response(
# add common record identifiers
stat_item.update(record_identifiers)
# move all 'stats' metrics to root level
stat_item.update(stat_item.pop("stats", {}))
# if report dimension is specified, they are found under 'dimension_stats' instead of 'stats'
# Example for report dimension output here: https://marketingapi.snapchat.com/docs/#insights-by-demo
if self.report_dimension:
stat_item.update(stat_item.pop("dimension_stats", {}))
else:
stat_item.update(stat_item.pop("stats", {}))

# Update state for last record in the stream
self.update_state_after_last_record(stat_item)
Expand Down Expand Up @@ -808,6 +908,7 @@ def streams(self, config: Mapping[str, Any]) -> List[Stream]:
),
"start_date": config["start_date"],
"end_date": config.get("end_date", default_end_date),
"report_dimension": config.get("report_dimension", "")
}

return [
Expand Down
8 changes: 8 additions & 0 deletions ...yte-integrations/connectors/source-snapchat-marketing/source_snapchat_marketing/spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,14 @@
"examples": ["2022-01-30"],
"order": 4,
"format": "date"
},
"report_dimension": {
pabloescoder marked this conversation as resolved.
Show resolved Hide resolved
"type": "string",
"title": "Report Dimension",
"description": "Used to get reporting insights based on different geographic, demographic, interest-based, and device-based dimensions.\n For usage refer: https://marketingapi.snapchat.com/docs/#reporting-insights-and-dimensions",
"pattern": "^(country|country,os|os,country|region|dma|gender|age|age,gender|gender,age|os|make|lifestyle_category)$",
"examples": ["country", "age", "os"],
"order": 5
}
}
},
Expand Down
41 changes: 21 additions & 20 deletions docs/integrations/sources/snapchat-marketing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,23 +111,24 @@ Snapchat Marketing API has limitations to 1000 items per page.

## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:--------------------------------------------------------------|
| 0.3.0 | 2023-05-22 | [26358](https://github.com/airbytehq/airbyte/pull/26358) | Remove deprecated authSpecification in favour of advancedAuth |
| 0.2.0 | 2023-05-10 | [25948](https://github.com/airbytehq/airbyte/pull/25948) | Introduce new field in the `Campaigns` stream schema |
| 0.1.16 | 2023-04-20 | [20897](https://github.com/airbytehq/airbyte/pull/20897) | Add missing fields to Basic Stats schema |
| 0.1.15 | 2023-03-02 | [22869](https://github.com/airbytehq/airbyte/pull/22869) | Specified date formatting in specification |
| 0.1.14 | 2023-02-10 | [22808](https://github.com/airbytehq/airbyte/pull/22808) | Enable default `AvailabilityStrategy` |
| 0.1.13 | 2023-01-27 | [22023](https://github.com/airbytehq/airbyte/pull/22023) | Set `AvailabilityStrategy` for streams explicitly to `None` |
| 0.1.12 | 2023-01-11 | [21267](https://github.com/airbytehq/airbyte/pull/21267) | Fix parse empty error response |
| 0.1.11 | 2022-12-23 | [20865](https://github.com/airbytehq/airbyte/pull/20865) | Handle 403 permission error |
| 0.1.10 | 2022-12-15 | [20537](https://github.com/airbytehq/airbyte/pull/20537) | Run on CDK 0.15.0 |
| 0.1.9 | 2022-12-14 | [20498](https://github.com/airbytehq/airbyte/pull/20498) | Fix output state when no records are read |
| 0.1.8 | 2022-10-05 | [17596](https://github.com/airbytehq/airbyte/pull/17596) | Retry 429 and 5xx errors when refreshing access token |
| 0.1.6 | 2022-07-21 | [14924](https://github.com/airbytehq/airbyte/pull/14924) | Remove `additionalProperties` field from specs |
| 0.1.5 | 2022-07-13 | [14577](https://github.com/airbytehq/airbyte/pull/14577) | Added stats streams hourly, daily, lifetime |
| 0.1.4 | 2021-12-07 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update titles and descriptions |
| 0.1.3 | 2021-11-10 | [7811](https://github.com/airbytehq/airbyte/pull/7811) | Add oauth2.0, fix stream_state |
| 0.1.2 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies |
| 0.1.1 | 2021-07-29 | [5072](https://github.com/airbytehq/airbyte/pull/5072) | Fix bug with incorrect stream\_state value |
| 0.1.0 | 2021-07-26 | [4843](https://github.com/airbytehq/airbyte/pull/4843) | Initial release supporting the Snapchat Marketing API |
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------------------------|
| 0.4.0 | 2023-09-24 | [30761](https://github.com/airbytehq/airbyte/pull/30761) | Add ability to break-down the Stats streams by the report_dimensions param |
| 0.3.0 | 2023-05-22 | [26358](https://github.com/airbytehq/airbyte/pull/26358) | Remove deprecated authSpecification in favour of advancedAuth |
| 0.2.0 | 2023-05-10 | [25948](https://github.com/airbytehq/airbyte/pull/25948) | Introduce new field in the `Campaigns` stream schema |
| 0.1.16 | 2023-04-20 | [20897](https://github.com/airbytehq/airbyte/pull/20897) | Add missing fields to Basic Stats schema |
| 0.1.15 | 2023-03-02 | [22869](https://github.com/airbytehq/airbyte/pull/22869) | Specified date formatting in specification |
| 0.1.14 | 2023-02-10 | [22808](https://github.com/airbytehq/airbyte/pull/22808) | Enable default `AvailabilityStrategy` |
| 0.1.13 | 2023-01-27 | [22023](https://github.com/airbytehq/airbyte/pull/22023) | Set `AvailabilityStrategy` for streams explicitly to `None` |
| 0.1.12 | 2023-01-11 | [21267](https://github.com/airbytehq/airbyte/pull/21267) | Fix parse empty error response |
| 0.1.11 | 2022-12-23 | [20865](https://github.com/airbytehq/airbyte/pull/20865) | Handle 403 permission error |
| 0.1.10 | 2022-12-15 | [20537](https://github.com/airbytehq/airbyte/pull/20537) | Run on CDK 0.15.0 |
| 0.1.9 | 2022-12-14 | [20498](https://github.com/airbytehq/airbyte/pull/20498) | Fix output state when no records are read |
| 0.1.8 | 2022-10-05 | [17596](https://github.com/airbytehq/airbyte/pull/17596) | Retry 429 and 5xx errors when refreshing access token |
| 0.1.6 | 2022-07-21 | [14924](https://github.com/airbytehq/airbyte/pull/14924) | Remove `additionalProperties` field from specs |
| 0.1.5 | 2022-07-13 | [14577](https://github.com/airbytehq/airbyte/pull/14577) | Added stats streams hourly, daily, lifetime |
| 0.1.4 | 2021-12-07 | [8429](https://github.com/airbytehq/airbyte/pull/8429) | Update titles and descriptions |
| 0.1.3 | 2021-11-10 | [7811](https://github.com/airbytehq/airbyte/pull/7811) | Add oauth2.0, fix stream_state |
| 0.1.2 | 2021-11-08 | [7499](https://github.com/airbytehq/airbyte/pull/7499) | Remove base-python dependencies |
| 0.1.1 | 2021-07-29 | [5072](https://github.com/airbytehq/airbyte/pull/5072) | Fix bug with incorrect stream\_state value |
| 0.1.0 | 2021-07-26 | [4843](https://github.com/airbytehq/airbyte/pull/4843) | Initial release supporting the Snapchat Marketing API |
Loading