Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Source Bing Ads: Stream budget and product dimension performance report #35201

Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
b3ddb10
added stream budget
darynaishchenko Feb 13, 2024
a3b14d9
added stream product dimensaion
darynaishchenko Feb 13, 2024
8c5d87a
updated streams
darynaishchenko Feb 13, 2024
1ae6396
added integration tests for new streams
darynaishchenko Feb 13, 2024
d3a6232
added resourse data for tests
darynaishchenko Feb 13, 2024
cb66910
updated streams value in unit test
darynaishchenko Feb 13, 2024
ead65d6
updated cat config with new streams
darynaishchenko Feb 13, 2024
3463422
updated docs and metadata
darynaishchenko Feb 13, 2024
3cb5312
format fix
darynaishchenko Feb 13, 2024
00a75a8
Merge branch 'master' into daryna/source-bing-ads/stream-budget-and-p…
darynaishchenko Feb 13, 2024
d73596e
updated changelog
darynaishchenko Feb 13, 2024
95d8e24
fixed cat config
darynaishchenko Feb 13, 2024
571146e
refactored code
darynaishchenko Feb 19, 2024
121cdbf
added BaseTest, updated TestBulkStreams and TestreportStreams
darynaishchenko Feb 20, 2024
897d2c3
updated client mock logic
darynaishchenko Feb 20, 2024
b07be18
fix hints
darynaishchenko Feb 20, 2024
92bb09d
updated product_dimension_performance_report data
darynaishchenko Feb 21, 2024
7b11e0f
updated TestSuiteReportStream with skipp for base test
darynaishchenko Feb 21, 2024
ba7c69d
remove redundant headers
darynaishchenko Feb 21, 2024
5d2f8fd
format fix
darynaishchenko Feb 21, 2024
4616384
updated state for streams
darynaishchenko Feb 21, 2024
ebd0921
deleted catalog builder
darynaishchenko Feb 21, 2024
c48aa81
updated BaseTest
darynaishchenko Feb 21, 2024
f4bae3b
updated response data
darynaishchenko Feb 21, 2024
a895fa4
updated TestBudgetStream
darynaishchenko Feb 21, 2024
e9346de
updated TestSuiteReportStream
darynaishchenko Feb 21, 2024
3374d11
updated test_return_records_from_given_csv_file for Budget stream test
darynaishchenko Feb 21, 2024
a6d89c2
updated resource data
darynaishchenko Feb 26, 2024
5380892
added unittest for ioe case
darynaishchenko Feb 26, 2024
b8e683f
updated integration tests with state validation in request
darynaishchenko Feb 26, 2024
ca13f4f
updated test dates for monthly report
darynaishchenko Feb 27, 2024
7156617
Merge branch 'master' into daryna/source-bing-ads/stream-budget-and-p…
darynaishchenko Feb 27, 2024
f55b223
updated vesrion in pyproject.toml
darynaishchenko Feb 27, 2024
ec0a696
test stream budget: fix date for incremental sync
darynaishchenko Feb 29, 2024
5a29581
budget and product: updated cat config
darynaishchenko Feb 29, 2024
aaad088
updated budget schema
darynaishchenko Feb 29, 2024
df4c4f8
Merge branch 'master' into daryna/source-bing-ads/stream-budget-and-p…
darynaishchenko Feb 29, 2024
66a0cc5
Merge branch 'master' into daryna/source-bing-ads/stream-budget-and-p…
darynaishchenko Mar 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,16 @@ acceptance_tests:
path: "integration_tests/expected_records.jsonl"
extra_records: yes
empty_streams:
- name: budget
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document why it is empty? I personally prefer to test things in integration anyway but this is unusual given how we do things today

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will update cat config with this info

bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_hourly
bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_daily
bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_weekly
bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_monthly
bypass_reason: "Empty stream, testing in integration tests"
- name: account_performance_report_hourly
bypass_reason: "Hourly reports are disabled, because sync is too long"
- name: ad_group_performance_report_hourly
Expand Down Expand Up @@ -543,6 +553,16 @@ acceptance_tests:
path: "integration_tests/expected_records_no_start_date.jsonl"
extra_records: yes
empty_streams:
- name: budget
bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_hourly
bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_daily
bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_weekly
bypass_reason: "Empty stream, testing in integration tests"
- name: product_dimension_performance_report_monthly
bypass_reason: "Empty stream, testing in integration tests"
- name: app_install_ads
bypass_reason: "Can not populate; new campaign with link to app needed; feature is not available yet"
- name: app_install_ad_labels
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,16 @@
"sync_mode": "full_refresh",
"destination_sync_mode": "append"
},
{
"stream": {
"name": "budget",
"json_schema": {},
"supported_sync_modes": ["full_refresh", "incremental"]
},
"sync_mode": "incremental",
"cursor_field": ["Modified Time"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "campaigns",
Expand Down Expand Up @@ -585,6 +595,46 @@
"sync_mode": "incremental",
"cursor_field": ["TimePeriod"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "product_dimension_performance_report_hourly",
"json_schema": {},
"supported_sync_modes": ["incremental", "full_refresh"]
},
"sync_mode": "incremental",
"cursor_field": ["TimePeriod"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "product_dimension_performance_report_daily",
"json_schema": {},
"supported_sync_modes": ["incremental", "full_refresh"]
},
"sync_mode": "incremental",
"cursor_field": ["TimePeriod"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "product_dimension_performance_report_weekly",
"json_schema": {},
"supported_sync_modes": ["incremental", "full_refresh"]
},
"sync_mode": "incremental",
"cursor_field": ["TimePeriod"],
"destination_sync_mode": "append"
},
{
"stream": {
"name": "product_dimension_performance_report_monthly",
"json_schema": {},
"supported_sync_modes": ["incremental", "full_refresh"]
},
"sync_mode": "incremental",
"cursor_field": ["TimePeriod"],
"destination_sync_mode": "append"
}
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ data:
connectorSubtype: api
connectorType: source
definitionId: 47f25999-dd5e-4636-8c39-e7cea2453331
dockerImageTag: 2.1.4
dockerImageTag: 2.2.0
dockerRepository: airbyte/source-bing-ads
documentationUrl: https://docs.airbyte.com/integrations/sources/bing-ads
githubIssueLabel: source-bing-ads
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -196,3 +196,12 @@ class AdGroupLabels(BingAdsBulkStream):

data_scope = ["EntityData"]
download_entities = ["AdGroupLabels"]


class Budget(BingAdsBulkStream):
"""
https://learn.microsoft.com/en-us/advertising/bulk-service/budget?view=bingads-13&viewFallbackFrom=bingads-13
"""

data_scope = ["EntityData"]
download_entities = ["Budgets"]
Original file line number Diff line number Diff line change
Expand Up @@ -670,6 +670,45 @@ class UserLocationPerformanceReportMonthly(UserLocationPerformanceReport):
report_aggregation = "Monthly"


class ProductDimensionPerformanceReport(BingAdsReportingServicePerformanceStream, ABC):
"""
https://learn.microsoft.com/en-us/advertising/reporting-service/productdimensionperformancereportrequest?view=bingads-13
"""

report_name: str = "ProductDimensionPerformanceReport"
report_schema_name = "product_dimension_performance_report"
primary_key = None

@property
def report_columns(self) -> Iterable[str]:
"""AccountId is not in reporting columns for this report"""
properties = list(self.get_json_schema().get("properties", {}).keys())
properties.remove("AccountId")
return properties

def transform(self, record: MutableMapping[str, Any], stream_slice: Mapping[str, Any], **kwargs) -> MutableMapping[str, Any]:
record = super().transform(record, stream_slice)
record["AccountId"] = stream_slice["account_id"]
return record


class ProductDimensionPerformanceReportHourly(HourlyReportTransformerMixin, ProductDimensionPerformanceReport):
report_aggregation = "Hourly"
report_schema_name = "product_dimension_performance_report_hourly"


class ProductDimensionPerformanceReportDaily(ProductDimensionPerformanceReport):
report_aggregation = "Daily"


class ProductDimensionPerformanceReportWeekly(ProductDimensionPerformanceReport):
report_aggregation = "Weekly"


class ProductDimensionPerformanceReportMonthly(ProductDimensionPerformanceReport):
report_aggregation = "Monthly"


class CustomReport(BingAdsReportingServicePerformanceStream, ABC):
transformer: TypeTransformer = TypeTransformer(TransformConfig.DefaultSchemaNormalization)
custom_report_columns = []
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"$schema": "https://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"Account Id": {
"type": ["null", "integer"]
},
"Type": {
"type": ["null", "string"]
},
"Status": {
"type": ["null", "string"]
},
"Id": {
"type": ["null", "integer"]
},
"Parent Id": {
"type": ["null", "integer"]
},
"Client Id": {
"type": ["null", "integer"]
},
"Modified Time": {
"type": ["null", "string"]
},
"Budget Id": {
"type": ["null", "integer"]
},
"Budget Name": {
"type": ["null", "string"]
},
"Budget": {
"type": ["null", "string"]
},
"Budget Type": {
"type": ["null", "string"]
}
}
}
Loading
Loading