✨ Source Facebook-Marketing: Add Selectable Auth (with migration & tests) #38304

cmm-airbyte · 2024-05-16T21:35:52Z

What

The Source Facebook Marketing Connector does not allow for choosing between providing an access token or using oauth, it only allows for users to define a source connection by following an oauth flow. This does not allow for use-cases in which a large organization may provision an access token to be used to authenticate to the Facebook Marketing data, rather than giving a user's personal Facebook account access.

How

The spec defines a credentials field that is a union between newly created OAuthCredentials and ServiceAccountCredentials. This allows for the user to select between using the Oauth flow and inserting Service Account Credentials.

Review guide

spec.py
config.py
source.py
sample_config.json
metadata.yaml
pyproject.toml
request_builder.py
test_ads_insights_action_product_id.py
test_videos.py
test_source.py

User Impact

The user is able to select between using the Oauth flow and inserting Service Account Credentials when setting up the Facebook Marketing Source.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

Source PR here: #37625

…anges to solve build issues, and changes to solve unit testing issues.

…h 11 warnings

…e config. Updated version to reflect Airbyte versioning standard (this use case is a major version upgrade due to configuration of credentials needed to be changed)

…ebook-marketing-selectable-auth

…ebook-marketing-selectable-auth' into JohnMikkelsonOvative/feature-facebook-marketing-selectable-auth

…eting-selectable-auth

vercel · 2024-05-16T21:35:56Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 6, 2024 5:06pm

CLAassistant · 2024-05-16T21:35:59Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 5 committers have signed the CLA.

✅ JohnMikkelsonOvative
✅ marcosmarxm
✅ bazarnov
❌ cristina.mariscal
❌ cmm-airbyte

cristina.mariscal seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

…irbyte into cmm-airbyte/fb_oauth

bazarnov · 2024-05-21T15:38:23Z

@cmm-airbyte We need to update the config in the GSM for the change to work, I believe.

maxi297 · 2024-05-21T18:45:02Z

We need to update the config in the GSM for the change to work, I believe.

I haven't check the change but it shouldn't be if the change is non-breaking with the config migration

maxi297

I'll test manually tomorrow morning but I don't see obvious issues with the code

maxi297 · 2024-05-21T18:45:43Z

airbyte-integrations/connectors/source-facebook-marketing/metadata.yaml

@@ -44,6 +44,9 @@ data:
              - "ads_insights_region"
              - "ads_insights_dma"
              - "ads_insights_action_product_id"
+      3.0.0:


What makes it breaking if we have the config migration? Does that mean that the user will have to opt-in on that change?

no, I was just guessing this because the PR cannot be directly reverted. I can do only a 2.2.0 version increment. WDYT?

I think 2.2.0 makes sense, yes! We should still document that the config format changes but there is a migration for backward compatibility but none for forward compatiblity

great. Thanks Max!

maxi297 · 2024-05-21T18:47:06Z

...grations/connectors/source-facebook-marketing/source_facebook_marketing/config_migrations.py

+        """
+        credentials = config.get("credentials", None)
+        return credentials is None or (
+            "access_token" not in credentials and not ("client_id" in credentials and "client_id" in credentials)


This condition feels hard to maintain. If something adds a new auth method, they will need to know that they have to update an old migration script

mmm.. I think if there is a new change, an iteration over this migration will be required. isn't it?

Let's take the following scenario: on the 2025-01-04, a developer is doing the work to add username/password as an authentication mechanism. To do so, they update the spec to have credentials.username and credentials.password and configure the authenticator properly in the source.py. Now, this condition will always be triggered which means that the GSM secrets and the database entries will be updated on every operation of the connector. For the source in practice, I don't think it changes that much but I fear that those updates on the platform or GSM side could create issues in the long run. Hence, I would try to make the condition specific to this update. Is there a case where only credentials not being present wouldn't be enough to trigger this migration?

Note if we keep the condition as-is: the "client_id" in credentials and "client_id" in credentials part of the condition should probably be "client_id" in credentials and "client_secret" in credentials

maxi297 · 2024-05-21T18:50:51Z

...grations/connectors/source-facebook-marketing/source_facebook_marketing/config_migrations.py

+    Starting from `3.0.0`, the `client_id`, `client_secret` and `access_token` will be placed at `credentials` path.
+    """
+
+    message_repository: MessageRepository = InMemoryMessageRepository()


This field is useless right now. We can just do print(create_connector_config_control_message(migrated_config).json(exclude_unset=True))

maxi297 · 2024-05-21T18:56:49Z

...grations/connectors/source-facebook-marketing/source_facebook_marketing/config_migrations.py

+    message_repository: MessageRepository = InMemoryMessageRepository()
+
+    @classmethod
+    def should_migrate(cls, config: Mapping[str, Any]) -> bool:


It feels like this should be private as it is only used internally. What do you think? The same comment applies to transform, modify_and_save and emit_control_message

…m-airbyte/fb_oauth

maxi297

LGTM! Before I approve, I think we had a list of manual tests to do by fixing the version in prod. I don't remember where this list is but can you explicit the tests you have done in the PR description so that we know if there is an issue what we know worked when we released? Once this is shared, I'll approve this change

…m-airbyte/fb_oauth

maxi297

LGTM! Thanks for the diligent testing and the throughouness with which you tackle this issue

JohnMikkelsonOvative and others added 9 commits April 24, 2024 16:21

EMRGE-1446 Selectable Auth working on local CLI. Still need to add ch…

eccc2a1

…anges to solve build issues, and changes to solve unit testing issues.

EMRGE-1446 Unit Tests for Selectable Auth all passing, 205 passed wit…

cda978e

…h 11 warnings

EMRGE-1446 Changed naming to reflect Facebook standard, updated sampl…

ccd385a

…e config. Updated version to reflect Airbyte versioning standard (this use case is a major version upgrade due to configuration of credentials needed to be changed)

Merge branch 'airbytehq:master' into JohnMikkelsonOvative/feature-fac…

4079845

…ebook-marketing-selectable-auth

Merge remote-tracking branch 'origin/JohnMikkelsonOvative/feature-fac…

378f275

…ebook-marketing-selectable-auth' into JohnMikkelsonOvative/feature-facebook-marketing-selectable-auth

Merge branch 'master' into JohnMikkelsonOvative/feature-facebook-mark…

a5f4017

…eting-selectable-auth

Merge branch 'master' into JohnMikkelsonOvative/feature-facebook-mark…

7939c8a

…eting-selectable-auth

merged from JohnMikkelsonOvative

b60755a

FB OAuth switch

dba3748

Merge branch 'master' into cmm-airbyte/fb_oauth

f973e69

octavia-squidington-iii added area/connectors Connector related issues connectors/source/facebook-marketing labels May 16, 2024

vercel bot deployed to Preview May 16, 2024 21:40 View deployment

cristina.mariscal added 4 commits May 18, 2024 14:44

fixing unit test

040e234

pull from master

c55e538

Merge branch 'cmm-airbyte/fb_oauth' of https://github.com/airbytehq/a…

ec1ef32

…irbyte into cmm-airbyte/fb_oauth

fixing formatting

7aa761e

cmm-airbyte changed the title ~~FB oauth origin~~ ✨ Source Facebook-Marketing: Add Selectable Auth (with migration & tests) May 21, 2024

cmm-airbyte marked this pull request as ready for review May 21, 2024 15:01

cmm-airbyte requested a review from a team May 21, 2024 15:01

modifying specs to test pass

e5f8341

cmm-airbyte requested a review from a team May 21, 2024 18:43

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label May 21, 2024

octavia-squidington-iv requested a review from a team May 21, 2024 18:45

vercel bot deployed to Preview May 21, 2024 18:48 View deployment

maxi297 reviewed May 21, 2024

View reviewed changes

cristina.mariscal added 2 commits June 3, 2024 03:22

removing auth type from root

c6b0fed

Merge branch 'master' of https://github.com/airbytehq/airbyte into cm…

5df66a9

…m-airbyte/fb_oauth

cmm-airbyte requested a review from maxi297 June 3, 2024 10:30

cristina.mariscal added 2 commits June 3, 2024 06:36

rm spec.json

7e9451f

merge from master

c3addd1

vercel bot deployed to Preview June 4, 2024 09:41 View deployment

version bump

eb2498b

maxi297 reviewed Jun 4, 2024

View reviewed changes

cristina.mariscal and others added 4 commits June 4, 2024 13:05

Merge branch 'master' of https://github.com/airbytehq/airbyte into cm…

2a03f59

…m-airbyte/fb_oauth

adding back spec

7021679

fix format

8fdca53

Merge branch 'master' into cmm-airbyte/fb_oauth

c476c0a

vercel bot deployed to Preview June 4, 2024 19:57 View deployment

bazarnov mentioned this pull request Jun 5, 2024

source-facebook-marketing: [autopull] Upgrade base image to v1.2.1 #39022

Closed

vercel bot deployed to Preview June 6, 2024 00:40 View deployment

cristina.mariscal added 2 commits June 5, 2024 20:05

Merge branch 'master' of https://github.com/airbytehq/airbyte into cm…

eb8b13f

…m-airbyte/fb_oauth

merge from master and remove auth_type

543d5ea

cmm-airbyte force-pushed the cmm-airbyte/fb_oauth branch from 3a6f33b to 543d5ea Compare June 6, 2024 02:06

bazarnov added 2 commits June 6, 2024 13:28

Merge remote-tracking branch 'origin/master' into cmm-airbyte/fb_oauth

709eb8d

fixed failed CAT test

ce31920

bazarnov approved these changes Jun 6, 2024

View reviewed changes

maxi297 approved these changes Jun 6, 2024

View reviewed changes

Merge branch 'master' into cmm-airbyte/fb_oauth

23e8a73

cmm-airbyte enabled auto-merge (squash) June 6, 2024 16:46

vercel bot deployed to Preview June 6, 2024 16:47 View deployment

Merge branch 'master' into cmm-airbyte/fb_oauth

1aa151d

cmm-airbyte disabled auto-merge June 6, 2024 17:02

vercel bot deployed to Preview June 6, 2024 17:06 View deployment

cmm-airbyte merged commit e3ea7cd into master Jun 6, 2024
28 of 30 checks passed

cmm-airbyte deleted the cmm-airbyte/fb_oauth branch June 6, 2024 20:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Source Facebook-Marketing: Add Selectable Auth (with migration & tests) #38304

✨ Source Facebook-Marketing: Add Selectable Auth (with migration & tests) #38304

cmm-airbyte commented May 16, 2024 •

edited

Loading

vercel bot commented May 16, 2024 •

edited

Loading

CLAassistant commented May 16, 2024 •

edited

Loading

bazarnov commented May 21, 2024

maxi297 commented May 21, 2024

maxi297 left a comment

maxi297 May 21, 2024

cmm-airbyte May 22, 2024

maxi297 May 22, 2024

cmm-airbyte May 22, 2024

maxi297 May 21, 2024

cmm-airbyte May 22, 2024

maxi297 May 22, 2024

maxi297 May 21, 2024

maxi297 May 21, 2024

maxi297 left a comment

maxi297 left a comment

✨ Source Facebook-Marketing: Add Selectable Auth (with migration & tests) #38304

✨ Source Facebook-Marketing: Add Selectable Auth (with migration & tests) #38304

Conversation

cmm-airbyte commented May 16, 2024 • edited Loading

What

How

Review guide

User Impact

Can this PR be safely reverted and rolled back?

vercel bot commented May 16, 2024 • edited Loading

CLAassistant commented May 16, 2024 • edited Loading

bazarnov commented May 21, 2024

maxi297 commented May 21, 2024

maxi297 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maxi297 left a comment

Choose a reason for hiding this comment

maxi297 left a comment

Choose a reason for hiding this comment

cmm-airbyte commented May 16, 2024 •

edited

Loading

vercel bot commented May 16, 2024 •

edited

Loading

CLAassistant commented May 16, 2024 •

edited

Loading