🎉 Source Recharge: migrate to Low-Code

[bazarnov]

What

Resolving: https://github.com/airbytehq/airbyte-internal-issues/issues/2917

How

• by using the airbyte-cdk.declarative (Low-code)
• extended schemas with missing properties
• added unit_tests for python-based stream orders to cover it's implementation
• added basic integration tests to cover the empty streams:
  • collections
  • discounts
  • onetimes
• updated expected records, since there are changes to schemas (more fields than before)

🚨 User Impact 🚨

No breaking changes for this Migration.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 20, 2024 8:41pm

[github-actions]

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

• PR name follows PR naming conventions
• Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
• Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
• You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
• Secrets in the connector's spec are annotated with airbyte_secret
• All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
• Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
• Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
• If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

1. Check for hidden checklists in your PR description

2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

[lazebnyi]

lgtm

[bazarnov]

I was also able to remove the CustomComponents used previously, because now the CDK allows to process the cursor_field values correctly, given the top priority to the actual RECORD cursor value! Nice!

[maxi297]

I had less time than I would have liked to review this again. Here are some comments:

• I could not tests the following streams as there is no test data in our sandbox environment

  • collections
  • discounts
  • onetimes

• I get 3 records for the manifest version instead of 45 in the current one for the stream products. It seems like it is duplication because the API does not consider the updated_at_min and updated_at_max. Can you confirm that? I could now find if I had raised this issue or not in the previous round of review

[bazarnov]

@maxi297 Regarding your comment above:

I could not tests the following streams as there is no test data in our sandbox environment
collections, discounts, sometimes

These are not populated in for the Sandbox, and are covered with Integration Tests.

I get 3 records for the manifest version instead of 45 in the current one for the stream products. It seems like it is duplication because the API does not consider the updated_at_min and updated_at_max. Can you confirm that? I could now find if I had raised this issue or not in the previous round of review

I confirm these are the duplicates. There are 3 records for the product stream, which is a Full-refresh stream.

[maxi297]

Cool! With the confirmation that it is duplication, LGTM!

[bazarnov]

Cool! With the confirmation that it is duplication, LGTM!

This is the proof from the sandbox account:

[lazebnyi]

It is look like breaking change as you change state format

[bazarnov]

The STATE output from the source-recharge 1.1.6 (current master version):

[
    {
        "subscriptions": {
            "updated_at": "2023-05-13T04:07:32+00:00"
        },
        "addresses": {
            "updated_at": "2023-05-13T04:07:34+00:00"
        },
        "charges": {
            "updated_at": "2023-05-13T04:16:51+00:00"
        },
        "collections": {},
        "customers": {
            "updated_at": "2023-05-13T04:16:36+00:00"
        },
        "onetimes": {},
        "discounts": {
            "updated_at": "2021-05-13T12:40:17+00:00"
        },
        "orders": {
            "updated_at": "2023-05-13T04:16:51+00:00"
        },
        "products": {},
        "shop": {},
        "metafields": {},
    }
]

The STATE output from the source-recharge 1.2.0 (low-code migrated version):

[
    {
        "subscriptions": {
            "updated_at": "2023-05-13T04:07:32+00:00"
        },
        "addresses": {
            "updated_at": "2023-05-13T04:07:34+00:00"
        },
        "charges": {
            "updated_at": "2023-05-13T04:16:51+00:00"
        },
        "collections": {},
        "customers": {
            "updated_at": "2023-05-13T04:16:36+00:00"
        },
        "onetimes": {},
        "discounts": {
            "updated_at": "2021-05-13T12:40:17+00:00"
        },
        "orders": {
            "updated_at": "2023-05-13T04:16:51+00:00"
        },
        "products": {},
        "shop": {},
        "metafields": {}
    }
]

I could be wrong, but I don't observe the difference you've found @lazebnyi. Could you please be more specific about where the breaking change takes place?

[lazebnyi]

lgtm

as we have two cursor_datetime_formats