[airbyte-cdk] Promote low-code types and cursor interface into Python CDK

[brianjlai]

What

As I was trying to implement RFR for low-code, a consistent theme was that I was writing boilerplate and making some questionable decisions on how to format and read state from checkpoint_reader.py. The reality is that a number of these concepts like StreamSlice that has separate cursor/partition value and the Cursor interface belong in the top most CDK and using them here makes a huge difference in the understandability of the RFR flow.

I decided to just yolo this thing out cuz why not. Looks like a lot of changes, but assuming we can get CATs and unit tests to pass, I have pretty high confidence because if I refactored the references wrong it would fail very loudly.

I also don't plan to republish connectors

How

Moved the types.py from /sources/declarative to /sources

• One thing to note here is I kept the existing types.py. I migrated all our repos over, but we might have customer's who've implemented custom low-code connectors w/ custom components and this avoids breaking them

Moved the Cursor to /sources/streams/checkpoint

• The promoted Cursor interface no longer inherits from StreamSlicer which is a low-code concept
• We have a new DeclarativeCursor which implements Cursor + StreamSlicer which actually makes more sense because a generic cursor as a concept doesn't actually need to be responsible for injecting values

IntelliJ/copilot magic to refactor everything to reference the appropriate types.

Review guide

1. /sources/types.py
2. /sources/streams/checkpoint/cursor.py

User Impact

Should be non-breaking as mentioned above.

Can this PR be safely reverted and rolled back?

• YES 💚
• NO ❌

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		May 10, 2024 0:25am

[girarda]

<3

[assaadhjb]

Hello @brianjlai @girarda, If I understood correctly, DeclarativeCursor would allow us to leverage stream slicers for example for incremental sync where no cursor field can be selected?
Any chance I can find documentation on how to use this DeclarativeCursor if it is the case, because I am interested in its usage! Thanks 🙏

[brianjlai]

Hi @assaadhjb, assuming that your connector is written in the low-code framework. You don't actually need to define the DeclarativeCursor within the manifest file itself, but rather by defining certain attributes of a stream like a paginator and that it hasn't defined an incrementalfield, then Resumable Full Refresh will be applied during a sync automatically. If you want to read more about this you refer to our blog post on it's usage: https://airbyte.com/blog/resumable-full-refresh-building-resilient-systems-for-syncing-data#leveraging-the-power-of-the-low-code-framework

If your custom connector is written using the Python CDK, then we do have some docs here that talk about synthetic cursors.