WIP: asyncio-based CDK

[clnoll]

Follow-up to the POC that shows how to integrate asyncio into the CDK and Salesforce connector.

This demonstrates how we might split out an asyncio-based CDK module.

High-level overview:

In this implementation, we interact with the CDK from the same entrypoint (although it's worth considering whether to create a new asynico-based entrypoint).

To make use of the asyncio-based concurrency, connectors will inherit from the AsyncAbstractSource and AsyncStream, which provide async versions of the abstractmethods required for the Airbyte commands. This code also includes a SourceDispatcher which can route requests to either synchronous or async sources, for backwards compatibility.

Outside of swapping out the requests library for aiohttp, there is just one other major change to support the async functionality, which is the introduction of a SourceReader. This was required to bridge the entrypoint's sync code with the async code.

The SourceReader is instantiated by the source. When instantiated, it creates a queue, and kicks off a single thread that will do the async work. Instead of emitting AirbyteMessages on this second thread, they're enqueued in the SourceReader's queue, and the async source, using SourceReader as an iterator, reads and emits items from the queue.

This will replace the infrastructure for the thread-based implementation, which I won't go into detail on here, but here's a diagram giving a simplified overview.

TODOs:

• There's a lot of duplicate code between the normal and asyncio-based CDK that can be consolidated.
• Consider propagating the asyncio code to the entrypoint so that we don't require the StreamReader.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)		Jan 22, 2024 7:15pm

[erohmensing]

@clnoll what's the status of this PR? Just wondering as I created a filter for PRs that have the extensibility team as a required reviewer and noticed it.

[clnoll]

@erohmensing this is not going anywhere any time soon. Closing it now.