Airbyte Multi-tenancy

[jscheel]

Hi there, I’m trying to understand how to use dagster with airbyte’s “powered by” multi-tenant offering (https://reference.airbyte.com/reference/powered-by-airbyte). This allows you to create workspaces with sources, destinations, and connections for each user in your system.

I have a pipeline in dagster that takes each account's data and builds a report from it (does more than this, but just for the sake of simplicity, sticking with this example). Currently, I am using partitions for each account and just processing a file that the user uploads. Now I am trying to use airflow instead of manually-uploaded files.

All of the airbyte examples in dagster assume one workspace that you create a resource for and that you build assets from, and that the configuration for your connection is essentially static. With a static set of connections, this is easy to set up. The dagster airbyte module has a function that loads each of the defined airbyte connections as asset specs. But in my case, the workspace and connection is set up when the account sets it up, not when I configure it in code and reload my code location. I'm not really certain how I should approach this dynamic nature of multi-tenancy.