There are two main cases (as of now) where we would like to allow Pieces running directly in-workers (and not on Docker):

• To be able to incorporate most of currently existing official Apache Airflow Operators as Pieces.
• To write simpler Pieces, with no or very few basic dependencies, that could be executed as PythonOperators

What we want:

• Pieces written as distributed as similarly as possible to the current standard: in repositories, with metadata.json, models.py and piece.py.
• Seamless usage of these in any workflow. No major change should be require in the frontend.
• These Pieces should be installed and tracked in Domino's DB just like in-docker Pieces. Perhaps we could have the differentiation as a new column in the Piece table.
• Consider maybe having such repositories also as default in new workspaces.

Challenges:

• Since these Pieces won’t be shipped in Docker images, they should be imported from the repositories releases. We should make the Pieces type explicit somewhere, as either in-worker or in-docker, perhaps in the repository's config.toml.
• most of currently existing official Apache Airflow Operators require pre-defined Connections. I don’t think we need to keep track of these in Domino’s DB, since they are already tracked in Airflow’s DB, but we need a smart way to name them and control access to them per workspace, similar to what we do with Secrets.
• I don't think we'd like to ask users to define the Airflow Connections. Instead, for in-worker Pieces that require a Connection instead of pure Secrets, we can add the arguments of the respective Airflow Connection as being the Piece's Secrets (to keep our standard) and we manage checking and creating the connections on our side. We can get, edit, create and delete Connections using the Airflow rest api
• Security: we might want to either keep a closed list of allowed in-worker Pieces, or to explicitly warn users whenever they install repositories which contain in-worker Pieces.