Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider utilizing GKE Operators in anticipating of shifting to Composer v2 #186

Open
leahecole opened this issue Sep 16, 2021 · 1 comment
Labels
feature request New feature or request

Comments

@leahecole
Copy link
Contributor

Composer v2 was released this week, and it uses GKE Autopilot under the hood. With GKE Autopilot, we can no longer use node pool affinity with the KubernetesPodOperator. In anticipation of eventually migrating to Composer v2 we can utilize the GKEStartPodOperator, a child of the KubernetesPodOperator in one of two ways:

  1. Create a long lasting GKE Cluster with node pools that is managed outside of Airflow, use the GKEStartPodOperator with node pool affinity as needed.
  2. Programmatically create an ephemeral GKE Cluster with node pools using the GKECreateClusterOperator, use the GKEStartPodOperator to launch pods with node pool affinity, and tear down the cluster with the GKEDeleteClusterOperator
@leahecole leahecole added the feature request New feature or request label Sep 16, 2021
@leahecole
Copy link
Contributor Author

This was brought up as a discussion regarding #180

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant