Airflow

Addons to the training :

Deferring

https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html

Scalability

Best practices / Worst practices

https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html

Fine tuning

Environment Variables

Dynamic Task Mapping

https://airflow.apache.org/docs/apache-airflow/2.3.0/concepts/dynamic-task-mapping.html#

Scalability

Scheduling Interval

https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html

Computing Optimisation

Astronomer

https://registry.astronomer.io/

Docker

https://github.com/puckel/docker-airflow