feat: Upgrade Airflow from 2.8.2 to 2.9.3 (DENG-5880)

[sean-rose]

Description

Some notable changes:

• 2.8.3
  • The SMTP provider is now pre-installed. (#37713)
• 2.8.4
  • n/a
• 2.9.0
  • Official support for Python 3.12 (#38025)
  • Adding ability to automatically set DAG to off after X times it failed sequentially (#36935)
  • Add task log grouping to UI (#38021)
  • Add color formatting for ANSI chars in logs from task executions (#37985)
  • Add button to clear only failed tasks in a dagrun. (#38217)
  • Improve ExternalTaskSensor Async Implementation (#36916)
  • A bunch of changes related to Airflow's datasets feature (which I don't think we use yet?)
• 2.9.1
  • Add workaround for datetime-local input in firefox+safari (#39261)
• 2.9.2
  • Fetch served logs also when task attempt is up for retry and no remote logs available (#39496)
• 2.9.3
  • Support for OpenTelemetry Metrics is no longer "Experimental" (#40286)

Related Tickets & Documents

• DENG-5880: Upgrade WTMO to Airflow 2.9

[sean-rose]

There were some additional changes in the official docker-compose.yaml file's environment: section, but I wasn't sure we should incorporate those.

[mikaeld]

LGTM, to be tested in stage/dev. Casual reminder to update the Airflow version in the helm chart values