Skip to content

Commit

Permalink
Revert "Load default Airflow connections on initialization (#182)" (#195
Browse files Browse the repository at this point in the history
)

This reverts commit f71d082.

*Issue #, if available:*
N/A

*Description of changes:*
Airflow startup is failing with:

```
mwaa-2101-scheduler  | [2024-11-27T19:46:26.107+0000] {cmd.py:85} INFO - File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/connection.py", line 379, in set_extra
mwaa-2101-scheduler  | [2024-11-27T19:46:26.107+0000] {cmd.py:85} INFO - fernet = get_fernet()
mwaa-2101-scheduler  | [2024-11-27T19:46:26.108+0000] {cmd.py:85} INFO - ^^^^^^^^^^^^
mwaa-2101-scheduler  | [2024-11-27T19:46:26.108+0000] {cmd.py:85} INFO - File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/crypto.py", line 90, in get_fernet
mwaa-2101-scheduler  | [2024-11-27T19:46:26.108+0000] {cmd.py:85} INFO - raise AirflowException(f"Could not create Fernet object: {value_error}")
mwaa-2101-scheduler  | [2024-11-27T19:46:26.108+0000] {cmd.py:85} INFO - airflow.exceptions.AirflowException: Could not create Fernet object: Fernet key must be 32 url-safe base64-encoded bytes.
```

If an airflow connection defines the `extra` field, it goes through an
optional encoding/decoding step if a fernet key is defined
[[reference](https://github.com/apache/airflow/blob/main/airflow/models/connection.py#L335)].
With #182, we are loading default provider connects, of which several
define `extra`'s. This presents an issue when running the MWAA images
locally, as we define a dummy fernet key in the docker compose
[[reference](https://github.com/aws/amazon-mwaa-docker-images/blob/f71d082cac983385349f3e30b8a650d5f5c5d619/images/airflow/2.9.2/docker-compose.yaml#L23)]
which is invalid as shown by the stack trace above.

Reverting default connections loading until we implement a check for
local image vs. mwaa environment.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
  • Loading branch information
mayushko26 authored Nov 27, 2024
1 parent f71d082 commit 7402454
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 23 deletions.
13 changes: 1 addition & 12 deletions images/airflow/2.10.1/python/mwaa/database/migrate.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,21 +55,10 @@ def _migrate_db():
logging.info("The database is now migrated.")


@with_db_lock(1234)
def _create_default_connections():
from airflow.cli.commands import connection_command as airflow_connection_command

try:
args = Namespace(verbose=False)
airflow_connection_command.create_default_connections(args)
logging.info("The default Airflow connections have been created.")
except TimeoutError:
logging.warning("Timed out on initializatin of default Airflow connections.")

def _main():
_verify_environ()
_migrate_db()
_create_default_connections()


if __name__ == "__main__":
_main()
Expand Down
12 changes: 1 addition & 11 deletions images/airflow/2.9.2/python/mwaa/database/migrate.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,21 +54,11 @@ def _migrate_db():
airflow_db_command.migratedb(args)
logging.info("The database is now migrated.")

@with_db_lock(1234)
def _create_default_connections():
from airflow.cli.commands import connection_command as airflow_connection_command

try:
args = Namespace(verbose=False)
airflow_connection_command.create_default_connections(args)
logging.info("The default Airflow connections have been created.")
except TimeoutError:
logging.warning("Timed out on initializatin of default Airflow connections.")

def _main():
_verify_environ()
_migrate_db()
_create_default_connections()


if __name__ == "__main__":
_main()
Expand Down

0 comments on commit 7402454

Please sign in to comment.