Airflow Azure Remote Logging Configuration

[valayDave]

Hello!,

I am using the helm chart to deploy the Airflow on kubernetes. I wanted to store the logs to Azure blob store and I modified the helm chart values to try and achieve this. Some of my helm values look like the following :

env:
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: 'wasb-storage-folder' 
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "azure_default"
  - name: AIRFLOW_CONN_AZURE_DEFAULT
    value: '{"login":"myaccountname"}'

secret:
  - envName: AZURE_CLIENT_ID
    secretName: azure-storage-credentials
    secretKey: AZURE_CLIENT_ID
  - envName: AZURE_CLIENT_SECRET
    secretName: azure-storage-credentials
    secretKey: AZURE_CLIENT_SECRET
  - envName: AZURE_TENANT_ID
    secretName: azure-storage-credentials
    secretKey: AZURE_TENANT_ID

The chart deploys successfully but since the container-name is hardcoded, the authentication fails and logs are not stored.

I am not able to dynamically provide the container name via some config variables or even via AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER env variable (based on the documentation ). Is it possible to do this via passing some environment variable so that I can deploy with helm without altering files given to the scheduler and other components (As stated in the docs) ?

[potiuk]

You can configure local settings as you wish. It is described in a few places for example here that you can put local settings in your config folder or PYTHONPATH and configure it as you wish. https://airflow.apache.org/docs/apache-airflow/stable/concepts/cluster-policies.html?highlight=cluster_policies

Obviously this is an answer that should likely be posted somewhere else 2.5 years ago as helpfully pointed out by @ijaniszewski - let me cross it out to not confuse people looking for help here.

[ijaniszewski]

@potiuk if you spend your time to read such post and answer, you could at least help

[potiuk]

Thanks for bringing my attention to something that was posted (and mistakenly wrongly answered - apparently that is an answer to completely different question that was added on a wrong question) 2.5 years ago. It was 2.5 years ago, so I have completely no memory of it.

Let me cross it out, but unfortunately I do not know the answer to your question. Maybe someone else will, if you have similar problem - I suggest you to open a new discussion for that, rather than look at a such an old discsusion @ijaniszewski .

Make sure to spend time on writing good description though, it then make it easier for people who try to help people who have problem mostly int their free time.

[ijaniszewski]

You can add connection ID to values.yaml, havent tried with secrets. Then, values file should include:

config:
  logging:
    remote_logging: "True"
    # WASB buckets should start with wasb just to help Airflow select correct handler
    # https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#remote-base-log-folder
    remote_base_log_folder: "wasb_staging" # folder on container, if not exists will be created. HAVE TO START WITH `wasb`
    remote_log_conn_id: "conn_id" # connection ID

  azure_remote_logging:
    remote_wasb_log_container: "airflow-logs" # azure storage container

or do it via env:

env:
  - name: AIRFLOW__LOGGING__REMOTE_LOGGING
    value: "True"
  - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
    value: 'wasb_staging' 
  - name: AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID
    value: "conn_id"
  - name: AIRFLOW__AZURE_REMOTE_LOGGING__REMOTE_WASB_LOG_CONTAINER
    value:  "airflow-logs"