How to deploy Flyte on Azure?

[davidmirror-ops]

Share here your findings/roadblocks when deploying/using Flyte on Azure.

That would be of great help for the community.

Thanks!

[MrMathias]

flytekit 1.5 does currently not mirror the functionality of:

• flytekit==1.4.2
• flytekitplugins-data-fsspec==1.4.2
• adlfs

Changing to 1.5 and running the template project throws:

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[azd8dwh4w2f648p4kxzd-n0-0] terminated with exit code (1). Reason [Error]. Message: 
zd/n0/data/0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-execute", line 8, in <module>
    sys.exit(execute_task_cmd())
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 476, in execute_task_cmd
    _execute_task(
  File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 354, in _execute_task
    _handle_annotated_task(ctx, _task_def, inputs, output_prefix)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 297, in _handle_annotated_task
    _dispatch_execute(ctx, task_def, inputs, output_prefix)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 160, in _dispatch_execute
    ctx.file_access.put_data(ctx.execution_state.engine_dir, output_prefix, is_multipart=True)
  File "/opt/venv/lib/python3.8/site-packages/flytekit/core/data_persistence.py", line 476, in put_data
    raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/flyte-972ilm0g/sandbox/local_flytekit/engine_dir to abfs://flyte-workflows/metadata/propeller/test-project-development-azd8dwh4w2f648p4kxzd/n0/data/0 (recursive=True).

Original exception: No plugin found for matching protocol of path abfs://flyte-workflows/metadata/propeller/test-project-development-azd8dwh4w2f648p4kxzd/n0/data/0
.

[MrMathias]

I am unsure if this is because of a change in config or if something broke towards abfs in 1.5

[MrMathias]

Getting started on Azure

As we rely on a custom stow storage and a specific plugin for fsspc to support azure blob, you should probably review these tools. One pointer for flytekit+fsspc is that the protocol (e.g., abfs//:) is used to switch to these data layer plugins.

Quirks to be aware of:

• The chart values to configure azure postgres and azure blob (for both meta data and raw data)
• The worker containers (currently) needs flytekit==1.4.2, flytekitplugins-data-fsspec==1.4.2, adlfs to communicate with azure blob

values-aks.yaml

storage:
  type: custom
  enableMultiContainer: true
  limits:
    maxDownloadMBs: 500000
    type: custom
  bucketName: "{{ .Values.userSettings.azure.containerName }}"
  custom:
    container: "{{ .Values.userSettings.azure.containerName }}"
    enable-multicontainer: true
    connection: {}
    type: stow
    stow:
      kind: azure
      config:
        account: "{{ .Values.userSettings.azure.storageAccountName }}"
        key: "{{ .Values.userSettings.azure.storageAccountKey }}"

configmap:
  core:
    propeller:
      # abfs protocol is required for Azure Blob Storage
      rawoutput-prefix: "abfs://{{ .Values.userSettings.bucketName }}/"

[MrMathias]

I would like this to be aimed at latest functional/stable release of flyte/flytekit for Azure.
I will expand this as far as I can, but I do not currently posses the full picture, so please do share all your insights, points and tweaks.

[davidmirror-ops]

Hey @MrMathias have you tried using Azure blob storage with flytekit 1.5?
It should work seamlessly without any additional plugin
https://github.com/flyteorg/flyte/releases/tag/v1.5.0

[davidmirror-ops]

Not only supported but now automated: Flyte on Azure is here :)
https://github.com/unionai-oss/deploy-flyte/tree/main/environments/azure/flyte-core