[BUG] FlyteDirectory fails for Azure

[Tom-Newton]

Describe the bug

Imagine:

@workflow
def test_workflow(directory: FlyteDirectory):
    ...

When using Azure, launching this workflow with a directory containing more than one file causes

Invalid value for '--directory': Failed to convert param: <Option directory>, value: /path/on/local/ to type: <class 'flytekit.types.directory.types.FlyteDirectory'>. Reason USER:AssertionError: error=Failed to put data from      │
│ /path/on/local/ to flyte://data/10ba872c9015435996c14abeeb15f846 (recursive=True).                                                                                                                                                    │
│                                                                                                                                                                                                                                                                              │
│ Original exception: unable to connect to account for Must provide either a connection_string or account_name with credentials!!, cause=unable to connect to account for Must provide either a connection_string or account_name with credentials!!

Expected behavior

I would expect this to run without error.

Root cause

The files are actually uploaded fine using FlyteFS. The problem is when it calls FlyteFS.extract_common() which comes after successful upload. This contains the snippet

        fs = fsspec.filesystem(get_protocol(native_urls[0]))
        sep = fs.sep

The problem is that adlfs.AzureBlobFileSystem() fails because the account_name is unknown. I'm thinking it would be best to avoid needing to initialise an fsspec filesystem just to determine the separator used on the filesystem in question.

Additional context to reproduce

No response

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[Tom-Newton]

I think we can straightforwardly switch from fsspec.filesystem to fsspec.get_filesystem_class. This avoids initialising the filesystem object, which is unnecessary because sep is a class attribute. I will make a PR.

[Tom-Newton]

Made a PR that fixes this flyteorg/flytekit#2564

[runllm]

The issue you're encountering is related to Flyte's handling of FlyteDirectory with Azure Blob Storage. The error arises when fsspec attempts to initialize an AzureBlobFileSystem without the necessary credentials, specifically during the call to FlyteFS.extract_common().

To work around this issue, you can ensure that the necessary Azure credentials are provided. One way to do this is by setting the AZURE_STORAGE_CONNECTION_STRING environment variable in your pod template.

Here is an example of how you can set this environment variable in your pod template:

env:
  - name: AZURE_STORAGE_CONNECTION_STRING
    valueFrom:
      secretKeyRef:
        name: flyteruntimesecret

Would you like more details on setting up the environment or addressing this issue in another way?

Sources:

• Azure Workload Identity not working with fsspec in flytekit
• Hi folks, I am attempting to set up Azure Blob through Stow
• Hello Back with more Azure adventures

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[Tom-Newton]

The bot hasn't fully understood. The problem at least for me is only when launching an execution so setting the environment variable in the pod template doesn't help. It also doesn't understand that its unnecessary to initialise the filesystem object.

[thomasjpfan]

Closing as this was resolved in flyteorg/flytekit#2564