Azure ImdsManagedIdentityProvider does not work in Azure functions

[tahaum]

It seems like ImdsManagedIdentityProvider assumes one is running in a Azure VM, even though comments indicate that it is intended to also work in Azure functions.

We see it when running

from deltalake import DeltaTable

account_name = "..."
delta_path = "az://..."
DeltaTable(delta_path, storage_options={"account_name": account_name})

(using the delta-rs package) in an Azure Function. Then we're getting the following error:

Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.9/site-packages/deltalake/table.py", line 250, in init
self._table = RawDeltaTable(
OSError: Generic MicrosoftAzure error: Error performing token request: response error "request error", after 10 retries: error sending request for url (http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com): error trying to connect: tcp connect error: Connection refused (os error 111)

It looks like the endpoint set here does not work for Azure Functions, but Azure VMs (and other resource types?):

arrow-rs/object_store/src/azure/credential.rs

Line 404 in 03d0505

.unwrap_or_else(|| "http://169.254.169.254/metadata/identity/oauth2/token".to_owned());

If we set the msi_endpoint in the storage_options explicitly using the IDENTITY_ENDPOINT environment variable found in the Azure Function we instead get the following error:

OSError: Generic MicrosoftAzure error: Error getting token response body: error decoding response body: missing field expires_in at line 1 column 1641

Our hypothesis is that this is caused by https://github.com/apache/arrow-rs/blame/03d0505fc864c09e6dcd208d3cdddeecefb90345/object_store/src/azure/credential.rs#L463 and the assumption that the token responses looks like this:

{
    "access_token": "TOKEN",
    "refresh_token": "",
    "expires_in": "3599",
    "expires_on": "1506484173",
    "not_before": "1506480273",
    "resource": "https://management.azure.com/",
    "token_type": "Bearer"
  }

In reality, the responses we get from the $IDENTITY_ENDPOINT looks like this (notice the absence of expires_in):

{
  "access_token": "TOKEN",
  "expires_on": "1698150006",
  "resource": "https://storage.azure.com",
  "token_type": "Bearer",
  "client_id": "..."
}

MSI in azure functions documentation: https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=portal%2Chttp#connect-to-azure-services-in-app-code
Related: delta-io/delta-rs#662
Maintainer (?) of relevant code: @roeap

[tustvold]

Your diagnosis would appear to be correct, thank you for the detailed report, I'll try to get a fix for you to try out

[tustvold]

label_issue.py automatically added labels {'object-store'} from #4977

[mortnstak]

Hi - still get the same error message when trying to run on an Azure function. Am running delta-rs version 0.15.3.

Result: Failure Exception: OSError: Generic MicrosoftAzure error: Error performing token request: Error after 10 retries in 12.738979181s, max_retries:10, retry_timeout:180s, source:error sending request for url (http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com): error trying to connect: tcp connect error: deadline has elapsed

[tustvold]

error trying to connect: tcp connect error: deadline has elapsed

That looks like a different error and indicates that the workload is unable to communicate with the metadata endpoint.

Have you followed the instructions here to configure an identity for your function?

[mortnstak]

Hi @tustvold - thanks for responding. I have indeed set up an identity for the function. And am able to read the parquet files of the dataset using pyarrow, pyarrowfs-adlgen2, and ManagedIdentityCredential.

[tahaum]

I think we might have encountered the same problem @mortnstak, and this works for us:

storage_options = {"account_name": account, "azure_msi_endpoint" : os.getenv("IDENTITY_ENDPOINT")}

[mortnstak]

Thanks a lot @tahaum - setting the azure_msi_endpoint resolved the issue.

[genegc]

We ran into the same issue when running in an azure app container. We needed to use:
storage_options = {"azure_msi_endpoint" : os.getenv("IDENTITY_ENDPOINT")}

[danielsandorsyntio]

Hello, I have ran into the same issue with the 0.17.3 version of the delta-rs library in Python.

I enabled the system assigned managed identity on my Azure Function. I assigned it the Contributor role for the storage account. I defined storage options as such:

storage_options = {
    "account_name": "labssupportdeltadestdev",
    "azure_msi_endpoint": getenv("IDENTITY_ENDPOINT"),
}

I get the following error:

Generic MicrosoftAzure error: Client error with status 403 Forbidden: 
<?xml version="1.0" encoding="utf-8"?>
<Error>
	<Code>AuthenticationFailed</Code>
	<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:5555555-b01e-0000-2e65-55555000
Time:2024-05-16T07:46:01.5423665Z
    </Message>
	<AuthenticationErrorDetail>The MAC signature found in the HTTP request 'lGJvodsICn0cdsdIdAuNss4dY33AZD9whZJ3Btd21x7A=' is not the same as any computed signature.
    </AuthenticationErrorDetail>
</Error>

I tried setting azure_object_id and azure_msi_resource_id as well using my managed identity object ID and the Azure resource ID as well, but the error persists.

Any advice on what I'm doing wrong?

[mortnstak]

@danielsandorsyntio this appears to be a general access issue. Ensure that the function app is able to read/write on the storage account without the delta-rs library.