Managed Service Identity does not work from Azure Container Apps

[jirikopecky]

What happened?

We have an application that manages Azure resources using Pulumi Automation API. This application is written in .NET and deployed as Azure Container App with Pulumi CLI installed in the container. We wanted to migrate away from using Service Principal to use MSI for authentication agains Azure. This was attempted by setting ARM_USE_MSI to true in the container environment (and of course creating user-assigned MSI with relevant permissions assigned and associating it with the container app).

While attempting to run Up on the stack we received following error:

Provider[azure-native, 0xc000ba4690].Configure() failed: err=please ensure you have installed Azure CLI version 2.0.79 or newer. Error parsing json result from the Azure CLI: launching Azure CLI: exec: "az": executable file not found in $PATH.

Expected Behavior

Pulumi uses configured managed identity to manage Azure resources

Steps to reproduce

1. Create app that uses Pulumi Automation API to manage Azure resources with azure-native (we used version 1.100.1)
2. Create container image with this app and Pulumi CLI installed
3. Deploy to Azure as Container App with MSI configured and ARM_USE_MSI environment variable set to true
4. Invoke the app and see it fail

Output of pulumi about

CLI
Version      3.64.0
Go Version   go1.20.3
Go Compiler  gc

Host
OS       debian
Version  11.6
Arch     x86_64

Pulumi locates its logs in /tmp by default
warning: Failed to read project: no Pulumi.yaml project file found (searching upwards from /). If you have not created a project yet, use `pulumi new` to do so: no project file found

There is no project since the app uses Automation API to create workspaces on-the-fly.

Additional context

I was able to capture following log from the provider:

I0421 09:14:33.372403     304 log.go:75] 2023/04/21 09:14:33 Testing if Service Principal / Client Certificate is applicable for Authentication..
I0421 09:14:33.372409     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Testing if Service Principal / Client Certificate is applicable for Authentication..<{%reset%}>)
I0421 09:14:33.372620     304 log.go:75] 2023/04/21 09:14:33 Testing if Multi Tenant Service Principal / Client Secret is applicable for Authentication..
I0421 09:14:33.372626     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Testing if Multi Tenant Service Principal / Client Secret is applicable for Authentication..<{%reset%}>)
I0421 09:14:33.372867     304 log.go:75] 2023/04/21 09:14:33 Testing if Service Principal / Client Secret is applicable for Authentication..
I0421 09:14:33.372875     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Testing if Service Principal / Client Secret is applicable for Authentication..<{%reset%}>)
I0421 09:14:33.373091     304 log.go:75] 2023/04/21 09:14:33 Testing if OIDC is applicable for Authentication..
I0421 09:14:33.373094     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Testing if OIDC is applicable for Authentication..<{%reset%}>)
I0421 09:14:33.373352     304 log.go:75] 2023/04/21 09:14:33 Testing if Managed Service Identity is applicable for Authentication..
I0421 09:14:33.373358     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Testing if Managed Service Identity is applicable for Authentication..<{%reset%}>)
I0421 09:14:33.373544     304 log.go:75] 2023/04/21 09:14:33 Testing if Obtaining a Multi-tenant token from the Azure CLI is applicable for Authentication..
I0421 09:14:33.373553     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Testing if Obtaining a Multi-tenant token from the Azure CLI is applicable for Authentication..<{%reset%}>)
I0421 09:14:33.373704     304 log.go:75] 2023/04/21 09:14:33 Testing if Obtaining a token from the Azure CLI is applicable for Authentication..
I0421 09:14:33.373710     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Testing if Obtaining a token from the Azure CLI is applicable for Authentication..<{%reset%}>)
I0421 09:14:33.373968     304 log.go:75] 2023/04/21 09:14:33 Using Obtaining a token from the Azure CLI for Authentication
I0421 09:14:33.373973     304 log.go:75] eventSink::Debug(<{%reset%}>2023/04/21 09:14:33 Using Obtaining a token from the Azure CLI for Authentication<{%reset%}>)
I0421 09:14:33.374217     304 log.go:75] Provider[azure-native, 0xc000ba4690].Configure() failed: err=please ensure you have installed Azure CLI version 2.0.79 or newer. Error parsing json result from the Azure CLI: launching Azure CLI: exec: "az": executable file not found in $PATH.

Looking at hashicorp/go-azure-helpers source I found following snippet. Mentioned environment variables are set within the Container App environment.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

@jirikopecky Thanks for the detailed issue description and logs and I apologize for this issue that you're running into. Looking into hashicorp/go-azure-helpers and the code snippet you provided, it appears that in order for MSI to be used as an authentication method, MSI_ENDPOINT and MSI_SECRET must be unset/empty as they specifically mention:
if the Endpoint and Sender are present this is App Service/Function Apps which we intentionally don't support at this time .

Having these 2 env variables set would result in the MSI authentication mode to be marked as not applicable, and skipped as an auth method.

Could you try to unset these 2 variables in your container and see if it resolves your issue?

[jirikopecky]

These environment variables are set automatically by Azure Container Apps runtime and as such are out of our control, so we cannot unset them.

[michal-bajer1]

Has there been any progress on this? Unsetting these variables also doesn't work, because then the library can't find the correct MSI endpoint.
I also tried creating workaround using az login --identity, but that is also blocked, because CLI login can only be used by User, not service principal.

[sureshgarre]

Is there any progress on this please? We are currently facing this issue when trying to run pulumi code within an Azure container app which authenticates to Azure using managed identity (MSI).

[JontyMC]

+1

[thomas11]

The new v2.70 release should fix this as it introduces a new authentication backend with support for various variants of MSI such as Service Fabric, App Service, Arc, Azure ML, and Cloud Shell. You can see all the details in the package source.

The new authentication backend is behind a feature toggle for the time being. Set environment variable PULUMI_ENABLE_AZCORE_BACKEND to "true" to use it. Please report back any issues you may be encountering.

As I don't have a working repro for this issue, I couldn't verify that the new backend fixes it, but I think it will.

[thomas11]

I'll optimistically close this issue but feel free to re-open if you still face issues.