Support for system-assigned and user-assigned managed identities #330

ulucinar · 2022-03-23T23:09:24Z

Description of your changes

This PR adds support for system-assigned and user-assigned managed identities, formerly known as Managed Service Identities (MSIs) via the Azure Identity client module.

An example ProviderConfig that uses a system-assigned managed identity is as follows:

apiVersion: azure.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: example
spec:
  credentials:
    source: InjectedIdentity
  subscriptionID: <Azure subscription ID>

Another example using a user-assigned managed identity:

apiVersion: azure.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: example
spec:
  credentials:
    source: InjectedIdentity
  subscriptionID: <Azure subscription ID>
  clientID: <Client ID of the user-assigned managed kubelet identity>

I have:

Read and followed Crossplane's contribution process.
Run make reviewable test to ensure this PR is ready for review.

How has this code been tested

I have successfully provisioned a ResourceGroup and an AKSCluster using a system-assigned managed identity and also a user-assigned managed identity with the ProviderConfigs given above.

- Managed identities are formerly known as MSIs Signed-off-by: Alper Rifat Ulucinar <[email protected]>

- If we are using an authentication scheme other than client secret returning a map of credentials secret data is no longer possible. Thus, we need a change in the function signature. - azure.GetAuthInfo now returns subscription ID, which is used by the downstream ARM resource SDK clients. Signed-off-by: Alper Rifat Ulucinar <[email protected]>

Signed-off-by: Alper Rifat Ulucinar <[email protected]>

ezgidemirel · 2022-04-22T15:23:12Z

pkg/clients/compute/aks.go

 // EnsureManagedCluster ensures the supplied AKS cluster exists, including
 // ensuring any required service principals and role assignments exist.


It would be good to update this comment since we're no longer ensuring SP creation and its role assignments.

Thank you @ezgidemirel. Removed mentions of the service principals and role assignments as we are no longer dealing with them.

ezgidemirel · 2022-04-22T15:29:12Z

pkg/clients/compute/aks.go

 // DeleteManagedCluster deletes the supplied AKS cluster, including its service
 // principals and any role assignments.


Same as above, we're not deleting SP or its role assignments

ezgidemirel · 2022-04-22T15:31:07Z

pkg/clients/compute/aks.go

@@ -280,38 +129,40 @@ func newManagedCluster(c *v1alpha3.AKSCluster, appID, secret string) containerse
 				{
 					Name:   to.StringPtr(AgentPoolProfileName),
 					Count:  &nodeCount,
-					VMSize: containerservice.VMSizeTypes(c.Spec.NodeVMSize),
+					VMSize: to.StringPtr(c.Spec.NodeVMSize),
+					Mode:   containerservice.AgentPoolModeSystem,


Even though system mode is set in here, I got the following error when I tried to create an AKS cluster with a user-managed identity

Warning CannotCreateExternalResource 96s (x25 over 3m21s) managed/akscluster.compute.azure.crossplane.io cannot create AKSCluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="MustDefineAtLeastOneSystemPool" Message="Must define at least one system pool."

I used following manifest:

apiVersion: compute.azure.crossplane.io/v1alpha3 kind: AKSCluster metadata: name: ezgi-akscluster labels: example: "true" spec: resourceGroupName: ezgi vnetSubnetIDRef: name: ezgi-sub location: West US 2 version: "1.21.7" nodeCount: 1 nodeVMSize: Standard_B2s dnsNamePrefix: ezgi-aks disableRBAC: false identity: identityNames: - ezgiidentity type: UserAssigned providerConfigRef: name: example writeConnectionSecretToRef: namespace: crossplane-system name: ezgi-akscluster

Hi @ezgidemirel,
Thank you for giving this a try. It looks like to be an issue specific to provisioning AKS clusters as you have mentioned below that provisioning some other resources were successful. Using SP credentials, I tried provisioning an AKS cluster with success using the example manifest from this PR. You are using the provider package ulucinar/provider-azure:build-3eacbce2, right?
I will also give it a try using a user-assigned managed identity in ProviderConfig and report back to you, although it should not be related.

Hello again,
Using the provider package ulucinar/provider-azure:build-3eacbce2, on an AKS cluster with a user-assigned managed kubelet identity, I tried to provision an AKS cluster using the following manifests successfully:

apiVersion: azure.crossplane.io/v1beta1 kind: ProviderConfig metadata: name: example spec: clientID: <client-id of the user-assigned managed kubelet identity> credentials: source: InjectedIdentity subscriptionID: <Azure subscription ID> --- apiVersion: compute.azure.crossplane.io/v1alpha3 kind: AKSCluster metadata: name: example-akscluster-alper2 labels: example: "true" spec: resourceGroupName: alper location: West US 2 version: "1.21.7" nodeCount: 1 nodeVMSize: Standard_B2s dnsNamePrefix: crossplane-aks disableRBAC: false identity: type: SystemAssigned providerConfigRef: name: example writeConnectionSecretToRef: namespace: crossplane-system name: example-akscluster

I have also verified accessing the provisioned AKS cluster using the credentials (kubeconfig) stored in the connection secret.

The AKS cluster I tried to provision has UserAssigned managed identity. I believe that's the reason why my attempt is failing.

Hi @ezgidemirel,
The following also succeeded in my cluster:

apiVersion: compute.azure.crossplane.io/v1alpha3 kind: AKSCluster metadata: name: example-akscluster-alper2 labels: example: "true" spec: resourceGroupName: alper location: West US 2 version: "1.21.7" nodeCount: 1 nodeVMSize: Standard_B2s dnsNamePrefix: crossplane-aks disableRBAC: false identity: type: UserAssigned identityNames: - control-plane-identity-alper providerConfigRef: name: example writeConnectionSecretToRef: namespace: crossplane-system name: example-akscluster

Updating resource manifests to use the same location didn't work. However, removing vnetSubnetIDRef from the spec did the trick. I could successfully provision an AKS cluster.

It turns out that we are not setting the default node pool's mode to System iff a subnet is specified :)
Fixed the bug and successfully provisioned an AKS cluster by specifying a subnet for its node pool.

Thank you very much @ezgidemirel for finding this issue and helping me in fixing it. Very much appreciated!

ezgidemirel · 2022-04-22T15:34:33Z

pkg/clients/compute/aks.go

+		for _, n := range c.Spec.Identity.IdentityNames {
+			resourceID := fmt.Sprintf(fmtUserAssignedManagedIdentityID, subscriptionID, c.Spec.ResourceGroupName, n)
+			p.Identity.UserAssignedIdentities[resourceID] = &containerservice.ManagedClusterIdentityUserAssignedIdentitiesValue{}
+		}


I still wonder what is the use case of having multiple user identities here :) I'll dive in later.

ezgidemirel · 2022-04-22T15:47:43Z

I tested the PR with user-assigned managed identity with following steps:

Created an identitiy

az identity create --name ezgiidentity --resource-group ezgi --location westeurope

Assigned Managed Identity Operator and Contributor roles to it. Seeing these role assignments' effects took some time.
Created an AKS cluster and set its identity and its kubelet's identity to the managed one above

 az aks create --resource-group ezgi --name ezgi-cluster --enable-managed-identity --assign-identity $ID --location westeurope

Installed crossplane

up uxp installed -n crossplane-system

Installed provider built from this branch

kc install provider ulucinar/provider-azure:build-3eacbce2

Created a provider config with user-managed identity ID

apiVersion: azure.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: example
spec:
  credentials:
    source: InjectedIdentity
  subscriptionID: <sub-id>
  clientID: <client-id>

Successfully created a resourcegroup with example manifest
Successfully created subnet, virtual network with example manifets
Failed to create an AKS cluster with following error

Warning  CannotCreateExternalResource  3m16s (x340 over 28m)   managed/akscluster.compute.azure.crossplane.io  cannot create AKSCluster: containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="MustDefineAtLeastOneSystemPool" Message="Must define at least one system pool."

While preparing a cluster with user-managed identity, I noticed that kubelet's identity should also be set. Following curl command helped me to troubleshoot. I created a pod with curl binary in the cluster and executed it inside it:

curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -H Metadata:true -s

Signed-off-by: Alper Rifat Ulucinar <[email protected]>

…d identities instead of the plain identity names Signed-off-by: Alper Rifat Ulucinar <[email protected]>

ezgidemirel

Thanks a ton @ulucinar for adding managed identity support, fixing an existing bug and, cleaning up some SP generation code for AKS clusters. I learned a lot while reviewing/testing it. Amazing work from an amazing mentor :)

turkenh

lgtm!

ghost · 2022-05-27T12:53:44Z

@turkenh I have a client who would like to use Manage Identities. I would be happy to try and help fix whatever CodeCov is saying is wrong but I do not have access to see what the problem is. Can you let us know what it is or help to get it addressed so that we can all start using these amazingly helpful changes from @ulucinar?

carct · 2022-08-31T14:08:32Z

any estimate on this feature delivery? it would be really useful

marianheinsen · 2023-01-27T13:25:57Z

I would also love to see this PR merged because it would simplify our current setup using Crossplane with Service Principals a lot!

ahilmathew · 2023-04-24T20:20:58Z

Any chance we can get this merged?

ghost · 2023-07-26T20:26:26Z

Any ETA here? Hoping this helps me automate giving the role "DNS Zone Contributor" to my AKS clusters so external-dns will work.

mazilu88 · 2023-09-14T16:18:46Z

I'm looking at implementing this provider. The feature you've introduced aligns perfectly with our current needs. Thank you for your valuable work @ulucinar !

I see this was approved on May 9, 2022 but is still not merged because of codegov check.

@turkenh / @ezgidemirel if possible, could you revisit this matter for a second evaluation? The report is no longer available, and a previous user mentioned that external users don't have access to it. Your assistance in this matter would be greatly appreciated.

undefinedhuman · 2023-12-12T18:52:11Z

Any updates? Would love to see this feature implemented

ulucinar marked this pull request as draft March 23, 2022 23:09

ulucinar force-pushed the fix-164 branch from 27eaa1b to 8202857 Compare March 25, 2022 07:28

ulucinar marked this pull request as ready for review March 25, 2022 07:33

ulucinar requested a review from ezgidemirel March 25, 2022 07:37

This was referenced Mar 31, 2022

Support for Service Principals for AKS Clusters #331

Open

[Proposal] Use System Managed Identity OR User Managed Identity for AKSCluster #292

Open

ulucinar added 4 commits April 19, 2022 15:17

Support for system-assigned and user-assigned managed identities

817e2c7

- Managed identities are formerly known as MSIs Signed-off-by: Alper Rifat Ulucinar <[email protected]>

AKSCluster now only supports managed identities

2cb289e

Signed-off-by: Alper Rifat Ulucinar <[email protected]>

Fix tests

0911262

Signed-off-by: Alper Rifat Ulucinar <[email protected]>

ulucinar force-pushed the fix-164 branch from 8202857 to 0911262 Compare April 19, 2022 16:36

ezgidemirel reviewed Apr 22, 2022

View reviewed changes

ulucinar added 2 commits April 28, 2022 22:27

Fix AKS cluster node-pool's mode when a subnet for it is specified

0689a6a

Signed-off-by: Alper Rifat Ulucinar <[email protected]>

AKSCluster now expects Azure resource IDs of the user-assigned manage…

845bebe

…d identities instead of the plain identity names Signed-off-by: Alper Rifat Ulucinar <[email protected]>

ulucinar requested review from ezgidemirel and turkenh April 28, 2022 20:28

ezgidemirel approved these changes Apr 29, 2022

View reviewed changes

turkenh approved these changes May 9, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for system-assigned and user-assigned managed identities #330

Support for system-assigned and user-assigned managed identities #330

ulucinar commented Mar 23, 2022 •

edited

Loading

ezgidemirel Apr 22, 2022

ulucinar Apr 28, 2022

ezgidemirel Apr 22, 2022

ulucinar Apr 28, 2022

ezgidemirel Apr 22, 2022

ulucinar Apr 22, 2022

ulucinar Apr 24, 2022

ezgidemirel Apr 25, 2022

ulucinar Apr 25, 2022

ezgidemirel Apr 26, 2022

ulucinar Apr 28, 2022

ulucinar Apr 28, 2022

ezgidemirel Apr 22, 2022

ezgidemirel commented Apr 22, 2022

ezgidemirel left a comment

turkenh left a comment

ghost commented May 27, 2022

carct commented Aug 31, 2022

marianheinsen commented Jan 27, 2023

ahilmathew commented Apr 24, 2023

ghost commented Jul 26, 2023

mazilu88 commented Sep 14, 2023

undefinedhuman commented Dec 12, 2023

		// EnsureManagedCluster ensures the supplied AKS cluster exists, including
		// ensuring any required service principals and role assignments exist.

		// DeleteManagedCluster deletes the supplied AKS cluster, including its service
		// principals and any role assignments.

Support for system-assigned and user-assigned managed identities #330

Are you sure you want to change the base?

Support for system-assigned and user-assigned managed identities #330

Conversation

ulucinar commented Mar 23, 2022 • edited Loading

Description of your changes

How has this code been tested

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ezgidemirel commented Apr 22, 2022

ezgidemirel left a comment

Choose a reason for hiding this comment

turkenh left a comment

Choose a reason for hiding this comment

ghost commented May 27, 2022

carct commented Aug 31, 2022

marianheinsen commented Jan 27, 2023

ahilmathew commented Apr 24, 2023

ghost commented Jul 26, 2023

mazilu88 commented Sep 14, 2023

undefinedhuman commented Dec 12, 2023

ulucinar commented Mar 23, 2022 •

edited

Loading