Skip to content

Commit

Permalink
feat: add cloudConfig support in member-agent (#939)
Browse files Browse the repository at this point in the history
  • Loading branch information
@britaniar
britaniar authored Nov 11, 2024
1 parent 67c349a commit 381e56e
Show file tree
Hide file tree
Showing 9 changed files with 158 additions and 54 deletions.
69 changes: 56 additions & 13 deletions charts/member-agent/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,61 @@ helm upgrade member-agent member-agent/ --namespace fleet-system

## Parameters

| Parameter | Description | Default |
|:-------------------------|:------------------------------------------------------|:------------------------------------------------|
| replicaCount | The number of member-agent replicas to deploy | `1` |
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/member-agent` |
| image.pullPolicy | Image pullPolicy | `IfNotPresent` |
| image.tag | The image tag to use | `v0.1.0` |
| affinity | The node affinity to use for pod scheduling | `{}` |
| tolerations | The toleration to use for pod scheduling | `[]` |
| resources | The resource request/limits for the container image | limits: "2" CPU, 4Gi, requests: 100m CPU, 128Mi |
| namespace | Namespace that this Helm chart is installed on. | `fleet-system` |
| logVerbosity | Log level. Uses V logs (klog) | `3` |
| propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` |
| region | The region where the member cluster resides | `` |
| Parameter | Description | Default |
|:------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------|
| replicaCount | The number of member-agent replicas to deploy | `1` |
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/member-agent` |
| image.pullPolicy | Image pullPolicy | `IfNotPresent` |
| image.tag | The image tag to use | `v0.1.0` |
| affinity | The node affinity to use for pod scheduling | `{}` |
| tolerations | The toleration to use for pod scheduling | `[]` |
| resources | The resource request/limits for the container image | limits: "2" CPU, 4Gi, requests: 100m CPU, 128Mi |
| namespace | Namespace that this Helm chart is installed on. | `fleet-system` |
| logVerbosity | Log level. Uses V logs (klog) | `3` |
| propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` |
| region | The region where the member cluster resides | `` |
| config.azureCloudConfig | The cloud provider configuration | **required if property provider is set to azure** |

## Override Azure cloud config

**If PropertyProvider feature is set to azure, then a cloud configuration is required.**
Cloud configuration provides resource metadata and credentials for `fleet-member-agent` to manipulate Azure resources.
It's embedded into a Kubernetes secret and mounted to the pods.
The values can be modified under `config.azureCloudConfig` section in values.yaml or can be provided as a separate file.


| configuration value | description | Remark |
|-------------------------------------------------------| --- |---------------------------------------------------------------------------|
| `cloud` | The cloud where resources belong. | Required. |
| `tenantId` | The AAD Tenant ID for the subscription where the Azure resources are deployed. | |
| `subscriptionId` | The ID of the subscription where resources are deployed. | |
| `useManagedIdentityExtension` | Boolean indicating whether or not to use a managed identity. | `true` or `false` |
| `userAssignedIdentityID` | ClientID of the user-assigned managed identity with RBAC access to resources. | Required for UserAssignedIdentity and omitted for SystemAssignedIdentity. |
| `aadClientId` | The ClientID for an AAD application with RBAC access to resources. | Required if `useManagedIdentityExtension` is set to `false`. |
| `aadClientSecret` | The ClientSecret for an AAD application with RBAC access to resources. | Required if `useManagedIdentityExtension` is set to `false`. |
| `resourceGroup` | The name of the resource group where cluster resources are deployed. | |
| `userAgent` | The userAgent provided when accessing resources. | |
| `location` | The region where resource group and its resources is deployed. | |
| `vnetName` | The name of the virtual network where the cluster is deployed. | |
| `vnetResourceGroup` | The resource group where the virtual network is deployed. | |

You can create a file `azure.yaml` with the following content, and pass it to `helm install` command: `helm install <release-name> <chart-name> --set propertyProvider=azure -f azure.yaml`

```yaml
config:
azureCloudConfig:
cloud: "AzurePublicCloud"
tenantId: "00000000-0000-0000-0000-000000000000"
subscriptionId: "00000000-0000-0000-0000-000000000000"
useManagedIdentityExtension: false
userAssignedIdentityID: "00000000-0000-0000-0000-000000000000"
aadClientId: "00000000-0000-0000-0000-000000000000"
aadClientSecret: "<your secret>"
userAgent: "fleet-member-agent"
resourceGroup: "<resource group name>"
location: "<resource group location>"
vnetName: "<vnet name>"
vnetResourceGroup: "<vnet resource group>"
```
## Contributing Changes
10 changes: 10 additions & 0 deletions charts/member-agent/templates/cloudconfig.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{{- if eq .Values.propertyProvider "azure" }}
apiVersion: v1
kind: Secret
metadata:
name: cloud-config
namespace: {{ .Values.namespace }}
type: Opaque
data:
config.json: {{ .Values.config.azureCloudConfig | toJson | indent 4 | b64enc | quote }}
{{- end }}
23 changes: 22 additions & 1 deletion charts/member-agent/templates/deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ spec:
{{- if .Values.propertyProvider }}
- --property-provider={{ .Values.propertyProvider }}
{{- end }}
{{- if eq .Values.propertyProvider "azure" }}
- --cloud-config=/etc/kubernetes/provider/config.json
{{- end }}
{{- if .Values.region }}
- --region={{ .Values.region }}
{{- end }}
Expand Down Expand Up @@ -80,10 +83,19 @@ spec:
httpGet:
path: /readyz
port: hubhealthz
{{- if not .Values.useCAAuth }}
{{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }}
volumeMounts:
{{- if not .Values.useCAAuth }}
- name: provider-token
mountPath: /config
{{- end }}
{{- if eq .Values.propertyProvider "azure" }}
- name: cloud-provider-config
mountPath: /etc/kubernetes/provider
readOnly: true
{{- end }}
{{- end }}
{{- if not .Values.useCAAuth }}
- name: refresh-token
image: "{{ .Values.refreshtoken.repository }}:{{ .Values.refreshtoken.tag }}"
imagePullPolicy: {{ .Values.refreshtoken.pullPolicy }}
Expand All @@ -102,10 +114,19 @@ spec:
volumeMounts:
- name: provider-token
mountPath: /config
{{- end }}
{{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }}
volumes:
{{- if not .Values.useCAAuth }}
- name: provider-token
emptyDir: {}
{{- end }}
{{- if eq .Values.propertyProvider "azure" }}
- name: cloud-provider-config
secret:
secretName: cloud-config
{{- end }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
13 changes: 13 additions & 0 deletions charts/member-agent/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,19 @@ config:
identityKey: "identity-key-path"
identityCert: "identity-cert-path"
CABundle: "ca-bundle-path"
azureCloudConfig:
cloud: ""
tenantId: ""
subscriptionId: ""
useManagedIdentityExtension: false
userAssignedIdentityID: ""
aadClientId: ""
aadClientSecret: ""
resourceGroup: ""
userAgent: ""
location: ""
vnetName: ""
vnetResourceGroup: ""

secret:
name: "hub-kubeconfig-secret"
Expand Down
3 changes: 3 additions & 0 deletions cmd/memberagent/main.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ var (
enableV1Beta1APIs = flag.Bool("enable-v1beta1-apis", false, "If set, the agents will watch for the v1beta1 APIs.")
propertyProvider = flag.String("property-provider", "none", "The property provider to use for the agent.")
region = flag.String("region", "", "The region where the member cluster resides.")
cloudConfigFile = flag.String("cloud-config", "/etc/kubernetes/provider/config.json", "The path to the cloud cloudconfig file.")
)

func init() {
Expand Down Expand Up @@ -367,6 +368,8 @@ func Start(ctx context.Context, hubCfg, memberConfig *rest.Config, hubOpts, memb
klog.V(2).Info("setting up the Azure property provider")
// Note that the property provider, though initialized here, is not started until
// the specific instance wins the leader election.
klog.V(1).InfoS("Property Provider is azure, loading cloud config", "cloudConfigFile", *cloudConfigFile)
// TODO (britaniar): load cloud config for Azure property provider.
pp = azure.New(region)
default:
// Fall back to not using any property provider if the provided type is none or
Expand Down
26 changes: 13 additions & 13 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,26 +14,26 @@ require (
github.com/onsi/gomega v1.35.1
github.com/prometheus/client_golang v1.19.1
github.com/prometheus/client_model v0.6.1
github.com/spf13/cobra v1.8.0
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.9.0
go.goms.io/fleet-networking v0.2.7
go.uber.org/atomic v1.11.0
go.uber.org/zap v1.27.0
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8
golang.org/x/exp v0.0.0-20241004190924-225e2abe05e6
golang.org/x/sync v0.8.0
golang.org/x/time v0.7.0
k8s.io/api v0.30.2
k8s.io/apiextensions-apiserver v0.30.2
k8s.io/apimachinery v0.30.2
k8s.io/client-go v0.30.2
k8s.io/component-base v0.30.2
k8s.io/klog/v2 v2.120.1
k8s.io/klog/v2 v2.130.1
k8s.io/metrics v0.25.2
k8s.io/utils v0.0.0-20240502163921-fe8a2dddb1d0
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8
sigs.k8s.io/cloud-provider-azure v1.28.2
sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50
sigs.k8s.io/controller-runtime v0.18.4
sigs.k8s.io/controller-runtime v0.18.5
sigs.k8s.io/work-api v0.0.0-20220407021756-586d707fdb2c
)

Expand All @@ -50,7 +50,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/privatedns/armprivatedns v1.3.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.6.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.3.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.1.0 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest v0.11.29 // indirect
Expand All @@ -60,13 +60,13 @@ require (
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.1 // indirect
github.com/aws/karpenter-core v0.32.2-0.20231109191441-e32aafc81fb5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blendle/zapdriver v1.3.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/emicklei/go-restful/v3 v3.12.1 // indirect
github.com/evanphx/json-patch v5.9.0+incompatible // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
Expand Down Expand Up @@ -96,8 +96,8 @@ require (
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/common v0.54.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/common v0.55.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/samber/lo v1.38.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.52.0 // indirect
Expand All @@ -106,7 +106,7 @@ require (
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.28.0 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/oauth2 v0.23.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/term v0.25.0 // indirect
golang.org/x/text v0.19.0 // indirect
Expand All @@ -116,7 +116,7 @@ require (
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/kube-openapi v0.0.0-20240521193020-835d969ad83a // indirect
k8s.io/kube-openapi v0.0.0-20240903163716-9e1beecbcb38 // indirect
knative.dev/pkg v0.0.0-20231010144348-ca8c009405dd // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
Expand Down
Loading

0 comments on commit 381e56e

Please sign in to comment.