Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add cloudConfig support in member-agent #939

Merged
merged 13 commits into from
Nov 11, 2024
Merged

feat: add cloudConfig support in member-agent #939

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a424e0d
add cloudConfig & some tests
britaniar Oct 24, 2024
b87330b
address comments
britaniar Nov 1, 2024
9ae01fb
add check if property provider is azure
britaniar Nov 5, 2024
1d99418
fix readme
britaniar Nov 5, 2024
18b5ca9
check if property provider is azure
britaniar Nov 5, 2024
662bbcd
remove fields
britaniar Nov 7, 2024
0f3174b
restore
britaniar Nov 7, 2024
bd3ab93
minor fix
britaniar Nov 7, 2024
90599f7
fix deployemnt yaml
britaniar Nov 7, 2024
d26c8fe
load cloud config
britaniar Nov 7, 2024
dea6c7a
fix valid config
britaniar Nov 7, 2024
4eeb3b7
address comments
britaniar Nov 7, 2024
ffc0979
fix setup and change to azureCloudConfig
britaniar Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 56 additions & 13 deletions charts/member-agent/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,61 @@ helm upgrade member-agent member-agent/ --namespace fleet-system

## Parameters

| Parameter | Description | Default |
|:-------------------------|:------------------------------------------------------|:------------------------------------------------|
| replicaCount | The number of member-agent replicas to deploy | `1` |
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/member-agent` |
| image.pullPolicy | Image pullPolicy | `IfNotPresent` |
| image.tag | The image tag to use | `v0.1.0` |
| affinity | The node affinity to use for pod scheduling | `{}` |
| tolerations | The toleration to use for pod scheduling | `[]` |
| resources | The resource request/limits for the container image | limits: "2" CPU, 4Gi, requests: 100m CPU, 128Mi |
| namespace | Namespace that this Helm chart is installed on. | `fleet-system` |
| logVerbosity | Log level. Uses V logs (klog) | `3` |
| propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` |
| region | The region where the member cluster resides | `` |
| Parameter | Description | Default |
|:------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------|
| replicaCount | The number of member-agent replicas to deploy | `1` |
| image.repository | Image repository | `ghcr.io/azure/azure/fleet/member-agent` |
| image.pullPolicy | Image pullPolicy | `IfNotPresent` |
| image.tag | The image tag to use | `v0.1.0` |
| affinity | The node affinity to use for pod scheduling | `{}` |
| tolerations | The toleration to use for pod scheduling | `[]` |
| resources | The resource request/limits for the container image | limits: "2" CPU, 4Gi, requests: 100m CPU, 128Mi |
| namespace | Namespace that this Helm chart is installed on. | `fleet-system` |
| logVerbosity | Log level. Uses V logs (klog) | `3` |
| propertyProvider | The property provider to use with the member agent; if none is specified, the Fleet member agent will start with no property provider (i.e., the agent will expose no cluster properties, and collect only limited resource usage information) | `` |
| region | The region where the member cluster resides | `` |
| config.azureCloudConfig | The cloud provider configuration | **required if property provider is set to azure** |

## Override Azure cloud config

**If PropertyProvider feature is set to azure, then a cloud configuration is required.**
Cloud configuration provides resource metadata and credentials for `fleet-member-agent` to manipulate Azure resources.
It's embedded into a Kubernetes secret and mounted to the pods.
The values can be modified under `config.azureCloudConfig` section in values.yaml or can be provided as a separate file.


| configuration value | description | Remark |
|-------------------------------------------------------| --- |---------------------------------------------------------------------------|
| `cloud` | The cloud where resources belong. | Required. |
| `tenantId` | The AAD Tenant ID for the subscription where the Azure resources are deployed. | |
| `subscriptionId` | The ID of the subscription where resources are deployed. | |
| `useManagedIdentityExtension` | Boolean indicating whether or not to use a managed identity. | `true` or `false` |
| `userAssignedIdentityID` | ClientID of the user-assigned managed identity with RBAC access to resources. | Required for UserAssignedIdentity and omitted for SystemAssignedIdentity. |
| `aadClientId` | The ClientID for an AAD application with RBAC access to resources. | Required if `useManagedIdentityExtension` is set to `false`. |
| `aadClientSecret` | The ClientSecret for an AAD application with RBAC access to resources. | Required if `useManagedIdentityExtension` is set to `false`. |
| `resourceGroup` | The name of the resource group where cluster resources are deployed. | |
| `userAgent` | The userAgent provided when accessing resources. | |
| `location` | The region where resource group and its resources is deployed. | |
| `vnetName` | The name of the virtual network where the cluster is deployed. | |
| `vnetResourceGroup` | The resource group where the virtual network is deployed. | |

You can create a file `azure.yaml` with the following content, and pass it to `helm install` command: `helm install <release-name> <chart-name> --set propertyProvider=azure -f azure.yaml`

```yaml
config:
azureCloudConfig:
cloud: "AzurePublicCloud"
tenantId: "00000000-0000-0000-0000-000000000000"
subscriptionId: "00000000-0000-0000-0000-000000000000"
useManagedIdentityExtension: false
userAssignedIdentityID: "00000000-0000-0000-0000-000000000000"
aadClientId: "00000000-0000-0000-0000-000000000000"
aadClientSecret: "<your secret>"
userAgent: "fleet-member-agent"
resourceGroup: "<resource group name>"
location: "<resource group location>"
vnetName: "<vnet name>"
vnetResourceGroup: "<vnet resource group>"
```

## Contributing Changes
10 changes: 10 additions & 0 deletions charts/member-agent/templates/cloudconfig.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{{- if eq .Values.propertyProvider "azure" }}
apiVersion: v1
britaniar marked this conversation as resolved.
Show resolved Hide resolved
kind: Secret
metadata:
name: cloud-config
namespace: {{ .Values.namespace }}
type: Opaque
data:
config.json: {{ .Values.config.azureCloudConfig | toJson | indent 4 | b64enc | quote }}
{{- end }}
23 changes: 22 additions & 1 deletion charts/member-agent/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ spec:
{{- if .Values.propertyProvider }}
- --property-provider={{ .Values.propertyProvider }}
{{- end }}
{{- if eq .Values.propertyProvider "azure" }}
- --cloud-config=/etc/kubernetes/provider/config.json
{{- end }}
{{- if .Values.region }}
- --region={{ .Values.region }}
{{- end }}
Expand Down Expand Up @@ -80,10 +83,19 @@ spec:
httpGet:
path: /readyz
port: hubhealthz
{{- if not .Values.useCAAuth }}
{{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }}
volumeMounts:
britaniar marked this conversation as resolved.
Show resolved Hide resolved
{{- if not .Values.useCAAuth }}
- name: provider-token
britaniar marked this conversation as resolved.
Show resolved Hide resolved
mountPath: /config
{{- end }}
{{- if eq .Values.propertyProvider "azure" }}
- name: cloud-provider-config
britaniar marked this conversation as resolved.
Show resolved Hide resolved
mountPath: /etc/kubernetes/provider
readOnly: true
{{- end }}
britaniar marked this conversation as resolved.
Show resolved Hide resolved
{{- end }}
{{- if not .Values.useCAAuth }}
- name: refresh-token
britaniar marked this conversation as resolved.
Show resolved Hide resolved
image: "{{ .Values.refreshtoken.repository }}:{{ .Values.refreshtoken.tag }}"
imagePullPolicy: {{ .Values.refreshtoken.pullPolicy }}
Expand All @@ -102,10 +114,19 @@ spec:
volumeMounts:
- name: provider-token
mountPath: /config
{{- end }}
{{- if or (not .Values.useCAAuth) (eq .Values.propertyProvider "azure") }}
volumes:
britaniar marked this conversation as resolved.
Show resolved Hide resolved
{{- if not .Values.useCAAuth }}
- name: provider-token
britaniar marked this conversation as resolved.
Show resolved Hide resolved
emptyDir: {}
{{- end }}
{{- if eq .Values.propertyProvider "azure" }}
- name: cloud-provider-config
britaniar marked this conversation as resolved.
Show resolved Hide resolved
secret:
secretName: cloud-config
{{- end }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
13 changes: 13 additions & 0 deletions charts/member-agent/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,19 @@ config:
identityKey: "identity-key-path"
identityCert: "identity-cert-path"
CABundle: "ca-bundle-path"
azureCloudConfig:
cloud: ""
tenantId: ""
subscriptionId: ""
useManagedIdentityExtension: false
userAssignedIdentityID: ""
aadClientId: ""
aadClientSecret: ""
resourceGroup: ""
userAgent: ""
britaniar marked this conversation as resolved.
Show resolved Hide resolved
location: ""
vnetName: ""
vnetResourceGroup: ""

secret:
name: "hub-kubeconfig-secret"
Expand Down
3 changes: 3 additions & 0 deletions cmd/memberagent/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ var (
enableV1Beta1APIs = flag.Bool("enable-v1beta1-apis", false, "If set, the agents will watch for the v1beta1 APIs.")
propertyProvider = flag.String("property-provider", "none", "The property provider to use for the agent.")
region = flag.String("region", "", "The region where the member cluster resides.")
cloudConfigFile = flag.String("cloud-config", "/etc/kubernetes/provider/config.json", "The path to the cloud cloudconfig file.")
)

func init() {
Expand Down Expand Up @@ -367,6 +368,8 @@ func Start(ctx context.Context, hubCfg, memberConfig *rest.Config, hubOpts, memb
klog.V(2).Info("setting up the Azure property provider")
// Note that the property provider, though initialized here, is not started until
// the specific instance wins the leader election.
klog.V(1).InfoS("Property Provider is azure, loading cloud config", "cloudConfigFile", *cloudConfigFile)
// TODO (britaniar): load cloud config for Azure property provider.
pp = azure.New(region)
default:
// Fall back to not using any property provider if the provided type is none or
Expand Down
26 changes: 13 additions & 13 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,26 +14,26 @@ require (
github.com/onsi/gomega v1.35.1
github.com/prometheus/client_golang v1.19.1
github.com/prometheus/client_model v0.6.1
github.com/spf13/cobra v1.8.0
github.com/spf13/cobra v1.8.1
github.com/spf13/pflag v1.0.5
github.com/stretchr/testify v1.9.0
go.goms.io/fleet-networking v0.2.7
go.uber.org/atomic v1.11.0
go.uber.org/zap v1.27.0
golang.org/x/exp v0.0.0-20240613232115-7f521ea00fb8
golang.org/x/exp v0.0.0-20241004190924-225e2abe05e6
golang.org/x/sync v0.8.0
golang.org/x/time v0.7.0
k8s.io/api v0.30.2
k8s.io/apiextensions-apiserver v0.30.2
k8s.io/apimachinery v0.30.2
k8s.io/client-go v0.30.2
k8s.io/component-base v0.30.2
k8s.io/klog/v2 v2.120.1
k8s.io/klog/v2 v2.130.1
k8s.io/metrics v0.25.2
k8s.io/utils v0.0.0-20240502163921-fe8a2dddb1d0
k8s.io/utils v0.0.0-20240711033017-18e509b52bc8
sigs.k8s.io/cloud-provider-azure v1.28.2
sigs.k8s.io/cloud-provider-azure/pkg/azclient v0.0.50
sigs.k8s.io/controller-runtime v0.18.4
sigs.k8s.io/controller-runtime v0.18.5
sigs.k8s.io/work-api v0.0.0-20220407021756-586d707fdb2c
)

Expand All @@ -50,7 +50,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/privatedns/armprivatedns v1.3.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/storage/armstorage v1.6.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.2.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azsecrets v1.3.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.1.0 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest v0.11.29 // indirect
Expand All @@ -60,13 +60,13 @@ require (
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.3.1 // indirect
github.com/aws/karpenter-core v0.32.2-0.20231109191441-e32aafc81fb5 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blendle/zapdriver v1.3.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/emicklei/go-restful/v3 v3.12.1 // indirect
github.com/evanphx/json-patch v5.9.0+incompatible // indirect
github.com/fsnotify/fsnotify v1.7.0 // indirect
github.com/go-logr/zapr v1.3.0 // indirect
Expand Down Expand Up @@ -96,8 +96,8 @@ require (
github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/prometheus/common v0.54.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/common v0.55.0 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
github.com/samber/lo v1.38.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.52.0 // indirect
Expand All @@ -106,7 +106,7 @@ require (
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.28.0 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/oauth2 v0.23.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/term v0.25.0 // indirect
golang.org/x/text v0.19.0 // indirect
Expand All @@ -116,7 +116,7 @@ require (
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
k8s.io/kube-openapi v0.0.0-20240521193020-835d969ad83a // indirect
k8s.io/kube-openapi v0.0.0-20240903163716-9e1beecbcb38 // indirect
knative.dev/pkg v0.0.0-20231010144348-ca8c009405dd // indirect
sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd // indirect
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
Expand Down
Loading
Loading