[Umbrella] [Karmada config && certificates] secret and path naming convention

[chaosi-zju]

Task one. [Karmada Config] secret and path naming convention

Naming convention:

• secretName: ${karmada_instance_name}-${component}-config
• volumeName: karmada-config
• volumeMountPath: /etc/karmada/config

${karmada_instance_name} defaults to value karmada, but theoretically it also could be karmada-xxx or xxx-karmada.

Needed PRs:

• local up: standardize the naming of karmada config in local up method #5679 @chaosi-zju
• karmadactl: standardize the naming of karmada config in karmadactl method #5797 @chaosi-zju
• helm: @help-wanted
• karmada-operator: @help-wanted

Examples:

Karmada config

1. karmada-aggregated-apiserver

secretName: karmada-aggregated-apiserver-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

2. karmada-controller-manager

secretName: karmada-controller-manager-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

3. karmada-scheduler

secretName: karmada-scheduler-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

4. karmada-descheduler

secretName: karmada-descheduler-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

5. karmada-metrics-adapter

secretName: karmada-metrics-adapter-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

6. karmada-search

secretName: karmada-search-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

7. karmada-webhook

secretName: karmada-webhook-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config

8. kube-controller-manager

secretName: kube-controller-manager-config
volumeName: karmada-config
volumeMountPath: /etc/karmada/config
command:
  - --kubeconfig=/etc/karmada/config/karmada.config
  - --authentication-kubeconfig=/etc/karmada/config/karmada.config
  - --authorization-kubeconfig=/etc/karmada/config/karmada.config

Task two. [Karmada Certificate] secret and path naming convention

Naming convention:

• Server Certificate:

  • secretName: ${karmada_instance_name}-${component}-cert
  • volumeName: server-cert
  • volumeMountPath: /etc/karmada/pki/server
  • fileName: ca.crt、tls.crt、tls.key

• Client Certificate:

  • Secret: ${karmada_instance_name}-${component}-${server}-client-cert
  • Volume: ${server}-client-cert
  • Volume mount path: /etc/karmada/pki/${server}-client
  • fileName: ca.crt、tls.crt、tls.key

${karmada_instance_name} defaults to value karmada, but theoretically it also could be karmada-xxx or xxx-karmada.

Needed PRs:

• lcoal up: standardize the naming of karmada secrets in local up method #5423 @chaosi-zju
• karmadactl: @help-wanted
• helm: @help-wanted
• karmada-operator: @help-wanted

Examples:

karmada certificates

1. karmada-etcd

command:
  - --cert-file=/etc/karmada/pki/server/tls.crt
  - --key-file=/etc/karmada/pki/server/tls.key
  - --trusted-ca-file=/etc/karmada/pki/server/ca.crt
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
volumes:
  - name: server-cert
    secret:
      secretName: etcd-cert
  - name: etcd-client-cert
    secret:
      secretName: etcd-etcd-client-cert

2. karmada-apiserver

command:
  - --etcd-cafile=/etc/karmada/pki/etcd-client/ca.crt
  - --etcd-certfile=/etc/karmada/pki/etcd-client/tls.crt
  - --etcd-keyfile=/etc/karmada/pki/etcd-client/tls.key
  - --service-account-key-file=/etc/karmada/pki/service-account-key-pair/sa.pub
  - --service-account-signing-key-file=/etc/karmada/pki/service-account-key-pair/sa.key
  - --proxy-client-cert-file=/etc/karmada/pki/front-proxy-client/tls.crt
  - --proxy-client-key-file=/etc/karmada/pki/front-proxy-client/tls.key
  - --requestheader-client-ca-file=/etc/karmada/pki/front-proxy-client/ca.crt
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki/server/tls.key
  - --client-ca-file=/etc/karmada/pki/server/ca.crt
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
    readOnly: true
  - name: front-proxy-client-cert
    mountPath: /etc/karmada/pki/front-proxy-client
    readOnly: true
  - name: service-account-key-pair
    mountPath: /etc/karmada/pki/service-account-key-pair
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-apiserver-cert
  - name: etcd-client-cert
    secret:
      secretName: karmada-apiserver-etcd-client-cert
  - name: front-proxy-client-cert
    secret:
      secretName: karmada-apiserver-front-proxy-client-cert
  - name: service-account-key-pair
    secret:
      secretName: karmada-apiserver-service-account-key-pair

3. karmada-aggregated-apiserver

command:
  - --etcd-cafile=/etc/karmada/pki/etcd-client/ca.crt
  - --etcd-certfile=/etc/karmada/pki/etcd-client/tls.crt
  - --etcd-keyfile=/etc/karmada/pki/etcd-client/tls.key
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki//server/tls.key
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-aggregated-apiserver-cert
  - name: etcd-client-cert
    secret:
      secretName: karmada-aggregated-apiserver-etcd-client-cert

4. karmada-scheduler

command:
  - --scheduler-estimator-ca-file=/etc/karmada/pki/scheduler-estimator-client/ca.crt
  - --scheduler-estimator-cert-file=/etc/karmada/pki/scheduler-estimator-client/tls.crt
  - --scheduler-estimator-key-file=/etc/karmada/pki/scheduler-estimator-client/tls.key
volumeMounts:
  - name: scheduler-estimator-client-cert
    mountPath: /etc/karmada/pki/scheduler-estimator-client
    readOnly: true
volumes:
  - name: scheduler-estimator-client-cert
    secret:
      secretName: karmada-scheduler-scheduler-estimator-client-cert

5. karmada-descheduler

command:
  - --scheduler-estimator-ca-file=/etc/karmada/pki/scheduler-estimator-client/ca.crt
  - --scheduler-estimator-cert-file=/etc/karmada/pki/scheduler-estimator-client/tls.crt
  - --scheduler-estimator-key-file=/etc/karmada/pki/scheduler-estimator-client/tls.key
volumeMounts:
  - name: scheduler-estimator-client-cert
    mountPath: /etc/karmada/pki/scheduler-estimator-client
    readOnly: true
volumes:
  - name: scheduler-estimator-client-cert
    secret:
      secretName: karmada-scheduler-scheduler-estimator-client-cert

6. karmada-scheduler-estimator

command:
  - --grpc-auth-cert-file=/etc/karmada/pki/server/tls.crt
  - --grpc-auth-key-file=/etc/karmada/pki/server/tls.key
  - --grpc-client-ca-file=/etc/karmada/pki/server/ca.crt
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-metrics-adapter-cert

7. karmada-metrics-adapter

command:
  - --client-ca-file=/etc/karmada/pki/server/ca.crt
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki/server/tls.key
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-metrics-adapter-cert

8. karmada-search

command:
  - --etcd-cafile=/etc/karmada/pki/etcd-client/ca.crt
  - --etcd-certfile=/etc/karmada/pki/etcd-client/tls.crt
  - --etcd-keyfile=/etc/karmada/pki/etcd-client/tls.key
  - --tls-cert-file=/etc/karmada/pki/server/tls.crt
  - --tls-private-key-file=/etc/karmada/pki/server/tls.key
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
  - name: etcd-client-cert
    mountPath: /etc/karmada/pki/etcd-client
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-search-cert
  - name: etcd-client-cert
    secret:
      secretName: karmada-search-etcd-client-cert

9. karmada-webhook

command:
  - --cert-dir=/etc/karmada/pki/server
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-webhook-cert

10. kube-controller-manager

# --client-ca-file verifies the cert of its client like kubelet and other controller
# --cluster-signing-key-file is used for signing certificates
# --root-ca-file is stored in service account type secret
command:
  - --client-ca-file=/etc/karmada/pki/ca/tls.crt
  - --cluster-signing-cert-file=/etc/karmada/pki/ca/tls.crt
  - --cluster-signing-key-file=/etc/karmada/pki/ca/tls.key
  - --root-ca-file=/etc/karmada/pki/ca/tls.crt
  - --service-account-private-key-file=/etc/karmada/pki/service-account-key-pair/sa.key
volumeMounts:
  - name: ca-cert
    mountPath: /etc/karmada/pki/ca
    readOnly: true
  - name: service-account-key-pair
    mountPath: /etc/karmada/pki/service-account-key-pair
    readOnly: true
volumes:
  - name: ca-cert
    secret:
      secretName: kube-controller-manager-ca-cert
  - name: service-account-key-pair
    secret:
      secretName: kube-controller-manager-service-account-key-pair

11. karmada-interpreter-webhook-example

command:
  - --cert-dir=/etc/karmada/pki/server
volumeMounts:
  - name: server-cert
    mountPath: /etc/karmada/pki/server
    readOnly: true
volumes:
  - name: server-cert
    secret:
      secretName: karmada-interpreter-webhook-example-cert

Legacy issue

• We initially hoped to standardize the secret name to a fixed name during this refactoring, using a format like karmada-scheduler-config (following the ${component}-config convention). However, a few users expressed the need to install two Karmada instances within the same namespace, which result in two sets of secrets, preventing us from establishing a fixed secret name.

Resolution: in helm or operator, component name is prefixed with karmada_instance_name, like karmada-xxx-scheduler, so its secret name is defined as ${karmada_instance_name}-${component}-config

• cert_rotation_controller of karmada-agent has hard-coded karmada config secret name, so if we rename the secret name of karmada-agent, it would affect the upgrade of karmada-agent.

karmada/pkg/controllers/certificate/cert_rotation_controller.go

Lines 59 to 60 in ce41488

	// KarmadaKubeconfigName is the name of the secret containing karmada-agent certificate.
	KarmadaKubeconfigName = "karmada-kubeconfig"

Resolution: this time the karmada-agent is not involved, only the control-plane components is changed.

[chaosi-zju]

/help

[karmada-bot]

@chaosi-zju:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[seanlaii]

Hi @chaosi-zju , could I work on the karmada-operator update in task one?