unpredictable extra args #12021

tagurus · 2025-03-05T11:19:13Z

What happened?

kubespray has ability to add Custom flags to control plane components. But in docs doesnt writes, which extra arg are possible.

E.G.
my extra args

kube_kubeadm_controller_extra_args:
  node-monitor-grace-period: 20s
  allocate-node-cidrs: true

kube_kubeadm_apiserver_extra_args:
  default-not-ready-toleration-seconds: 10
  default-unreachable-toleration-seconds: 10

kubelet_config_extra_args:
  nodeStatusUpdateFrequency: 5s 
  nodeStatusReportFrequency: 5s

kubelet is ok

but api-server and controller-manager in generated kubeadmconfig

  - name: default-not-ready-toleration-seconds  DEFAULT VALUES
    value: "300"
  - name: default-unreachable-toleration-seconds
    value: "300"
...
  - name: "default-not-ready-toleration-seconds"   MY ARGS
    value: "10"
  - name: "default-unreachable-toleration-seconds"
    value: "10"

controllerManager:
  extraArgs:
  - name: node-monitor-grace-period DEFAULT
    value: "40s"

...
  - name: "node-monitor-grace-period" DUPLICATE
    value: "20s"
  - name: "allocate-node-cidrs" OK
    value: "True"

I suspect - jinja generete this value from defaut var

{% if kube_apiserver_pod_eviction_not_ready_timeout_seconds is defined %}
  - name: default-not-ready-toleration-seconds
    value: "{{ kube_apiserver_pod_eviction_not_ready_timeout_seconds }}"
{% endif %}
{% if kube_apiserver_pod_eviction_unreachable_timeout_seconds is defined %}
  - name: default-unreachable-toleration-seconds
    value: "{{ kube_apiserver_pod_eviction_unreachable_timeout_seconds }}"
{% endif %}

 - name: node-monitor-grace-period
    value: "{{ kube_controller_node_monitor_grace_period }}"

It is another vars, when classic extra

In this case
Add section in docementation about default vars, which opposite extra args
or
Make default args more obviously, for avoid collision

What did you expect to happen?

extra args added in kubeadm config is expected and planned

How can we reproduce it (as minimally and precisely as possible)?

add some extra args in k8s-cluster.yaml and deploy

OS

Ubuntu 22.04.5 LTS

Version of Ansible

2.16.10

Version of Python

3.10.12

Version of Kubespray (commit)

release-2.27

Network plugin used

calico

Full inventory with variables

.

Command used to invoke ansible

ansible-playbook -i ./inventory/kubetest/hosts.yaml -u *** -b --private-key *** cluster.yml --tags "master"

Output of ansible run

.

Anything else we need to know

No response

The text was updated successfully, but these errors were encountered:

tico88612 · 2025-03-06T07:38:17Z

/triage accepted

VannTen · 2025-03-06T13:33:09Z

Aren't duplicate in k8s components a case of last wins ?

tico88612 · 2025-03-07T14:44:59Z

@tagurus v1beta4 should allow duplication and be used last. Could you confirm that?

FYI: https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta4/

I realized that our other problem should be that extra Args is not at the end (it should be at the end, but we put it in the middle)

tagurus · 2025-03-13T12:50:36Z

@tico88612
kubead-config with dublicates in controller and api-server (dublicate in bottom)

apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.1.2
  bindPort: 6443
certificateKey: fbfbf29b3bff5e2dabeddfcf7fd8e51feed4813b4adb0eefc52a477ca3fdbaaa
nodeRegistration:
  name: "test-master-1"
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
  criSocket: unix:///var/run/containerd/containerd.sock
  kubeletExtraArgs:
  - name: cloud-provider
    value: external
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
clusterName: test.teamlead
encryptionAlgorithm: RSA-2048
etcd:
  local:
    imageRepository: "quay.io/coreos"
    imageTag: "v3.5.16"
    dataDir: "/var/lib/etcd"
    extraArgs:
    - name: metrics
      value: basic
    - name: election-timeout
      value: "5000"
    - name: heartbeat-interval
      value: "250"
    - name: auto-compaction-retention
      value: "8"
    - name: snapshot-count
      value: "10000"
    serverCertSANs:
      - "etcd.kube-system.svc.test.teamlead"
      - "etcd.kube-system.svc"
      - "etcd.kube-system"
      - "etcd"
    peerCertSANs:
      - "etcd.kube-system.svc.test.teamlead"
      - "etcd.kube-system.svc"
      - "etcd.kube-system"
      - "etcd"
dns:
  imageRepository: registry.k8s.io/coredns
  imageTag: v1.11.3
networking:
  dnsDomain: test.teamlead
  serviceSubnet: "192.168.1.32/27"
  podSubnet: "192.168.1.64/26"
kubernetesVersion: v1.31.4
controlPlaneEndpoint: 192.168.1.2:6443
certificatesDir: /etc/kubernetes/ssl
imageRepository: registry.k8s.io
apiServer:
  extraArgs:
  - name: etcd-compaction-interval
    value: "5m0s"
  - name: default-not-ready-toleration-seconds
    value: "300"
  - name: default-unreachable-toleration-seconds
    value: "300"
  - name: anonymous-auth
    value: "True"
  - name: authorization-mode
    value: "Node,RBAC"
  - name: bind-address
    value: "0.0.0.0"
  - name: apiserver-count
    value: "1"
  - name: endpoint-reconciler-type
    value: lease
  - name: service-node-port-range
    value: "30000-32767"
  - name: service-cluster-ip-range
    value: "192.168.1.32/27"
  - name: kubelet-preferred-address-types
    value: "InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP"
  - name: profiling
    value: "False"
  - name: request-timeout
    value: "1m0s"
  - name: enable-aggregator-routing
    value: "False"
  - name: service-account-lookup
    value: "True"
  - name: storage-backend
    value: "etcd3"
  - name: allow-privileged
    value: "true"
  - name: audit-policy-file
    value: "/etc/kubernetes/audit-policy/apiserver-audit-policy.yaml"
  - name: audit-log-path
    value: "/var/log/audit/kube-apiserver-audit.log"
  - name: audit-log-maxage
    value: "30"
  - name: audit-log-maxbackup
    value: "10"
  - name: audit-log-maxsize
    value: "100"
  - name: "default-not-ready-toleration-seconds"
    value: "10"
  - name: "default-unreachable-toleration-seconds"
    value: "10"
  - name: tls-min-version
    value: "VersionTLS12"
  - name: tls-cipher-suites
    value: "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
  - name: event-ttl
    value: "0h10m0s"
  extraVolumes:
  - name: audit-policy
    hostPath: /etc/kubernetes/audit-policy
    mountPath: /etc/kubernetes/audit-policy
  - name: audit-logs
    hostPath: /var/log/kubernetes/audit
    mountPath: /var/log/audit
    readOnly: false
  - name: usr-share-ca-certificates
    hostPath: /usr/share/ca-certificates
    mountPath: /usr/share/ca-certificates
    readOnly: true
  certSANs:
  - "kubernetes"
  - "kubernetes.default"
  - "kubernetes.default.svc"
  - "kubernetes.default.svc.test.teamlead"
  - "157.180.16.146"
  - "localhost"
  - "127.0.0.1"
  - "test-master-1"
  - "lb-apiserver.kubernetes.local"
  - "157.180.21.176"
  - "192.168.1.2"
  - "test-master-1.test.teamlead"
controllerManager:
  extraArgs:
  - name: node-monitor-grace-period
    value: "40s"
  - name: node-monitor-period
    value: "5s"
  - name: cluster-cidr
    value: "192.168.1.64/26"
  - name: service-cluster-ip-range
    value: "192.168.1.32/27"
  - name: node-cidr-mask-size
    value: "26"
  - name: profiling
    value: "False"
  - name: terminated-pod-gc-threshold
    value: "12500"
  - name: bind-address
    value: "0.0.0.0"
  - name: leader-elect-lease-duration
    value: "15s"
  - name: leader-elect-renew-deadline
    value: "10s"
  - name: "node-monitor-grace-period"
    value: "20s"
  - name: "allocate-node-cidrs"
    value: "True"
  - name: "node-cidr-mask-size-ipv4"
    value: "26"
  - name: configure-cloud-routes
    value: "false"
  - name: tls-min-version
    value: "VersionTLS12"
  - name: tls-cipher-suites
    value: "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
scheduler:
  extraArgs:
  - name: bind-address
    value: "0.0.0.0"
  - name: config
    value: "/etc/kubernetes/kubescheduler-config.yaml"
  - name: profiling
    value: "False"
  - name: tls-min-version
    value: "VersionTLS12"
  - name: tls-cipher-suites
    value: "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
  extraVolumes:
  - name: kubescheduler-config
    hostPath: /etc/kubernetes/kubescheduler-config.yaml
    mountPath: /etc/kubernetes/kubescheduler-config.yaml
    readOnly: true
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes:
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig:
  qps: 5
clusterCIDR: "192.168.1.64/26"
configSyncPeriod: 15m0s
conntrack:
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: False
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: "test-master-1"
iptables:
  masqueradeAll: False
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  excludeCIDRs: []
  minSyncPeriod: 0s
  scheduler: rr
  syncPeriod: 30s
  strictARP: False
  tcpTimeout: 0s
  tcpFinTimeout: 0s
  udpTimeout: 0s
metricsBindAddress: 127.0.0.1:10249
mode: ipvs
nodePortAddresses: []
oomScoreAdj: -999
portRange:
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
clusterDNS:
- 169.254.25.10

but in manifests eg api-server default-not-ready-toleration-second, default-unreachable-toleration-seconds

api server
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.2:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.2
    - --allow-privileged=true
    - --anonymous-auth=True
    - --apiserver-count=1
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    - --audit-log-path=/var/log/audit/kube-apiserver-audit.log
    - --audit-policy-file=/etc/kubernetes/audit-policy/apiserver-audit-policy.yaml
    - --authorization-mode=Node,RBAC
    - --bind-address=0.0.0.0
    - --client-ca-file=/etc/kubernetes/ssl/ca.crt
    - --default-not-ready-toleration-seconds=10
    - --default-not-ready-toleration-seconds=300
    - --default-unreachable-toleration-seconds=10
    - --default-unreachable-toleration-seconds=300
    - --enable-admission-plugins=NodeRestriction
    - --enable-aggregator-routing=False
    - --enable-bootstrap-token-auth=true
    - --endpoint-reconciler-type=lease
    - --etcd-cafile=/etc/kubernetes/ssl/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/ssl/apiserver-etcd-client.crt
    - --etcd-compaction-interval=5m0s
    - --etcd-keyfile=/etc/kubernetes/ssl/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --event-ttl=0h10m0s
    - --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP
    - --profiling=False
    - --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key
    - --request-timeout=1m0s
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.test.teamlead
    - --service-account-key-file=/etc/kubernetes/ssl/sa.pub
    - --service-account-lookup=True
    - --service-account-signing-key-file=/etc/kubernetes/ssl/sa.key
    - --service-cluster-ip-range=192.168.1.32/27
    - --service-node-port-range=30000-32767
    - --storage-backend=etcd3
    - --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt
    - --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - --tls-min-version=VersionTLS12
    - --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.31.4
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.1.2
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.1.2
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.1.2
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /var/log/audit
      name: audit-logs
    - mountPath: /etc/kubernetes/audit-policy
      name: audit-policy
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/ssl
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
  hostNetwork: true
  priority: 2000001000
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /var/log/kubernetes/audit
      type: ""
    name: audit-logs
  - hostPath:
      path: /etc/kubernetes/audit-policy
      type: ""
    name: audit-policy
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/ssl
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: ""
    name: usr-share-ca-certificates
status: {}

In case "last value applied" - this value default-unreachable-toleration-seconds=300, instead custom extra-arg

in controller-manager similar

- --node-monitor-grace-period=20s
- --node-monitor-grace-period=40s

It is bad

tico88612 · 2025-03-15T16:29:49Z

@tagurus Ok, I understand this problem. I will report it to Kubeadm first to make sure it is intentional or unintentional.
And then we'll decide whether we want to fix it or not.

tico88612 · 2025-03-18T07:06:51Z

@VannTen according to the reply, extraArgs will be in ascending order if there are duplicate names.
K8s components don't support flags where you could pass a flag multiple times but the order matters for priority.

FYI: kubernetes/kubeadm#3173 (comment)

Maybe we could try to use kubeadm patch for resolve this situation.

tico88612 · 2025-03-18T07:08:35Z

/assign

tagurus · 2025-03-19T08:00:09Z

@tico88612

well, i have read this article. It turns out that biggest value has priority in flags.

In this scenario with kubespray, we can use default variable name, eg "kube_apiserver_pod_eviction_not_ready_timeout_seconds" instead extra arg, for avoid problem as temporary measure.

better make note in doc section

VannTen · 2025-03-19T08:15:43Z

Maybe we could try to use kubeadm patch for resolve this situation.

That can be a workaround (and they can defined in Kubespray inventory now) but I don't think that's a clean solution. IMO the long term solution could be instead of direct Jinja rendering define a intermediate variable like `kubeadm_*flags` , and then do something like `kubeam_*flags | combine(kubeadm_*extra_flags) | to_yaml`

tagurus added the kind/bug label Mar 5, 2025

k8s-ci-robot added the triage/accepted label Mar 6, 2025

tico88612 mentioned this issue Mar 15, 2025

v1beta4 extraArgs duplicate values policy kubernetes/kubeadm#3173

Closed

k8s-ci-robot assigned tico88612 Mar 18, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unpredictable extra args #12021

unpredictable extra args #12021

tagurus commented Mar 5, 2025

tico88612 commented Mar 6, 2025

VannTen commented Mar 6, 2025

tico88612 commented Mar 7, 2025

tagurus commented Mar 13, 2025

tico88612 commented Mar 15, 2025

tico88612 commented Mar 18, 2025

tico88612 commented Mar 18, 2025

tagurus commented Mar 19, 2025

VannTen commented Mar 19, 2025 via email

unpredictable extra args #12021

unpredictable extra args #12021

Comments

tagurus commented Mar 5, 2025

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

OS

Version of Ansible

Version of Python

Version of Kubespray (commit)

Network plugin used

Full inventory with variables

Command used to invoke ansible

Output of ansible run

Anything else we need to know

tico88612 commented Mar 6, 2025

VannTen commented Mar 6, 2025

tico88612 commented Mar 7, 2025

tagurus commented Mar 13, 2025

tico88612 commented Mar 15, 2025

tico88612 commented Mar 18, 2025

tico88612 commented Mar 18, 2025

tagurus commented Mar 19, 2025

VannTen commented Mar 19, 2025 via email