merge multiple files listed in KUBECONFIG environment var

[adietish]

Is your enhancement related to a problem? Please describe

One can provide kubectl with multiple kube config files via the KUBECONFIG env variable. Kubectl would merge those and let the user use them.
ex.

• export KUBECONFIG=${HOME}/.kube/config:${HOME}/.kube/minikube.yaml:${HOME}/.kube/sandbox.yaml
• empty ~/.kube/config with only current-context pointing to a context that's defined in further file

apiVersion: v1
kind: Config
clusters: null
contexts: null
current-context: minikube
preferences: {}
users: null

• ~/.kube/minikube.yml

apiVersion: v1
kind: Config
clusters:
  - cluster:
      certificate-authority: /Users/adietish/.minikube/ca.crt
      extensions:
        - extension:
            last-update: Wed, 07 Aug 2024 11:05:40 CEST
            provider: minikube.sigs.k8s.io
            version: v1.33.1
          name: cluster_info
      server: https://127.0.0.1:39093
    name: minikube
contexts:
  - context:
      cluster: minikube
      extensions:
        - extension:
            last-update: Wed, 07 Aug 2024 11:05:40 CEST
            provider: minikube.sigs.k8s.io
            version: v1.33.1
          name: context_info
      namespace: default
      user: minikube
    name: minikube
users:
  - name: minikube
    user:
      client-certificate: /Users/adietish/.minikube/profiles/minikube/client.crt
      client-key: /Users/adietish/.minikube/profiles/minikube/client.key

• ~/.kube/sandbox.yml

apiVersion: v1
kind: Config
clusters:
  - cluster:
      server: https://XXXX.openshiftapps.com:6443
    name: api-sandbox-m3-1530-p1-openshiftapps-com:6443
users:
  - name: adietish/XXXX-openshiftapps-com:6443
    user:
      token: sha256~YYYYYYYY
contexts:
  - context:
      cluster: XXXX-openshiftapps-com:6443
      namespace: adietish-dev
      user: adietish/XXXX-openshiftapps-com:6443
    name: adietish-dev/XXXX-openshiftapps-com:6443/adietish 

In kubectl I can then have config pointing to minikube or sandbox, both would work. In kubernetes-client, only the 1st file is used which then fails because it is missing url, user, etc.

Describe the solution you'd like

It would be great if kubernetes-client could merge these files into one config like kubectl does.

Describe alternatives you've considered

No response

Additional context

No response

[adietish]

I am working on a patch as this configuration/setup looks rather common when a user has lots of clusters. It allows a user to split up the configuration into smaller chunks rather to have a single big kube config.

[adietish]

Patch seems rather simple (just looking, didnt code so far):
call io.fabric8.kubernetes.client.Config#mergeKubeConfigContents for each file that's present in the system variable, making sure that overrides/merges happen like in kubectl

[adietish]

For my own documentation:
if the KUBECONFIG env variable is set (~/.kube/config:<additional file>:<additional file>), kubectl is using the first file in the list and adds

• contexts
• clusters
• users

that are found in the additional files.
The code for it is in client_go

[rohanKanojia]

if the KUBECONFIG env variable is set (~/.kube/config:<additional file>:<additional file>), kubectl is using the first file in the list and adds

@adietish : I'm sorry for replying late, after reading your comment I realized we had merged something similar a few years ago. However, I'm not sure whether it's completely aligned with kubectl behavior or not.

#1306

[adietish]

@rohanKanojia thx for the pointer, I had already looked into this. This PR only takes the 1st of the given files into account. All further ones are ignored. See here: https://github.com/fabric8io/kubernetes-client/pull/1306/files#diff-ee58b9555742a6e84ec59a74115410be46af05081afae5cf79a4516283ce37c7R444
There's a 2nd issue which add details to the logging warning but didnt' change the handling: #2174
(see https://github.com/fabric8io/kubernetes-client/pull/2181/files#diff-ee58b9555742a6e84ec59a74115410be46af05081afae5cf79a4516283ce37c7R477)

LOGGER.warn("Found multiple Kubernetes config files [{}], using the first one: [{}]. 
    If not desired file, please change it by doing `export KUBECONFIG=/path/to/kubeconfig` on Unix systems 
    or `$Env:KUBECONFIG=/path/to/kubeconfig` on Windows.", fileNames, fileNames[0]);

What I want to improve is to allow several files to be used. In quarkusio/quarkus#8946 (comment) @manusa brought up how such a setup would work:
Reading the kubectl docs, testing and reading of kubectl code suggests the following behaviour:

• only the first file sets the current-context. None of the additional files in KUBECONFIG env var can set the current-context. (tested)
• all the additional files listed in KUBECONFIG contribute contexts, clusters and users .

redhat-developer/intellij-kubernetes#779 provides an exemplary setup where the 1st file sets the current-context but has no contexts, nor clusters, nor users. Then there are separate files for each cluster (ex. minikube.yml, sandbox.yml, etc.) which contribute a context, a cluster and a user.

[manusa]

Reading the kubectl docs, testing and reading of kubectl code suggests the following behaviour:

* only the first file sets the `current-context`. None of the additional files in `KUBECONFIG` env var can set the `current-context`.  (tested)

* all the additional files listed in `KUBECONFIG` contribute `contexts`, `clusters` and `users` .

This seems fine.

Implementation should be quite easy for merging.
However, I think there are several places where the underlying configuration is updated (token refresh, etc.). These might be the tricky parts, since we'll probably need to traverse all configs until the config containing the token is found and updated.

[adietish]

@manusa thx for the pointer, I found

kubernetes-client/kubernetes-client-api/src/main/java/io/fabric8/kubernetes/client/utils/OpenIDConnectionUtils.java

Lines 188 to 225 in 342fb8a

	public static OAuthToken persistOAuthToken(Config currentConfig, OAuthToken oAuthToken, String token) {
	final Map<String, String> authProviderConfig = new HashMap<>();
	if (oAuthToken != null) {
	authProviderConfig.put(ID_TOKEN_KUBECONFIG, oAuthToken.idToken);
	authProviderConfig.put(REFRESH_TOKEN_KUBECONFIG, oAuthToken.refreshToken);
	// Persist in memory
	Optional.of(currentConfig).map(Config::getAuthProvider).map(AuthProviderConfig::getConfig)
	.ifPresent(c -> c.putAll(authProviderConfig));
	}
	// Persist in file
	if (currentConfig.getFile() != null && currentConfig.getCurrentContext() != null) {
	try {
	final io.fabric8.kubernetes.api.model.Config kubeConfig = KubeConfigUtils.parseConfig(currentConfig.getFile());
	final String userName = currentConfig.getCurrentContext().getContext().getUser();
	NamedAuthInfo namedAuthInfo = kubeConfig.getUsers().stream().filter(n -> n.getName().equals(userName)).findFirst()
	.orElseGet(() -> {
	NamedAuthInfo result = new NamedAuthInfo(userName, new AuthInfo());
	kubeConfig.getUsers().add(result);
	return result;
	});
	if (namedAuthInfo.getUser() == null) {
	namedAuthInfo.setUser(new AuthInfo());
	}
	if (namedAuthInfo.getUser().getAuthProvider() == null) {
	namedAuthInfo.getUser().setAuthProvider(new AuthProviderConfig());
	}
	namedAuthInfo.getUser().getAuthProvider().getConfig().putAll(authProviderConfig);
	if (Utils.isNotNullOrEmpty(token)) {
	namedAuthInfo.getUser().setToken(token);
	}
	KubeConfigUtils.persistKubeConfigIntoFile(kubeConfig, currentConfig.getFile().getAbsolutePath());
	} catch (IOException ex) {
	LOGGER.warn("oidc: failure while persisting new tokens into KUBECONFIG", ex);
	}
	}
	return oAuthToken;
	}

Is this what you were referring to?

[adietish]

I have to specify my suggestion in #6240 (comment) according to the Kubectl docs:
kubeconfigs listed in the KUBECONFIG env var are fully merged while values in a latter file will not override a prior one. First in wins.

Never change the value or map key. Example: Preserve the context of the first file to set current-context. Example: If two files specify a red-user, use only values from the first file's red-user. Even if the second file has non-conflicting entries under red-user, discard them.

And then, when using kubectl --kubeconf <file> only a single file is allowed (mo merging happens).

If the --kubeconfig flag is set, use only the specified file. Do not merge. Only one instance of this flag is allowed.