eks-multiple-clusters-KUBECONFIG

Many people complain accidentally executing commands on the wrong cluster. In most cases, this happens because you’re in the directory containing manifests for cluster-1, but you apply it to cluster-2 as that was the active context.

To prevent this scenario, you can use direnv tool which lets you automatically set environment variables based on the directory tree you’re in. This way, when navigate to the directory of cluster-1 manifests, direnv will set $KUBECONFIG to cluster-1 and prevent the disaster.





# https://ahmet.im/blog/mastering-kubeconfig/

# https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

# https://stackoverflow.com/questions/63629990/set-kubeconfig-environment-variable-dynamically

# https://direnv.net/

KUBECONFIG



Mastering the KUBECONFIG file
Ahmet Alp Balkan published on 12 February 2019
There is a kubeconfig file behind every working kubectl command. This file typically lives at $HOME/.kube/config. Having written kubectx, I’ve interacted with kubeconfigs long enough to write some tips about how to deal with them.

If you’re not familiar with kubeconfig files, read the documentation first.

Tip 1: Know the kubeconfig precedence
If you’re using kubectl, here’s the preference that takes effect while determining which kubeconfig file is used.

use --kubeconfig flag, if specified
use KUBECONFIG environment variable, if specified
use $HOME/.kube/config file
With this, you can easily override kubeconfig file you use per-kubectl command:

kubectl get pods --kubeconfig=file1
kubectl get pods --kubeconfig=file2

# or:

KUBECONFIG=file1 kubectl get pods
KUBECONFIG=file2 kubectl get pods
Although this precedence list not officially specified in the documentation it is codified here. If you’re developing client tools for Kubernetes, you should consider using cli-runtime library which will bring the standard --kubeconfig flag and $KUBECONFIG detection to your program.

Tip 2: Using multiple kubeconfigs at once
Sometimes you have a bunch of small kubeconfig files (e.g. one per cluster) but you want to use them all at once, with tools like kubectl or kubectx that work with multiple contexts at once.

To do that, you need a “merged” kubeconfig file. Tip #3 explains how you can merge the kubeconfigs into a single file, but you can also merge them in-memory.

By specifying multiple files in KUBECONFIG environment variable, you can temporarily stitch kubeconfig files together and use them all in kubectl.

export KUBECONFIG=file1:file2
kubectl get pods --context=cluster-1
kubectl get pods --context=cluster-2
Tip 3: Merging kubeconfig files
Since kubeconfig files are structured YAML files, you can’t just append them to get one big kubeconfig file, but kubectl can help you merge these files:

KUBECONFIG=file1:file2:file3 kubectl config view --merge --flatten > out.txt
Tip 4: Extracting a context from a kubeconfig file
Let’s say you followed Tip #4 and have a merged kubeconfig file. Now you want to extract a cluster’s information to a portable kubeconfig file that only has the parts you need to connect to that cluster. Run:

KUBECONFIG=in.txt kubectl config view \
    --minify --flatten --context=context-1 > out.txt
In this command, we extract data about context-1 from in.txt to out.txt. The --minify flag allows us to extract only info about that context, and the --flatten flag allows us to keep the credentials unredacted.

Tip 5: Use kubectl without a kubeconfig
kubectl command offers a bunch of command line flags (run kubectl options to see) that allow you to override pretty much every piece of information it reads from a kubeconfig file.

For example, if I wanted to use my local Docker for Mac cluster without a kubeconfig file, I would first look at kubectl config view --context=docker-for-desktop and extract the information to the following flags:

KUBECONFIG= kubectl get nodes \
    --server https://localhost:6443 \
    --user docker-for-desktop \
    --client-certificate my.cert \
    --client-key my.key \
    --insecure-skip-tls-verify
It gets tricky (and impossible) to use as your kubeconfig gets complicated, like when you have an auth plugin with various fields you can’t configure via a CLI.

Don’t forget to set your $KUBECONFIG to empty (as seen above) to prevent accidentally picking up some settings from the ~/.kube/config file.

(Bonus) Tip 6: Auto-$KUBECONFIG based on directory
Many people complain accidentally executing commands on the wrong cluster. In most cases, this happens because you’re in the directory containing manifests for cluster-1, but you apply it to cluster-2 as that was the active context.

To prevent this scenario, you can use direnv tool which lets you automatically set environment variables based on the directory tree you’re in. This way, when navigate to the directory of cluster-1 manifests, direnv will set $KUBECONFIG to cluster-1 and prevent the disaster.

(Bonus) Tip 7: Know which context you’re pointing at
This is not directly about munging KUBECONFIG files, but kube-ps1 (which I proudly advised on its early development) that lets you see the current namespace/context you’re on in your bash/zsh prompt.

$ kubeon
{⎈ |N/A:N/A} $ export KUBECONFIG=f2
{⎈ |docker-for-desktop:default} $ export KUBECONFIG=f1
{⎈ |gke_ahmetb-samples-playground_us-central1-b_demo:kube-system} $ kubens default
Active namespace is "default".
{⎈ |gke_ahmetb-samples-playground_us-central1-b_demo:default} $ kubeoff
$
You can turn it on/off per-shell, or globally with -g flag to kubeon/kubeoff.

(Bonus) Tip 8: Save GKE contexts to separate files
When you create a GKE cluster (or retrieve its credentials) through the gcloud command, it normally modifies your default ~/.kube/config file. However, you can set $KUBECONFIG for gcloud to save cluster credentials to a file:

KUBECONFIG=c1.yaml gcloud container clusters get-credentials "cluster-1"