Certificate Chain Validation Fails When Running Fabric8 Kubernetes Client Inside a Pod

[fengyinqiao]

Describe the bug

Description

I am encountering an issue when trying to connect to the Kubernetes cluster using the Fabric8 Kubernetes client from within a pod. The connection fails with the error: "Certificate chain is not valid".

Interestingly, when I use kubectl with the same kubeconfig file (--kubeconfig option), I am able to connect without any issues. This problem only occurs when using Fabric8 within the pod.

Environment Details

• Java Version: 17
• Spring Boot Version: 2.5.15
• Fabric8 Kubernetes Client Version: 6.9.2
• Pod Container Information:
  • OS: CentOS Linux 7 (Core)
  • Kernel Version: 4.19.91-26.6.al7.x86_64
  • Details from /etc/os-release:

    NAME="CentOS Linux"
    VERSION="7 (Core)"
    ID="centos"
    ID_LIKE="rhel fedora"
    VERSION_ID="7"
    PRETTY_NAME="CentOS Linux 7 (Core)"
    ANSI_COLOR="0;31"
    CPE_NAME="cpe:/o:centos:centos:7"
    HOME_URL="https://www.centos.org/"
    BUG_REPORT_URL="https://bugs.centos.org/"
    

Issue Details

• kubectl works without issues: Using kubectl --kubeconfig inside the same pod works as expected, and I can connect to the Kubernetes API server without any certificate issues.
• Fabric8 client fails with certificate error: When trying to connect using the Fabric8 Kubernetes client, it fails with the "Certificate chain is not valid" error.

Steps to Reproduce

1. Run a pod using CentOS 7 with the following kernel version: 4.19.91-26.6.al7.x86_64.
2. Inside the pod, try connecting to the Kubernetes cluster using kubectl --kubeconfig. The connection should work.
3. Use the Fabric8 Kubernetes client with the same configuration, which results in a certificate chain validation error.

Expected Behavior

The Fabric8 Kubernetes client should successfully establish a connection to the Kubernetes API server, similar to the kubectl command, without any certificate validation issues.

Actual Behavior

The Fabric8 Kubernetes client fails with a certificate chain validation error.

Fabric8 Kubernetes Client version

6.9.2

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

v1.26.15-aliyun.1

Environment

Linux

Fabric8 Kubernetes Client Code and Logs

code is as below:

private void initializeClient(File file) {
        try {
            String kubeConfigPath = file.getAbsolutePath();
            Config config = Config.fromKubeconfig(Files.readString(Path.of(kubeConfigPath)));
            KubernetesClient client = new KubernetesClientBuilder().withConfig(config).build();
            clients.put(file.getName(), client);
            informerFactories.put(file.getName(), initializeInformers(client));
            logger.info("Kubernetes client initialized for config: {}", file.getName());
        } catch (Exception e) {
            logger.error("Failed to initialize Kubernetes client for config: {}", file.getName());
            throw new RuntimeException("Failed to initialize Kubernetes client for config: " + file.getName(), e);
        }
    }

Caused by: java.lang.RuntimeException: Failed to initialize Kubernetes client for config: test-kubeconfig
        at com.test.k8s.KubernetesClientsManager.initializeClient(KubernetesClientsManager.java:91)
        at com.test.k8s.KubernetesClientsManager.lambda$new$0(KubernetesClientsManager.java:38)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129)
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:191)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newBuilder(OkHttpClientFactory.java:82)
        at io.fabric8.kubernetes.client.okhttp.OkHttpClientFactory.newBuilder(OkHttpClientFactory.java:29)
        at io.fabric8.kubernetes.client.KubernetesClientBuilder.getHttpClient(KubernetesClientBuilder.java:90)
        at io.fabric8.kubernetes.client.KubernetesClientBuilder.build(KubernetesClientBuilder.java:79)
        at com.test.k8s.KubernetesClientsManager.initializeClient(KubernetesClientsManager.java:85)
        ... 5 common frames omitted
Caused by: java.security.KeyStoreException: Certificate chain is not valid
        at java.base/sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:646)
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:589)
        at java.base/sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:112)
        at java.base/java.security.KeyStore.setKeyEntry(KeyStore.java:1167)
        at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:159)
        at io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:288)
        at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:188)
        at io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:177)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.applyCommonConfiguration(HttpClientUtils.java:188)
        ... 10 common frames omitted

Additional context

This issue might be related to differences in how kubectl and Fabric8 handle the certificate chain or the CA bundle.

Could you please help to identify if there is an issue with the way Fabric8 is validating the certificate chain in this particular setup or suggest any workarounds that could be used here?

Thank you for your support.

No response

[rohanKanojia]

Not 100% sure if it's related but do you have optional bouncycastle dependencies added to your project?

        <dependency>
            <groupId>org.bouncycastle</groupId>
            <artifactId>bcpkix-jdk18on</artifactId>
        </dependency>
        <dependency>
            <groupId>org.bouncycastle</groupId>
            <artifactId>bcprov-jdk18on</artifactId>
        </dependency>

Is it possible to try it with a newer version of KubernetesClient?

[fengyinqiao]

After decoding the client-certificate-data in kubeconfig via base64, I discovered two identical certificates. Through various debugging and analysis, I found that the JDK’s certificate verification logic requires that: the issuer of the first certificate matches the subject of the second certificate, the issuer of the second certificate matches the subject of the third certificate, and so on, verifying up to the root certificate. Since the two identical certificates did not satisfy this verification logic, an error was thrown: "Certificate chain is not valid." I then removed one of the duplicate certificates and, after re-verification, the issue was resolved. I have already contacted my cloud provider regarding this certificate issue. Many thanks to the community for your attention.

sun.security.pkcs12.PKCS12KeyStore#validateChain from JDK17:

  /*
   * Validate Certificate Chain
   */
  private boolean validateChain(Certificate[] certChain)
  {
      for (int i = 0; i < certChain.length-1; i++) {
          X500Principal issuerDN =
              ((X509Certificate)certChain[i]).getIssuerX500Principal();
          X500Principal subjectDN =
              ((X509Certificate)certChain[i+1]).getSubjectX500Principal();
          if (!(issuerDN.equals(subjectDN)))
              return false;
      }

      // Check for loops in the chain. If there are repeated certs,
      // the Set of certs in the chain will contain fewer certs than
      // the chain
      Set<Certificate> set = new HashSet<>(Arrays.asList(certChain));
      return set.size() == certChain.length;
  }

my crt decoded via base64:

$ echo $CRT | base64 -d
-----BEGIN CERTIFICATE-----
balabala
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
balabala
-----END CERTIFICATE-----

[fengyinqiao]

Interestingly, even with duplicate certificates, verification using kubectl or client-go presented no issues. I don't fully understand the reasoning behind this specific verification logic in the JDK.