Create CA/Issuer certificates with openssl

[anorm]

In the documentation, it says it's possible to generate the necessary CA and issuer keys and certificates using openssl. I've given it a shot but after installing, stuff don't work as expected.

I'm generating keys and certificates like this:

#!/bin/bash
# Create CA private key
if [ ! -f ca-private.pem ]; then
    openssl ecparam -name prime256v1 -genkey -noout -out ca-private.pem
fi
# Create CA public key
openssl ec -in ca-private.pem -pubout -out ca-public.pem
# Create self signed CA certificate
openssl req -x509 -new -key ca-private.pem -days 365 -out ca.crt -subj "/CN=root.linkerd.cluster.local"


# Create issuer private key
if [ ! -f issuer-private.pem ]; then
    openssl ecparam -name prime256v1 -genkey -noout -out issuer-private.pem
fi
# Create issuer public key
openssl ec -in issuer-private.pem -pubout -out issuer-public.pem
# Create certificate signing request
openssl req -new -key issuer-private.pem -out issuer.csr -subj "/CN=issuer.linkerd.cluster.local"
# Create issuer cert by signing request
openssl x509 -req -in issuer.csr -days 180 -CA ca.crt -CAkey ca-private.pem -CAcreateserial -out issuer.crt
rm issuer.csr

The resulting issuer certificate looks good to me:

$ openssl x509 -in issuer.crt -text -noout
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            22:8b:fc:fa:2a:a4:8d:19:db:b3:ad:91:05:83:f5:a8:ab:14:98:d7
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = root.linkerd.cluster.local
        Validity
            Not Before: Sep  4 21:39:53 2021 GMT
            Not After : Mar  3 21:39:53 2022 GMT
        Subject: CN = issuer.linkerd.cluster.local
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:18:59:58:57:79:45:92:65:06:a4:1b:d0:3e:ee:
                    dc:3e:a9:f8:a1:16:b1:4c:4b:fc:90:c4:91:16:6a:
                    4f:4f:a9:e8:b0:e9:02:d2:10:b8:06:21:a7:32:a9:
                    db:86:ee:3f:77:be:cf:f9:16:05:8f:b2:2b:12:58:
                    7a:e1:12:97:10
                ASN1 OID: prime256v1
                NIST CURVE: P-256
    Signature Algorithm: ecdsa-with-SHA256
         30:44:02:20:0b:18:e7:d1:c5:09:6a:71:1c:5b:dd:05:46:5c:
         dd:4f:ce:59:24:7f:0a:1a:a3:52:c6:9b:6b:88:59:b9:30:18:
         02:20:57:3f:a5:29:6d:77:ce:fa:31:20:94:86:16:33:d0:2d:
         23:50:c6:1f:0c:24:85:d8:6d:e7:ff:5f:cf:d3:77:af

Then I install linkerd2 from stable using helm, filling in the ca.crt, issuer.crt and issuer.key in the values.yml file as such:

identityTrustAnchorsPEM: |
  -----BEGIN CERTIFICATE-----
  MIIBnjCCAUWgAwIBAgIUONTiBhpLpJ2nyGNVrlMDzDE89fkwCgYIKoZIzj0EAwIw
  JTEjMCEGA1UEAwwacm9vdC5saW5rZXJkLmNsdXN0ZXIubG9jYWwwHhcNMjEwOTA0
  MjEzOTUzWhcNMjIwOTA0MjEzOTUzWjAlMSMwIQYDVQQDDBpyb290LmxpbmtlcmQu
  Y2x1c3Rlci5sb2NhbDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABK3dfyoYY7Ws
  pNZXW299k6sZBvAk+yQ7YDhbmByUwwKEQxDdSu197OY0xBk2YPUWu6lAm1dR6lJb
  veRAjVTOgsWjUzBRMB0GA1UdDgQWBBS5X6Pnw2pEMcAphFNnEbCnJlJOmjAfBgNV
  HSMEGDAWgBS5X6Pnw2pEMcAphFNnEbCnJlJOmjAPBgNVHRMBAf8EBTADAQH/MAoG
  CCqGSM49BAMCA0cAMEQCIBZGA3c1grQm3vAT+9gKl+eDJpumagRwKmgWWb4xSFDL
  AiBY4btsvLHbCoah5jwFdnk8vfV/NKv0pATBUrlpvXKeyw==
  -----END CERTIFICATE-----
identity:
  issuer:
    crtExpiry: "2022-03-03T21:13:16Z"
    tls:
      crtPEM: |
        -----BEGIN CERTIFICATE-----
        MIIBRTCB7QIUIov8+iqkjRnbs62RBYP1qKsUmNcwCgYIKoZIzj0EAwIwJTEjMCEG
        A1UEAwwacm9vdC5saW5rZXJkLmNsdXN0ZXIubG9jYWwwHhcNMjEwOTA0MjEzOTUz
        WhcNMjIwMzAzMjEzOTUzWjAnMSUwIwYDVQQDDBxpc3N1ZXIubGlua2VyZC5jbHVz
        dGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEGFlYV3lFkmUGpBvQ
        Pu7cPqn4oRaxTEv8kMSRFmpPT6nosOkC0hC4BiGnMqnbhu4/d77P+RYFj7IrElh6
        4RKXEDAKBggqhkjOPQQDAgNHADBEAiALGOfRxQlqcRxb3QVGXN1Pzlkkfwoao1LG
        m2uIWbkwGAIgVz+lKW13zvoxIJSGFjPQLSNQxh8MJIXYbef/X8/Td68=
        -----END CERTIFICATE-----
      keyPEM: |
        -----BEGIN EC PRIVATE KEY-----
        MHcCAQEEIK7uXEDht//8J7rEkI7ib7wsO0n73vQF3iYVGXswKGChoAoGCCqGSM49
        AwEHoUQDQgAEGFlYV3lFkmUGpBvQPu7cPqn4oRaxTEv8kMSRFmpPT6nosOkC0hC4
        BiGnMqnbhu4/d77P+RYFj7IrElh64RKXEA==
        -----END EC PRIVATE KEY-----

After helm completes installing the release, I can see the linkerd-identity-issuer secret in the linkerd namespace holding the certificate and key as expected.

When running linkerd check --proxy it stops at pod/linkerd-controller-59fb4748b-crrds container public-api is not ready

Checking the logs of the controller pod yields:

kubectl logs -n linkerd linkerd-controller-59fb4748b-crrds linkerd-proxy
...
[   897.572378s] ERROR ThreadId(01) daemon:identity: rustls::session: TLS alert received: Message {
    typ: Alert,
    version: TLSv1_2,
    payload: Alert(
        AlertMessagePayload {
            level: Fatal,
            description: AccessDenied,
        },
    ),
}
...

The error message seems certificate related, so I'm sure I did something wrong. But what?

---

Continuing my investigation, I uninstalled the linkerd helm release and installed using linkerd install | kubectl apply -f -. The logs of the linkerd-proxy in the linkerd-controller pod now yields:

...
[    27.343267s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-controller.linkerd.serviceaccount.identity.linkerd.cluster.local
...

So this is what the proxy was unable to achieve when installed using helm.

But I'm no closer to solving my problems.

[adleong]

One thing I noticed when trying to follow the steps you took with openssl is that the certs it generates do not have the CA flag. When trying to install with these certs and the Linkerd CLI, I get the following error:

linkerd install \
  --identity-trust-anchors-file ca-ssl.crt \
  --identity-issuer-certificate-file issuer.crt \
  --identity-issuer-key-file issuer-key-ssl.pem \
  | kubectl apply -f -
Error: failed to validate issuer credentials: issuer cert is not a CA

More generally, I'd recommend following the instructions in the docs which use step. If you prefer to use openssl directly, I'd recommend comparing the openssl generated certs to the step generated ones to see exactly how they differ.

[anorm]

This was indeed the problem.

Furthermore, there's a known bug in openssl v1.1.1. From the man page

$ man openssl-x509
...
BUGS
       Extensions in certificates are not transferred to certificate requests and vice versa.
...

So, even though I tried adding the extension to the signing request, the resulting certificate did not have the CA.

A working script for generating the certificates looks like this:

#!/bin/bash
# Create CA private key
if [ ! -f ca-private.pem ]; then
    openssl ecparam -name prime256v1 -genkey -noout -out ca-private.pem
fi
# Create CA public key
openssl ec -in ca-private.pem -pubout -out ca-public.pem
# Create self signed CA certificate
openssl req -x509 -new -key ca-private.pem -days 365 -out ca.crt -subj "/CN=root.linkerd.cluster.local"

# Create issuer private key
if [ ! -f issuer-private.pem ]; then
    openssl ecparam -name prime256v1 -genkey -noout -out issuer-private.pem
fi
# Create issuer public key
openssl ec -in issuer-private.pem -pubout -out issuer-public.pem
# Create certificate signing request (BUG: the extension added here will be ignored by the signing)
openssl req -new -key issuer-private.pem -out issuer.csr -subj "/CN=identity.linkerd.cluster.local" \
    -addext basicConstraints=critical,CA:TRUE
# Create issuer cert by signing request
openssl x509 \
    -extfile /etc/ssl/openssl.cnf \
    -extensions v3_ca \
    -req \
    -in issuer.csr \
    -days 180 \
    -CA ca.crt \
    -CAkey ca-private.pem \
    -CAcreateserial \
    -extensions v3_ca \
    -out issuer.crt
rm issuer.csr

Also, I'm using openssl over snap because I wanted to learn more about certificate trust chains etc.

Thanks for your answer :-)

[adleong]

awesome! happy to help!

[wibed]

for anyone stumbling on this,

• related issue on the bug
  Why does the x509 command not copy extension in certificate request? openssl/openssl#10458

and heres an improved version of the script:

#!/bin/bash
# Create CA private key
if [ ! -f ca-private.pem ]; then
    openssl ecparam -name prime256v1 -genkey -noout -out ca-private.pem
fi
# Create CA public key
openssl ec -in ca-private.pem -pubout -out ca-public.pem
# Create self signed CA certificate
openssl req -x509 -new -key ca-private.pem -days 365 -out ca.crt -subj "/CN=root.linkerd.cluster.local"

# Create issuer private key
if [ ! -f issuer-private.pem ]; then
    openssl ecparam -name prime256v1 -genkey -noout -out issuer-private.pem
fi
# Create issuer public key
openssl ec -in issuer-private.pem -pubout -out issuer-public.pem
# Create certificate signing request (BUG: the extension added here will be ignored by the signing)
openssl req -new -key issuer-private.pem -out issuer.csr -subj "/CN=identity.linkerd.cluster.local" \
    -addext basicConstraints=critical,CA:TRUE
# Create issuer cert by signing request
openssl x509 \
    -req \
    -in issuer.csr \
    -days 180 \
    -CA ca.crt \
    -CAkey ca-private.pem \
    -CAcreateserial \
    -out issuer.crt \
    -copy_extensions copyall
rm issuer.csr

[cccsss01]

Just to be clear, with the script you published, you'd use this?
--set-file identityTrustAnchorsPem=ca.crt
--set-file identity.issuer.tls.crtPEM=issuer.crt
--set-file identity.issuer.tls.keyPEM=issuer-private.pem \