randomly i got this error " Connection to node 1 (xxxx.eu-west-3.amazonaws.com) failed authentication due to: SSL handshake failed"

[hervekhg]

Hello,

I use vault as PKI to generate regularly cert, key and p12 with short TTL that is then used by AKHQ to authenticate to MSK through TLS.

The connection and the renew of cert works well, AKHQ read correctly the update p12 file, but ramdomly it crash with the error message : Connection to node 1 (b-1.hg-mykafka-test.xxxx.c3.kafka.eu-west-3.amazonaws.com) failed authentication due to: SSL handshake as if it had decided to not read anymore the updated p12.

I made a test with TTL of 1 minute (Every minutes new cert was generated), it worked during 42 minutes after more than 30 renew then crashed with the SSL handshake error.

I made the same test with TTL of 5 minutes, it worked during around 30 minutes then crashed with the same error

With TTL of 1h, it crashed after the first renew.

I do not understand what is happening, and i don't see any logic on that behavior. Could you help me please ?

[tchiotludo]

Really complex setup here ! must be for a bank I assume 😄

To be honest, I'm not really sure that this kind of deployment is supported by micronaut or kafka client.
If you ask me, I think the client must be cached and the ssl credentials will also be catched somewhere.

To help I need to understand :

• what kind of deployment are you using (compute, k8s, ...) ?
• how do you replace the certificate (just replace the file on disk, others ?)
• did you send any notice (http call) to akhq in order to let him know that there is some kind of change ?
• which ttl do you want for ssl at the end ?

[hervekhg]

Hello @tchiotludo,

The answer of your questions...

• The deployment are pods running on Kubernetes (EKS)
• Vault replace the certificate by erase the previous files by the new one
• I did not do any special action on AKHQ. It detect automatically when the certicate is updated. (For example, if the cert is not valid or the right of the p12 is not correct I got SSL Error message and when cert file is good, is running well before crashing ramdomly with no logic. It's not depend on ttl)
• A the end TLL of maximum 1h hour is our goal.

As I said everything works well during certain time (that have any correlation with TTL before crashing). Not easy to debug

[tchiotludo]

When you roll out the pod with the same ssl certificate (that throw SSL handshake), does it works ?

[hervekhg]

Hello,

The algorithm used (ECDSA) seems the root causes. With RSA we did not faced this problem

Thanks.