EKS Linkerd - Pods after injection get restarted many times

[kenwiths999]

Hi guys, I have a trouble when inject linkerd to my pods. Pods restart many times after inject the linkerd, this issue appear when the pod is running (after k8s startup-Probe finish), and after a while, it restarted. When my pod is started (container SpringBoot Java app), I also test health-check api from java app, and these apis work well. Here is pod description:

I think these health check Liveness and Readiness 502 503 come from linkerd container, causing my pod restarted.
Here is linkerd container log from my pod

Defaulted container "linkerd-proxy" out of: linkerd-proxy, java-app-springboot, linkerd-init (init)
[     0.003493s]  INFO ThreadId(01) linkerd2_proxy::rt: Using multi-threaded proxy runtime cores=2
[     0.004442s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.004461s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.004465s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.004468s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[     0.004471s]  INFO ThreadId(01) linkerd2_proxy: Local identity is fan-prod-app-sva.myapp.serviceaccount.identity.linkerd.cluster.local
[     0.004475s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.004482s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.029450s]  INFO ThreadId(04) daemon:identity: linkerd_app: Certified identity id=fan-prod-app-sva.myapp.serviceaccount.identity.linkerd.cluster.local
[    25.648401s]  INFO ThreadId(02) inbound:server{port=5353}:rescue{client.addr=10.3.14.77:36816}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[    35.558237s]  INFO ThreadId(02) inbound:server{port=5353}:rescue{client.addr=10.3.14.77:34632}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[    40.557345s]  INFO ThreadId(02) inbound:server{port=5353}:rescue{client.addr=10.3.14.77:34648}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
[    45.558435s]  INFO ThreadId(03) inbound:server{port=5353}:rescue{client.addr=10.3.14.77:52882}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

Here is linkerd config in deployment yaml:

  template:
    metadata:
      annotations:
        config.linkerd.io/proxy-cpu-request: "{{ .Values.linkerd.cpu.request }}"
        config.linkerd.io/proxy-cpu-limit: "{{ .Values.linkerd.cpu.limit }}"
        config.linkerd.io/proxy-memory-limit: "{{ .Values.linkerd.memory.limit }}"
        config.linkerd.io/proxy-memory-request: "{{ .Values.linkerd.memory.request }}"
        config.alpha.linkerd.io/proxy-wait-before-exit-seconds: "70s" #SIGTERM signal from Kubernetes
        config.linkerd.io/image-pull-policy: "IfNotPresent"
        config.linkerd.io/proxy-await: "true"
        config.linkerd.io/shutdown-grace-period: "500s" #Shut-Down immediately
        linkerd.io/inject: enabled

This issue does not always appear, it is occur when Rolling Update or run/scale new pods then, sometimes pod restart with no reason.

I also find that when pod is terminating with delay time preStop 80s. All connection to mongo + aws sqs + aws redis is corrupted

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to sqs.ap-southeast-2.amazonaws.com:443 [sqs.ap-southeast-2.amazonaws.com/3.104.83.198] failed: Connection refused (Connection refused)

org.mongodb.driver.cluster - Exception in monitor thread while connecting to server java-app-shard-99-99.999.mongodb.net:27017
com.mongodb.MongoSocketOpenException: Exception opening socket

Then I try to remove linkerd from my pod, and this case work perfectly.

My linkerd helm

Any ideals for these issue ? hope you guys can help!

EKS version 1.27
Linkerd version 2.13

[alpeb]

What exact version of linkerd are you on? I see you're using Redis, whose port (6379) should be by default treated as opaque by linkerd, but there was an issue with protocol detection and opaque ports that got fixed in stable-2.13.3.

[kenwiths999]

Here is link where I config linkerd for my EKS. Follow this, my linkerd is already in ver 2.13
https://linkerd.io/2.13/getting-started/

[realhuseyn]

I also have faced the same issue. Especially, our NodeJS applications are restarting frequently