Protocol detection issues

[edgarkz]

Hey Team,
Need assistance with linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s

We have meshed services based on dotnet protobuf-net.Grpc library, we do utilize grpc connections but for some reason we have hundreds of detection errors in the proxy log
Our setup includes:
Linkerd 2.13.6 on eks 1.25

Any suggestions how to make linkerd identify those connections as grpc so we can have enabled all the advanced l7 features?

[     0.002529s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.003127s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.003133s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.003136s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.003138s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.003141s]  INFO ThreadId(01) linkerd2_proxy: Local identity is global-service-account.core.serviceaccount.identity.linkerd.cluster.local
[     0.003143s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003146s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.019203s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=global-service-account.core.serviceaccount.identity.linkerd.cluster.local
[ 33957.357177s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[ 34330.339311s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[ 34462.278765s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[ 77236.369340s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[ 77488.271154s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[ 77748.115139s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[106568.178961s]  INFO ThreadId(01) outbound:proxy{addr=172.20.41.116:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[110923.742770s]  INFO ThreadId(01) outbound:proxy{addr=172.20.41.116:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[111172.342381s]  INFO ThreadId(01) outbound:proxy{addr=172.20.41.116:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[112233.814754s]  INFO ThreadId(01) outbound:proxy{addr=172.20.41.116:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[112911.178765s]  INFO ThreadId(01) outbound:proxy{addr=172.20.41.116:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[113040.652552s]  INFO ThreadId(01) outbound:proxy{addr=172.20.41.116:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[113452.771622s]  INFO ThreadId(01) outbound:proxy{addr=172.20.41.116:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[120465.583731s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[120615.998217s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[120709.452589s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[120838.042800s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[121084.728586s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
[121207.577083s]  INFO ThreadId(01) outbound:proxy{addr=172.20.237.189:8081}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s

[alpeb]

We've got some protocol detection timeout troubleshooting in the docs. Can you take a look and see if any of that applies to your case?

[edgarkz]

Hi @alpeb thanks for the above but none of the suggested solutions (opaque \ skip ports) provide a solution in our case, we have gRPC connections between microservices however linkerd can't detect any and all the connections are meshed as tcp traffic only.