Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The 'spring gateway' client frequently reports a "get services from nacos server fail Waited 3000 milliseconds Client not connected, current status:UNHEALTHY" exception #12999

Open
Robert-QiLin opened this issue Dec 27, 2024 · 0 comments
Open

The 'spring gateway' client frequently reports a "get services from nacos server fail Waited 3000 milliseconds Client not connected, current status:UNHEALTHY" exception #12999

Robert-QiLin opened this issue Dec 27, 2024 · 0 comments

Comments

@Robert-QiLin
Copy link

Spring Gateway service intermittently reports a timeout exception.

  • Version [nacos-server 2.4.3, nacos-client 2.4.3]
  • SDK [spring cloud 2023.0.3, spring-boot 3.2.11 spring-cloud-alibaba 2023.0.1.2, spring 6.1.14]

[exception behavior]:
2024-12-26T19:58:27.622+08:00 ERROR 3431369 --- [cbsc-gateway] [ndedElastic-789] c.a.c.n.d.r.NacosReactiveDiscoveryClient : get services from nacos server fail,

com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 71976 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@4c30f251[status=PENDING, info=[GrpcFuture{clientCall=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7ad1f4e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@10210d44, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@65f07418}}}]]
at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:82) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:647) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:624) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:447) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.getServiceList(NamingGrpcClientProxy.java:368) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.getServiceList(NamingClientProxyDelegate.java:159) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.NacosNamingService.getServicesOfServer(NacosNamingService.java:515) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.NacosNamingService.getServicesOfServer(NacosNamingService.java:503) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.cloud.nacos.discovery.NacosServiceDiscovery.getServices(NacosServiceDiscovery.java:70) ~[spring-cloud-starter-alibaba-nacos-discovery-2023.0.1.2.jar!/:2023.0.1.2]
at com.alibaba.cloud.nacos.discovery.reactive.NacosReactiveDiscoveryClient.lambda$getServices$3(NacosReactiveDiscoveryClient.java:87) ~[spring-cloud-starter-alibaba-nacos-discovery-2023.0.1.2.jar!/:2023.0.1.2]
at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46) ~[reactor-core-3.6.11.jar!/:3.6.11]
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.6.11.jar!/:3.6.11]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.6.11.jar!/:3.6.11]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.6.11.jar!/:3.6.11]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 71976 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@4c30f251[status=PENDING, info=[GrpcFuture{clientCall=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7ad1f4e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@10210d44, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@65f07418}}}]]
at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:531) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:79) ~[nacos-client-2.4.3.jar!/:na]
... 18 common frames omitted

2024-12-26T19:58:27.622+08:00 ERROR 3431369 --- [cbsc-gateway] [ask-Scheduler-1] c.a.c.n.discovery.NacosDiscoveryClient : get service name from nacos server failed.

com.alibaba.nacos.api.exception.NacosException: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 107677 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@11536a85[status=PENDING, info=[GrpcFuture{clientCall=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7ad1f4e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@10210d44, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@65f07418}}}]]
at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:82) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:647) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:624) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:447) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.getServiceList(NamingGrpcClientProxy.java:368) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.getServiceList(NamingClientProxyDelegate.java:159) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.NacosNamingService.getServicesOfServer(NacosNamingService.java:515) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.NacosNamingService.getServicesOfServer(NacosNamingService.java:503) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.cloud.nacos.discovery.NacosServiceDiscovery.getServices(NacosServiceDiscovery.java:70) ~[spring-cloud-starter-alibaba-nacos-discovery-2023.0.1.2.jar!/:2023.0.1.2]
at com.alibaba.cloud.nacos.discovery.NacosDiscoveryClient.getServices(NacosDiscoveryClient.java:80) ~[spring-cloud-starter-alibaba-nacos-discovery-2023.0.1.2.jar!/:2023.0.1.2]
at org.springframework.cloud.client.discovery.composite.CompositeDiscoveryClient.getServices(CompositeDiscoveryClient.java:68) ~[spring-cloud-commons-4.1.4.jar!/:4.1.4]
at com.github.xiaoymin.knife4j.spring.gateway.discover.ServiceChangeListener.discover(ServiceChangeListener.java:49) ~[knife4j-gateway-spring-boot-starter-4.5.0.jar!/:na]
at jdk.internal.reflect.GeneratedMethodAccessor101.invoke(Unknown Source) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:365) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:237) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:168) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:452) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:385) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.cloud.gateway.route.RouteRefreshListener.reset(RouteRefreshListener.java:73) ~[spring-cloud-gateway-server-4.1.5.jar!/:4.1.5]
at org.springframework.cloud.gateway.route.RouteRefreshListener.resetIfNeeded(RouteRefreshListener.java:68) ~[spring-cloud-gateway-server-4.1.5.jar!/:4.1.5]
at org.springframework.cloud.gateway.route.RouteRefreshListener.onApplicationEvent(RouteRefreshListener.java:62) ~[spring-cloud-gateway-server-4.1.5.jar!/:4.1.5]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:452) ~[spring-context-6.1.14.jar!/:6.1.14]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:385) ~[spring-context-6.1.14.jar!/:6.1.14]
at com.alibaba.cloud.nacos.discovery.NacosDiscoveryHeartBeatPublisher.publishHeartBeat(NacosDiscoveryHeartBeatPublisher.java:104) ~[spring-cloud-starter-alibaba-nacos-discovery-2023.0.1.2.jar!/:2023.0.1.2]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.1.14.jar!/:6.1.14]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: java.util.concurrent.TimeoutException: Waited 3000 milliseconds (plus 107677 nanoseconds delay) for com.alibaba.nacos.shaded.io.grpc.stub.ClientCalls$GrpcFuture@11536a85[status=PENDING, info=[GrpcFuture{clientCall=ClientCallImpl{method=MethodDescriptor{fullMethodName=Request/request, type=UNARY, idempotent=false, safe=false, sampledToLocalTracing=true, requestMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@7ad1f4e4, responseMarshaller=com.alibaba.nacos.shaded.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller@10210d44, schemaDescriptor=com.alibaba.nacos.api.grpc.auto.RequestGrpc$RequestMethodDescriptorSupplier@65f07418}}}]]
at com.alibaba.nacos.shaded.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:531) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.common.remote.client.grpc.GrpcConnection.request(GrpcConnection.java:79) ~[nacos-client-2.4.3.jar!/:na]
... 38 common frames omitted

2024-12-26T19:58:28.026+08:00 ERROR 3431369 --- [cbsc-gateway] [ndedElastic-789] c.a.c.n.d.r.NacosReactiveDiscoveryClient : get services from nacos server fail,

com.alibaba.nacos.api.exception.NacosException: Client not connected, current status:UNHEALTHY
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:645) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.common.remote.client.RpcClient.request(RpcClient.java:624) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.requestToServer(NamingGrpcClientProxy.java:447) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.gprc.NamingGrpcClientProxy.getServiceList(NamingGrpcClientProxy.java:368) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.remote.NamingClientProxyDelegate.getServiceList(NamingClientProxyDelegate.java:159) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.NacosNamingService.getServicesOfServer(NacosNamingService.java:515) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.nacos.client.naming.NacosNamingService.getServicesOfServer(NacosNamingService.java:503) ~[nacos-client-2.4.3.jar!/:na]
at com.alibaba.cloud.nacos.discovery.NacosServiceDiscovery.getServices(NacosServiceDiscovery.java:70) ~[spring-cloud-starter-alibaba-nacos-discovery-2023.0.1.2.jar!/:2023.0.1.2]
at com.alibaba.cloud.nacos.discovery.reactive.NacosReactiveDiscoveryClient.lambda$getServices$3(NacosReactiveDiscoveryClient.java:87) ~[spring-cloud-starter-alibaba-nacos-discovery-2023.0.1.2.jar!/:2023.0.1.2]
at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46) ~[reactor-core-3.6.11.jar!/:3.6.11]
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194) ~[reactor-core-3.6.11.jar!/:3.6.11]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.6.11.jar!/:3.6.11]
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.6.11.jar!/:3.6.11]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Robert-QiLin