When southbound service is down the Gateway returns 500 instead of 503

[pj892031]

Describe the bug
If a service is down, but it is still in the registry, the response from Gateway should be 503 (service is unavailable), but it returns 500 instead.

Steps to Reproduce

1. start gateway, discovery, zaas, and mockService (within the static definition)
2. call discoverable client (ie. https://localhost:10010/dcpassticket/api/v1/request)
3. response is 500

Expected behavior
the response is 503 with a detailed message

Logs

io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/[0:0:0:0:0:0:0:1]:10012
	Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below: 
Assembly trace from producer [reactor.core.publisher.MonoFlatMap] :
	reactor.core.publisher.Mono.flatMap(Mono.java:3179)
	reactor.netty.transport.TransportConnector.lambda$connect$3(TransportConnector.java:171)
Error has been observed at the following site(s):
	*__________Mono.flatMap ⇢ at reactor.netty.transport.TransportConnector.lambda$connect$3(TransportConnector.java:171)
	*____________Mono.defer ⇢ at reactor.netty.transport.TransportConnector.lambda$connect$5(TransportConnector.java:169)
	|_       Mono.retryWhen ⇢ at reactor.netty.transport.TransportConnector.lambda$connect$5(TransportConnector.java:176)
	*________Flux.concatMap ⇢ at reactor.util.retry.RetrySpec.lambda$generateCompanion$6(RetrySpec.java:360)
	|_     Flux.onErrorStop ⇢ at reactor.util.retry.RetrySpec.lambda$generateCompanion$6(RetrySpec.java:379)
	*__Flux.deferContextual ⇢ at reactor.util.retry.RetrySpec.generateCompanion(RetrySpec.java:357)
	*____Mono.onErrorResume ⇢ at reactor.netty.transport.TransportConnector.lambda$connect$6(TransportConnector.java:166)
	*__________Mono.flatMap ⇢ at reactor.netty.transport.TransportConnector.connect(TransportConnector.java:165)
	*___________Mono.create ⇢ at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnectionAllocator.connectChannel(DefaultPooledConnectionProvider.java:535)
	*_____Mono.contextWrite ⇢ at reactor.netty.resources.PooledConnectionProvider.lambda$acquire$3(PooledConnectionProvider.java:174)
	*___________Mono.create ⇢ at reactor.netty.resources.PooledConnectionProvider.acquire(PooledConnectionProvider.java:126)
	*___________Mono.create ⇢ at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:210)
	|_       Mono.retryWhen ⇢ at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:275)
	*________Flux.concatMap ⇢ at reactor.util.retry.RetrySpec.lambda$generateCompanion$6(RetrySpec.java:360)
	|_     Flux.onErrorStop ⇢ at reactor.util.retry.RetrySpec.lambda$generateCompanion$6(RetrySpec.java:379)
	*__Flux.deferContextual ⇢ at reactor.util.retry.RetrySpec.generateCompanion(RetrySpec.java:357)
	*______Mono.flatMapMany ⇢ at reactor.netty.http.client.HttpClientFinalizer.responseConnection(HttpClientFinalizer.java:102)
	*______Mono.flatMapMany ⇢ at org.springframework.cloud.gateway.filter.NettyRoutingFilter.filter(NettyRoutingFilter.java:134)
	*_____________Flux.then ⇢ at org.springframework.cloud.gateway.filter.NettyRoutingFilter.filter(NettyRoutingFilter.java:199)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*_____________Mono.then ⇢ at org.springframework.cloud.gateway.filter.ReactiveLoadBalancerClientFilter.filter(ReactiveLoadBalancerClientFilter.java:148)
	|_       Mono.doOnError ⇢ at org.springframework.cloud.gateway.filter.ReactiveLoadBalancerClientFilter.filter(ReactiveLoadBalancerClientFilter.java:149)
	|_     Mono.doOnSuccess ⇢ at org.springframework.cloud.gateway.filter.ReactiveLoadBalancerClientFilter.filter(ReactiveLoadBalancerClientFilter.java:153)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*__________Mono.flatMap ⇢ at org.zowe.apiml.gateway.filters.AbstractAuthSchemeFactory.invoke(AbstractAuthSchemeFactory.java:192)
	*__________Mono.flatMap ⇢ at org.zowe.apiml.gateway.filters.AbstractAuthSchemeFactory.lambda$createGatewayFilter$22(AbstractAuthSchemeFactory.java:301)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*_____________Mono.then ⇢ at org.zowe.apiml.gateway.filters.PageRedirectionFilterFactory.lambda$apply$1(PageRedirectionFilterFactory.java:94)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	|_     Mono.doOnSuccess ⇢ at org.springframework.cloud.gateway.filter.factory.RetryGatewayFilterFactory.lambda$apply$16(RetryGatewayFilterFactory.java:238)
	|_       Mono.doOnError ⇢ at org.springframework.cloud.gateway.filter.factory.RetryGatewayFilterFactory.lambda$apply$16(RetryGatewayFilterFactory.java:238)
	|_       Mono.retryWhen ⇢ at org.springframework.cloud.gateway.filter.factory.RetryGatewayFilterFactory.lambda$apply$16(RetryGatewayFilterFactory.java:244)
	|_      Mono.repeatWhen ⇢ at org.springframework.cloud.gateway.filter.factory.RetryGatewayFilterFactory.lambda$apply$16(RetryGatewayFilterFactory.java:249)
	|_      Mono.fromDirect ⇢ at org.springframework.cloud.gateway.filter.factory.RetryGatewayFilterFactory.lambda$apply$16(RetryGatewayFilterFactory.java:252)
	*________Flux.concatMap ⇢ at reactor.retry.DefaultRetry.apply(DefaultRetry.java:120)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	|_     Mono.doOnSuccess ⇢ at org.springframework.cloud.gateway.filter.factory.SpringCloudCircuitBreakerFilterFactory$1.filter(SpringCloudCircuitBreakerFilterFactory.java:100)
	|_       Mono.transform ⇢ at org.springframework.cloud.circuitbreaker.resilience4j.ReactiveResilience4JCircuitBreaker.run(ReactiveResilience4JCircuitBreaker.java:90)
	|_         Mono.timeout ⇢ at org.springframework.cloud.circuitbreaker.resilience4j.ReactiveResilience4JCircuitBreaker.run(ReactiveResilience4JCircuitBreaker.java:93)
	|_       Mono.doOnError ⇢ at org.springframework.cloud.circuitbreaker.resilience4j.ReactiveResilience4JCircuitBreaker.run(ReactiveResilience4JCircuitBreaker.java:97)
	|_   Mono.onErrorResume ⇢ at org.springframework.cloud.circuitbreaker.resilience4j.ReactiveResilience4JCircuitBreaker.run(ReactiveResilience4JCircuitBreaker.java:101)
	|_   Mono.onErrorResume ⇢ at org.springframework.cloud.gateway.filter.factory.SpringCloudCircuitBreakerFilterFactory$1.filter(SpringCloudCircuitBreakerFilterFactory.java:132)
	*____________Mono.error ⇢ at org.springframework.cloud.gateway.filter.factory.SpringCloudCircuitBreakerFilterFactory$1.lambda$filter$1(SpringCloudCircuitBreakerFilterFactory.java:107)
	*____________Mono.error ⇢ at org.springframework.cloud.gateway.filter.factory.SpringCloudCircuitBreakerResilience4JFilterFactory.handleErrorWithoutFallback(SpringCloudCircuitBreakerResilience4JFilterFactory.java:51)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*_____________Mono.then ⇢ at org.springframework.cloud.gateway.filter.factory.SecureHeadersGatewayFilterFactory$1.filter(SecureHeadersGatewayFilterFactory.java:95)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	|_     Mono.doOnSuccess ⇢ at org.springframework.cloud.gateway.filter.GatewayMetricsFilter.filter(GatewayMetricsFilter.java:75)
	|_       Mono.doOnError ⇢ at org.springframework.cloud.gateway.filter.GatewayMetricsFilter.filter(GatewayMetricsFilter.java:76)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*_____________Mono.then ⇢ at org.springframework.cloud.gateway.filter.NettyWriteResponseFilter.filter(NettyWriteResponseFilter.java:69)
	|_      Mono.doOnCancel ⇢ at org.springframework.cloud.gateway.filter.NettyWriteResponseFilter.filter(NettyWriteResponseFilter.java:101)
	|_       Mono.doOnError ⇢ at org.springframework.cloud.gateway.filter.NettyWriteResponseFilter.filter(NettyWriteResponseFilter.java:102)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*__________Mono.flatMap ⇢ at org.springframework.cloud.gateway.support.ServerWebExchangeUtils.cacheRequestBody(ServerWebExchangeUtils.java:377)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	|_       Mono.doFinally ⇢ at org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter.filter(RemoveCachedBodyFilter.java:35)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*____________Mono.defer ⇢ at org.springframework.cloud.gateway.handler.FilteringWebHandler$DefaultGatewayFilterChain.filter(FilteringWebHandler.java:123)
	*_____________Mono.then ⇢ at org.springframework.web.reactive.result.SimpleHandlerAdapter.handle(SimpleHandlerAdapter.java:46)
Original Stack Trace:
Caused by: java.net.ConnectException: Connection refused: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:857)

Details

• Version and build number: 3.0.0+

Additional context
Exception io.netty.channel.AbstractChannel$AnnotatedConnectException is cached by org.zowe.apiml.gateway.controllers.GatewayExceptionHandler#handleInternalError. There could also be another exception that leads to 500.