Upgrading Amazon Elasticsearch 7.x to OpenSearch 1.0 throws TCP error

[viral73]

I am getting Validation error while setting up cross-cluster search across 2 upgraded OpenSearch 1.0 domains. Both domains are in same VPC and Security Groups and also have met all 3 pre-reqs:
FGAC enabled
Node-to-node encryption enabled
Created after 06/2020

Error from CW logs:
[2021-09-29T13:30:22,264][WARN ][o.o.c.NodeConnectionsService] [6d0c08d5974adef60c469f321770d303] failed to connect to {4283ef20b232bf4a5e8b92e3f80ab1d6}{aN5KJrngQhqTaIHfqGPDhw}{ufxk_dVHSx-9y3FN2ALnJQ}{IP}{IP}{imr}{dp_version=20210501, box_type=dedicated-master, distributed_snapshot_deletion_enabled=false, cold_enabled=true, adv_sec_enabled=true, AMAZON_INTERNAL, cross_cluster_transport_address=IP, shard_indexing_pressure_enabled=true, AMAZON_INTERNAL} (tried [1] times) ConnectTransportException[[4283ef20b232bf4a5e8b92e3f80ab1d6][IP] connect_exception]; nested: AnnotatedConnectException[Connection refused: PATH]; nested: ConnectException[Connection refused]; at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:991) at org.opensearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:211) at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:55) at PATH(CompletableFuture.java:859) at PATH(CompletableFuture.java:837) at PATH(CompletableFuture.java:506) at PATH(CompletableFuture.java:2088) at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:70) at org.opensearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:81) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at PATH(Thread.java:834) Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: PATH Caused by: java.net.ConnectException: Connection refused at PATH(Native Method) at PATH(SocketChannelImpl.java:779) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at

Note - I configured net new 2 OpenSearch 1.0 Cluster domain in same VPC and SG and they are able to connect seamlessly for cross-cluster connection. This clearly shows issue is not with Network or Security Groups but conversion from AES -> OS breaks this feature.

Happy to provide more details.

Viral Shah