Sentry deployed in Kubernetes bottleneck syncing

[gonzalomarcote]

So we have deployed 2 sentries in AWS EKS with 3 x t3.2xlarge instances.
Both sentries with enough resources and IOPS in storage:

requests:
        memory: 16384Mi
        cpu: "4000m"
...
parameters:
  fstype: ext4
  iops: "16000"
  throughput: "1000"

And the thing is that they take a lot to sync.
Showing the logs we can check that is only syncing about 10 blocks/second with external peers. Which we consider is too slow.
AWS EKS networking is the typical with the default (CNI plugin) with ALB,s for external rpc api services inbound traffic and the pods sentries outbound going trough NAT gateways (3 ngw each one for one private subnet in each AZ).

Checking the resources we can see they are underused in CPU/memory/IOPS aspects, and AFAIK AWS nat gateways should not have any outbound limit. So we are struggling our heads to try to understand where the bottleneck is and why sentries does not sync much faster as they are plenty of resources.

We are thinking about some networking issue but not completely sure. Due the dynamic nature of the cluster nodes can complicate access to external peers by having to constantly maintain the list of node IPs, as these can change. So we have tried this approach https://blog.damavis.com/en/adding-static-outbound-ips-in-amazon-eks/ with some improvements but not yet with the desired ones.

Any help in this aspect about perquisites or best practises deploying them in K8s would be much appreciated.