Would like to provide a best pratics for AWS PV Migration

[Eroyi]

Not sure if its worth for create a pull request to put on the USAGE.md, i might just leave it here if you find this is helpful.

For migrating PV on AWS, there are few additional configuration required:

1. Amazon EKS require specify annotation on service.yaml to create a valid LoadBalancer ( nlb / alb / slb ). Check which kind of Load Balancer you are using.
2. After LB created, the FQDN on the External IP would take sometime to be answerable.
3. Service behind the FQDN need to be connectable, and for most of the cases it is. But if you applied additional security or network-isolation config, you may wanna lift those restriction temporarily.

Command:

pv-migrate migrate \
--source-kubeconfig /root/.kube/config \
--source-context arn:aws:eks:ap-southeast-1:000000000000cluster/non-prod \
--source-namespace mongodb \
  #  Specify the source cluster

--dest-kubeconfig /root/.kube/config \
--dest-context arn:aws:eks:ap-southeast-1:000000000000:cluster/dev \
--dest-namespace mongodb \
  #  Specify the destination cluster

--helm-set \
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-nlb-target-type"="instance",\
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-name"="pv-migrate",\
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-scheme"="internet-facing",\
sshd\.service\.annotations\."service\.beta\.kubernetes\.io\/aws-load-balancer-type"="external",\
  #  In order to create the service with correct ExternalIP ( FQDN ), append these configuratio on sshd.service.annotation gives it the ability to talk with aws-loadbalancer plugin.

rsync.maxRetries=20,\
rsync.retryPeriodSeconds=60 \
  #  FQDN registration could take a while before it can be resolve correctly, usually within 5-10 minuets. Default retries is 10 times and period is 5 seconds, which is too fast, the whole migration process will failed before FQDN become valid.

--ignore-mounted \
  #  Set this flag if your database or service is fully disconnected from any activity, yet you are not allowed to scale down/delete those pod.

--log-level debug --log-format json \
--dest-delete-extraneous-files \
  #  Only set this flag when you know what you are doing

--strategies lbsvc \
  #  There is no point to use any other strategies since we are doing this by LoadBalancer strategy

--helm-timeout 10m \
  #  Sometimes EKS API could take a little while longer on creating resources, default is 1m

non-prod-mongodb-pv dev-mongodb-pv
  #  Specify the old db pvc and new db pvc

The correct ExternalIP should looks like this 👇

[luis-alen]

I just want to add that the default ELB idle connection timeout is 60s and that might cause rsync to lose its connection, especially when re-syncing a huge directory tree with thousands of files. In that case it's also worth adding an annotation to change it to a value at least greater than the sshd pod ClientAliveInternal setting of 300s. I'd recommend setting it to 1h to be on the safe side. The annotation that helped me achieve this was service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"

Please see: #320 (comment)