Velero version 1.13.0 sends outbound traffic to China IP

[rizwank23]

Hi Team,

We planned to test velero in one of our eks clusters with chart version 6.0.0 and later we switched to another cluster. So in the first cluster, the velero pod is setup and it is up and running. But we deleted the S3 bucket. The pod is up and running but it is not able to find the default backup storage location.

Now the issue is, some china IP got hit using velero aws plugin and our organization has raised security concern on this. We tried setting up the same setup is one of our personal aws account with the same version. Now also we can see the outbound traffic to that IP. Is there any issue with the version 1.13.0

One more thing I would like to know is, If the backup location s3 bucket is deleted and if it is not able to find it, what exactly it will do. Will it try to push logs to any default location or any other path.

We are more concerned now to use this velero tool and looking for some help. Can someone help on the issue.

Thanks
Rizwan

[kaovilai]

If you can list the specific outbound traffic that would be helpful to isolate which component it came from.

[kaovilai]

And is it resolved by v1.15.0? or did it occur in versions prior to 1.13.0?

[rizwank23]

We didnt go to higher version, it was running with 1.13.0 only. Destination ip 54.222.96.37 dstport=443

[rizwank23]

[blackpiglet]

I couldn't open the attached image.
After clicking, this discussion page is loaded again.
The IP belongs to AWS S3. Could you check whether any configuration points to the cn-north-1 region's S3 bucket?

[rizwank23]

[rizwank23]

[rizwank23]

We use only one region us-east-1 but S3 should be global. Also we used the same helm chart with just the bucket name change and IAM role arn to access that particular bucket.

And if the bucket does not exist and velero pod is running, what it exactly does. Does it try to put something to any default location?

[rizwank23]

When mentioned bucket exists, we are not seeing this issue. When the mentioned bucket does not exist, we see this pointing to s3.cn-north-1.amazonaws.com.cn

[kaovilai]

Seems like aws sdk issue. Velero code do not have this address

[rizwank23]

Seems like aws sdk issue. Velero code to not have this address

Thanks @kaovilai One last question. If the bucket does not exist and velero pod is running, what it exactly does. Does it try to put something to any default location?

[kaovilai]

No it errors.

[rizwank23]

Thanks for the response. Could you please test and let us know what caused the issue to hit China region.

…

On Thu, Nov 14, 2024 at 8:32 PM blackpiglet ***@***.***> wrote: Thanks. I will test this on my developing environment and come back. I remember there was code that tried to create the bucket if it didn't exist, but it shouldn't change the region. — Reply to this email directly, view it on GitHub <#8406 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADL5UIKYAD2SRONL6EGG7CT2AS3QLAVCNFSM6AAAAABRYF5E32VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRVGU3DKNI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- ...Rizwan

[blackpiglet]

Could this be the reason?
https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/40c18f15bd464b0e6c1f6158a6bd3373fd1420b9/velero-plugin-for-aws/object_store.go#L133-L142

If the region and s3Url keys are not set for the BSL, the Velero AWS plugin will try a hint of the bucket region.
It's possible to ping the available region endpoints.

[rizwank23]

Region and bucket details we have added in our values.yaml file and while installing the velero we are passing this values yaml file. Attachin

g screen shot for reference.

[rizwank23]

helm install velero1 vmware-tanzu/velero \
--version 6.0.0 \
--namespace velero1 \
--create-namespace \
--set-file credentials.secretContents.cloud=/home/ec2-user/velero/velerocred \
--set configuration.backupStorageLocation[0].name=mybucket3056 \
--set configuration.backupStorageLocation[0].provider=aws \
--set configuration.backupStorageLocation[0].bucket=mybucket3056 \
--set configuration.backupStorageLocation[0].config.region=us-east-1 \
--set configuration.volumeSnapshotLocation[0].name=aws-snapshot \
--set configuration.volumeSnapshotLocation[0].provider=aws \
--set configuration.volumeSnapshotLocation[0].config.region=us-east-1 \
--set initContainers[0].name=velero-plugin-for-aws \
--set initContainers[0].image=velero/velero-plugin-for-aws:v1.8.1 \
--set initContainers[0].volumeMounts[0].mountPath=/target \
--set initContainers[0].volumeMounts[0].name=plugins

Using this command also we tried to install where we pass bucket name and region.

[blackpiglet]

Please try with the velero-plugin-for-aws v1.9.x version.
https://github.com/vmware-tanzu/velero-plugin-for-aws?tab=readme-ov-file#compatibility

I think this is related to how AWS SDK v1 works.
The 1.9.x version bumped the AWS SDK to v2.