Support for custom route to internet gateway in AWS

[kapilraju]

/area networking
/kind enhancement
/priority 3
/platform aws

Brief summary:

We are using Gardener provisioned cluster in AWS. As per our network design we use Transit gateways and attach the VPC hosting Gardener cluster to a transit gateway in another AWS account which acts as a traffic hub.
We are creating a VPC on our own and supplying this VPC id while provisioning Gardener cluster. Whereas Gardener creates subnet, routes etc.

Limitations we are facing:

We cannot use our transit account for outbound connectivity and instead we have to attach an internet gateway to the same VPC hosting the gardener cluster.
We are adding routes to transit gateway manually after the cluster is provisioned for intranet connectivity.

What we would like to achieve?

We want the outbound and intranet connectivity via the transit gateway. So we would like to hear from you what is the best solution here. We can manage subnets, routes in our terraform code if you could guide us on the requirement of gardener cluster and then while provisioning the gardener cluster we can provide the subnet, route ids as inputs along with VPC id.
Another possibility we can think of is to have a way to supply additional routes information during provisioning. This way gardener can manage the subnet, route etc but we can provide the additional route which we would want to add for intranet connectivity or outbound connectivity.

[kapilraju]

@vlerenc Do you need any additional information to consider this feature request?

[dkistner]

Hi @kapilraju,

thanks for the request.

I guess you need to add those routes currently manually to the route tables of all zones you want to support in your cluster?
AFAIK a route can be potentially attached to multiple types of gateways (e.g. internet gateway, nat gateway, transit gateway).

Could you give an example how such route configuration in your scenario would look like in Terraform?

I'm also thinking how this should look like from an API perspective. As said routes could be added to multiple types of gateways. One example could be a user who have a peered vpc and want for any reason to use the internet gateway in the peered vpc to route the egress.

Just wanted to say there are probably a lot of combinations possible, which bring a lot of complexity (mainly validation I guess), if we allow bring your own routes. So I wanted to understand the use case better. What's the reason why the traffic need to get routed to the transit instead of the internet gateway directly? Is the reason security or a complex vpc peering setup?