Create network resources for AWS Local Zones and Wavelength Zones

[mtulio]

/kind feature

Describe the solution you'd like:

Support of creation network infrastructure, subnets, and gateways, required for AWS
Local Zones and Wavelength zones, allowing users to create compute nodes (non-control
plane) specifying those subnets.

AWS Local Zones allow end-users to deploy cloud resources closer to the metropolitan
regions and offer a limited resource, like Instance Type, Block Storage, Application Load
Balancers, Nat Gateways etc.
The network infrastructure required to create nodes in Local Zones are:

• public and private subnets,
• public subnet associated with a route table with Internet Gateway as a default router
• private subnet associated with a route table with a valid Nat Gateway*
• *Currently, Nat Gateway is not globally supported by Local Zone locations, and currently there
  is not API to query if it is offered in the zone.

AWS Wavelength zones are infrastructure running in carrier infrastructure. It also have
limitations of resources offered in those locations.

Network resources required for Wavelength zones:

• Carrier Gateway attached to the VPC
• public and private subnets
• public "carrier" route table with the default route entry targeting the Carrier Gateway
• private route table with the default route entry pointing to a valid Nat Gateway*
• public subnet associated with the public "carrier" route table
• private subnet associated with the private route table
• *Same Nat GW limitation as Local Zone.

Not Goal:

• Create control plane nodes, or required resources, in remote zones.
• Compute/Machine resource creation.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Currently, OpenShift creates the network resources required to deploy only compute
nodes in AWS Local Zones and Wavelength zones, and we would like to use Cluster API to
deploy those resources. Machines (compute nodes) are then
created, outside Cluster API.

Considering the limitation of network-based load balancers, we used to recommend
to the users to use the AWS Application Load Balancer controller to
create ingress traffic into those zones (day-2).

For the gateway for private subnets, we are reusing the same Nat Gateway for the
Parent Zone (zone attribute), when it exists, otherwise, the
"default" private route table will be used to associate the zone in the remote location.
If possible, it would be nice to deploy Nat Gateway in the location, when supported
(currently, only a few Local Zone locations support it), otherwise, use some strategy like that.

For the gateway for public subnets in Wavelength Zones, we are creating a single
"public carrier" route table, with the default route entry to the Carrier Gateway,
previously created and associated with the VPC,

For more details feel free to read the enhancement with more details here.

The suggested Local Zones workflow looks like this:

• User creates the Subnets (spec.Network.Subnets) specifying all required zones to
  create the cluster, including Local Zone or Wavelength Zones, in the existing API
  field SubnetSpec.AvailabilityZone - keeping the valid placeholder ID to
  teach Cluster API to create subnets.

  zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-nyc-1a"}	
  for _, zone := range zones {
    cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
        AvailabilityZone: zone,
        CidrBlock:        privateCIDRs[idxCIDR].String(),
	ID:               fmt.Sprintf("%s-subnet-private-%s", infraID, zone),
	IsPublic:         false,
	})
    cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
        AvailabilityZone: zone,
	CidrBlock:        publicCIDRs[idxCIDR].String(),
	ID:               fmt.Sprintf("%s-subnet-public-%s", infraID, zone),
	IsPublic:         true,
    })
    idxCIDR++
  }

• Network reconciliation creates the Carrier Gateway, attaching to the VPC
• Subnet reconciliation creates the public and private subnets into those zones,
  then discovers the zone attributes* for the subnet's zone
• Subnets prevent returning subnets created in Wavelength Zone locations
  to the regular flow, preventing issues in the controllers consuming the standard flow
• Route reconciliation check whether the public subnet "is Wavelength Zone", and find the
  appropriate route and gateway for public subnets in Carrier Gateways
• Route reconciliation check whether the subnet "is Wavelength Zone", and find the
  appropriate gateway for private subnets
• Regular cluster creation flow

*Required API changes for the proposed workflow:

• Add the following fields to the SubnetSpec:

	// ZoneType defines a zone type for this subnet.
	ZoneType *string `json:"zoneType,omitempty"`

	// ParentZoneName defines a parent zone name for the zone that the subnet is created,
	// when applied. Available only in zone types local-zone or wavelength-zone.
	ParentZoneName *string `json:"parentZoneName,omitempty"`

The suggested Wavelength Zones workflow looks like this:

• User creates the Subnets (spec.Network.Subnets) specifying all required zones to
  create the cluster, including Local Zone or Wavelength Zones, in the existing API
  field SubnetSpec.AvailabilityZone - keeping the valid placeholder ID to
  teach Cluster API to create subnets.

  zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-wl1-nyc-wlz-1"}
...

• Subnet reconciliation creates the public and private subnets into those zones,
  then discovers the zone attributes* for the subnet's zone
• Subnets prevent returning subnets created in Wavelength Zone locations
  to the regular flow, preventing issues in the controllers consuming the standard flow
• Route reconciliation check whether the subnet "is Local Zone", and find the
  appropriate gateway for private subnets
• Regular cluster creation flow

*Required API changes for the proposed workflow:

• Add the following fields to the VPCSpec:

	// CarrierGatewayID is the id of the Carrier internet gateway associated with the VPC.
	// +optional
	CarrierGatewayID *string `json:"carrierGatewayId,omitempty"`

Tasks

Beta Give feedback

1. ✨ Introduce edge subnets to support AWS Local Zones #4882
   approved cncf-cla: yes kind/api-change kind/feature lgtm needs-priority ok-to-test release-note size/XXL +9
   

   
2. ✨ Support of managing subnets on AWS Wavelength Zones #4901
   approved cncf-cla: yes kind/api-change kind/feature lgtm needs-priority ok-to-test release-note-action-required size/XXL +9
   

   
3. 🌱 Implement e2e tests for edge subnets

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.