NE-1517: Set EIP for NLB Ingress controller. #1593

miheer · 2024-03-12T11:04:02Z

This EP describes how to set EIP for NLB ingress controller.

Epic link: https://issues.redhat.com/browse/NE-1274

openshift-ci-robot · 2024-03-22T16:06:53Z

@miheer: This pull request references NE-1517 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

This EP describes how to set EIP for NLB ingress controller.

Epic link: https://issues.redhat.com/browse/NE-1274

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

candita · 2024-03-27T15:38:26Z

/assign @frobware
/assign @Miciah
/assign

frobware

My main concerns revolve around the expectation that updating EIP allocation IDs will automatically update an existing NLB. From my brief experimentation, it seems that the system only recognises these allocations, as specified by the annotation, at the time of creation. The document implies at several points that changing EIP allocation IDs is possible and that such changes will be reconciled with the existing ELB without necessitating its recreation. It would be beneficial to conduct further tests to determine if modifications to the annotation result in immediate updates on a functioning ELB.

frobware · 2024-03-28T11:36:25Z

enhancements/ingress/set_eip_nlb_ingress.md

+This is a feature request to enhance the IngressController API to be able to support static IPs during
+- Install time
+- Custom NLB ingress controller creation 
+- Reconfiguration of the router.


As elastic IP allocations can only be added at creation time, what does reconfiguration imply recreation?

I don't understand how the router is involved at all. Should this say "Updating the IngressController"? Andy's question then applies.

Updating the IngressController is accurate.

frobware · 2024-03-28T11:37:26Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+- As an administrator, I want to provision EIPs and use them with an NLB IngressController.
+- As a user or admin, I want to ensure EIPs are used with NLB on default router at install time.
+- As a user, I want to reconfigure default router to use EIPs.


As per my previous comment: reconfiguration would have to be destruction followed by creation as elastic allocation IDs need to be done at creation time.

Yes the workflow has changed. We will recreate the LB.

frobware · 2024-03-28T11:41:35Z

enhancements/ingress/set_eip_nlb_ingress.md

+        eip-allocations: eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted> 
+    subnets:
+    - subnet-1
+    - subnet-2
+    - subnet-3


In the proposed design, will there be a check to ensure the count of EIP allocation IDs matches the number of public subnets? Without this, one might only discover that the ingress controller cannot deploy after much progress in the installation process...

One option is we can add a field to mention the number of public subnets. Then using CEL at API level we could try to compare the number of eip-allocations they provided and the the number of public subnets

Or I think we may put this as a Non-Goal for this EP.

Andy said "in the installation process". Validation to ensure installation succeeds can be done by the installer.

I added this point in the Open Questions

frobware · 2024-03-28T11:42:03Z

enhancements/ingress/set_eip_nlb_ingress.md

+ # Please edit the object below. Lines beginning with a '#' will be ignored,
+ # and an empty file will abort the edit. If an error occurs while saving this file will be
+ # reopened with the relevant failures.
+ #


Nit: Maybe remove the noise?

frobware · 2024-03-28T11:44:07Z

enhancements/ingress/set_eip_nlb_ingress.md

+       aws:
+         type: NLB
+           networkLoadBalancer:
+             eip-allocations: eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>,eipalloc-<redacted>


Could we update the example to use sequential numbers for the EIP allocation IDs? This would make it clearer. Suggesting: "eipalloc-12345, eipalloc-12346, eipalloc-12347, eipalloc-12348, eipalloc-12349". Ditto for the other usage where we use .

frobware · 2024-03-28T11:56:41Z

enhancements/ingress/set_eip_nlb_ingress.md

+- The API field for `eip-allocations` field in the Ingress Controller CR needs to be set as follows
+  in the file `operator/v1/types_ingress.go`:
+```go
+// AWSNetworkLoadBalancerParameters holds configuration parameters for an
+// AWS Network load balancer.
+type AWSNetworkLoadBalancerParameters struct {
+    EIPAllocations EIPAllocations `json:"eip-allocations,omitempty"`        
+}
+
+type EIPAllocations string
+
+```


This appears to be duplication. Aren't the types also described above?

We can do something like this

// AWSNetworkLoadBalancerParameters holds configuration parameters for an // AWS Network load balancer. type AWSNetworkLoadBalancerParameters struct { EIPAllocations configv1.EIPAllocations `json:"eip-allocations,omitempty"` }

In operator/v1/types_ingress.go we already have a type ProviderLoadBalancerParameters with an AWSLoadBalancerParameters. Is that where this is supposed to go? https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L473

not sure what you mean please check openshift/api@261093f#diff-891d2cdcb01603ef2a2c1bfb2f6ef32cf2acf8b1471871a5a6e27308d50050f9R641

Okay, my confusion was because the struct was empty to begin with. I thought you were adding a new struct type. Btw, your api PR shows this as a slice:

type AWSNetworkLoadBalancerParameters struct { EIPAllocations []configv1.EIPAllocations `json:"eipAllocations,omitempty"` }

OK so I hope your question is answered.

frobware · 2024-03-28T11:58:26Z

enhancements/ingress/set_eip_nlb_ingress.md

+   The admin sets the `eip-allocations` in the Ingress Controller CR. Then same process
+   as per point 1.c.ii and 1.c.iii is followed by the ingress operator. 
+
+#### 3. Update the existing EIP for the IngressController:


As previously mentioned, will changes to EIP allocations actually be reconciled after creation?

#1593 (comment)

@frobware we need to recreate the LB service.

frobware · 2024-03-28T12:00:11Z

enhancements/ingress/set_eip_nlb_ingress.md

+#### 3. Update the existing EIP for the IngressController:
+
+   When the admin changes the eip-allocations field in the Ingress Controller CR, the watch on the
+   `eip-allocations` field by the ingress operator scales a new deployment for the Ingress Controller CR.


Oh, so if you change the EIP allocations then you get a "brand new" ingresscontroller? That might be surprising as it will drop all traffic. Seems subtle.

@frobware I need to test again. It looks like if the service is updated with the annotation the kcm should update the eips. https://github.com/brooksgarrett/kubernetes/blob/72dd54ec4856ccb7bd2b08258d95f3d0b932fb11/pkg/cloudprovider/providers/aws/aws.go#L4157

It looks like https://github.com/openshift/enhancements/compare/45ad01df72b72174f9347fcb54179af4661cbd19..1137a8acba5a4c72ae496c34fa93b97dd18d3990 resolved this question by removing "scales a new deployment" and instead describing the workflow for recreating the service.

frobware · 2024-03-28T12:01:14Z

enhancements/ingress/set_eip_nlb_ingress.md

+  in the Ingress Controller CR and then test if the Ingress Controller
+  with service type load balancer with the annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations`
+  has value of `eip-allocations` field from the Ingress Controller CR was updated.
+


Is there a plan for testing when the EIP allocations are plumbed in the install-config.yaml?

There will be unit tests in the installer.

frobware · 2024-03-28T12:02:19Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+## Upgrade / Downgrade Strategy
+
+During upgrade, user should be able to populate the eip_allocations_field via installer


If EIP allocations are only reconciled at creation time can this work? The cluster is upgrading which implies it has a deployed ingresscontroller.

Yes you are right. During the upgrade we won't have this option for populating the field. So, the cluster admin will have to set the eip-allocations field once the cluster is installed.

Miciah · 2024-03-28T16:03:42Z

enhancements/ingress/set_eip_nlb_ingress.md

+- As an administrator, I want to provision EIPs and use them with an NLB IngressController.
+- As a user or admin, I want to ensure EIPs are used with NLB on default router at install time.
+- As a user, I want to reconfigure default router to use EIPs.
+- As a user of OpenShift on AWS (ROSA), I want to use static IPs (and therefore AWS Elastic IPs) so that 


These roles are ambiguous. Is the administrator or user in all cases the cluster administrator? If there are multiple roles, can you define them?

OK. From my understanding we shall have only one user i.e the cluster-admin as he is the one who is responsible for creating routers.

Miciah · 2024-03-28T16:14:30Z

enhancements/ingress/set_eip_nlb_ingress.md

+### User Stories
+
+- As an administrator, I want to provision EIPs and use them with an NLB IngressController.
+- As a user or admin, I want to ensure EIPs are used with NLB on default router at install time.
+- As a user, I want to reconfigure default router to use EIPs.
+- As a user of OpenShift on AWS (ROSA), I want to use static IPs (and therefore AWS Elastic IPs) so that 
+  I can configure appropriate firewall rules.
+  I want the default AWS Load Balancer that they use (NLB) for their router to use these EIPs.


Another one:

As a cluster-admin who has purchased a block of public IPv4 addresses, I want to use these IP addresses, so that I can avoid Amazon's new charge for public IP addresses.

miheer · 2024-03-29T20:21:11Z

My main concerns revolve around the expectation that updating EIP allocation IDs will automatically update an existing NLB. From my brief experimentation, it seems that the system only recognises these allocations, as specified by the annotation, at the time of creation. The document implies at several points that changing EIP allocation IDs is possible and that such changes will be reconciled with the existing ELB without necessitating its recreation. It would be beneficial to conduct further tests to determine if modifications to the annotation result in immediate updates on a functioning ELB.

@frobware #1593 (comment)

candita · 2024-04-01T14:59:18Z

enhancements/ingress/set_eip_nlb_ingress.md

+  - ""
+---
+
+# Set AWS EIP For NLB Ingress Controller


Please define EIP and NLB at least once in this proposal.

Yes added under "Proposal"

candita · 2024-04-01T15:38:30Z

enhancements/ingress/set_eip_nlb_ingress.md

+### Goals
+- Users are able to use EIP for a NLB Ingress Controller.
+- Users are able to create a NLB default Ingress Controller during install time with the EIPs specified during install.
+- Any existing ingress controller should be able to be reconfigured to use EIPs. 


As mentioned earlier, "reconfigure" should really be "update with restart", right?

I see that https://github.com/openshift/enhancements/compare/1137a8acba5a4c72ae496c34fa93b97dd18d3990..79d098963268d49c7e4b7d64ffe1e25878c2b6c0 implemented this suggestion.

To me, "restart" implies that the router pod has to be restarted. How about this?

- The ELB of an existing IngressController should be able to be reconfigured to use specific EIPs by updating the IngressController CR and recreating the ELB.

added with slight changes.

candita · 2024-04-01T17:18:17Z

enhancements/ingress/set_eip_nlb_ingress.md

+In this enhancement we would be adding API fields in the installer and the ingress controller CRD
+to set the AWS EIP for the NLB Ingress Controller. The cluster ingress operator will then create 
+the ingress controller and get the EIP assigned to the service LoadBalancer with the annotation 
+`service.beta.kubernetes.io/aws-load-balancer-eip-allocations`


Can you add a note about why the annotation is in kubernetes.io rather than openshift.io?

Looks like this was addressed in https://github.com/openshift/enhancements/compare/1137a8acba5a4c72ae496c34fa93b97dd18d3990..79d098963268d49c7e4b7d64ffe1e25878c2b6c0. However, that change uses service.beta.kubernetes.io/aws-load-balancer-eip-allocations as the target of a link, which will not work. Better to use https://kubernetes.io/docs/reference/labels-annotations-taints/#service-beta-kubernetes-io-aws-load-balancer-eip-allocations as a documentation reference.

candita · 2024-04-01T17:19:18Z

enhancements/ingress/set_eip_nlb_ingress.md

+cluster. A cluster administrator also has the rights to modify the cluster level components.
+
+#### Set EIP using an installer:
+1. Cluster administrator creates adds the following in the install-config.yaml


Suggested change

1. Cluster administrator creates adds the following in the install-config.yaml

1. Cluster administrator creates or adds the following in the install-config.yaml

updated in a different way

candita · 2024-11-05T23:24:27Z

enhancements/ingress/set_eip_nlb_ingress.md

+% oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"endpointPublishingStrategy":{"type":"LoadBalancerService","loadBalancer":{"providerParameters":{"type":"AWS","aws":{"type":"NLB","nlbParameters":{"eipAllocations":[]}}}}}}}'
+```
+
+After the changes are saved, the operator will delete and create the load balancer service automatically.


Suggested change

After the changes are saved, the operator will delete and create the load balancer service automatically.

After the changes are saved, the operator will automatically delete and create the load balancer service.

candita · 2024-11-05T23:40:12Z

enhancements/ingress/set_eip_nlb_ingress.md

+#### Unmanaged EIPAllocation Annotation Migration Workflow
+
+Migrating an unmanaged `service.beta.kubernetes.io/aws-load-balancer-eip-allocations`
+service annotation to `spec.endpointPublishingStrategy.loadBalancer.providerParameters.aws.networkLoadBalancerParameters.eipAllocations`
+after upgrading to 4.y doesn't require a cluster admin to delete the LoadBalancer-type
+service:
+
+1. Cluster admin confirms that initially `service.beta.kubernetes.io/aws-load-balancer-eip-allocations`
+   is set on the LoadBalancer-type service managed by the IngressController.
+2. The cluster admin upgrades OpenShift to v4.y and the service annotation is not
+   changed or removed. For more details see [section](#4-any-change-to-the-service-annotation-does-not-trigger-the-operator-to-manage-the-load-balancer-service)  
+3. After upgrading, the IngressController will emit a `LoadBalancerProgressing` condition
+   with `Status: True` because the spec's `EIPAllocations` (an empty slice `[]`) does not equal
+   the current annotation.
+4. In this case, the cluster admin must directly update
+   `spec.endpointPublishingStrategy.loadBalancer.providerParameters.aws.networkLoadBalancerParameters.eipAllocations` to the
+   current value of the `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` service.
+5. The Ingress Operator will resolve the `LoadBalancerProgressing` condition back to
+   `Status: False` as long as `EIPAllocations` is equivalent to `service.beta.kubernetes.io/aws-load-balancer-eip-allocations`.


I'm not really sure what this is for. If it is a hint to be followed when upgrading, it probably should go into the section on handling upgrades. Otherwise, it needs some substantial editing. Couldn't you just say something like:

If you're upgrading to 4.x and currently using service.beta.kubernetes.io/aws-load-balancer-eip-allocations, be sure to update the eipAllocations manually after an upgrade.

candita · 2024-11-05T23:47:27Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+#### Updating EIP in the default Classic IngressController after installation
+
+We support updating the EIP in the default Classic `IngressController`.


Don't use the word "support" in an EP, people may get the wrong idea.
However, I thought this EP was only for NLBs.

candita · 2024-11-06T17:33:25Z

enhancements/ingress/set_eip_nlb_ingress.md

+If the cluster-admin manually sets the annotation, then in that case syncIngressControllerStatus will reflect it to status.
+More details on the implementation are as follows:
+
+#### 1. Create a new IngressController with EIPs:


If this is implmentation details for a custom (not default) IngressController, you should say:

Suggested change

#### 1. Create a new IngressController with EIPs:

#### 1. When creating a customer IngressController with EIPs:

candita · 2024-11-06T17:35:30Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+#### 1. Create a new IngressController with EIPs:
+
+   The admin sets the `eipAllocations` in the `IngressController` CR. The cluster ingress operator then creates a `IngressController` by:


Suggested change

The admin sets the `eipAllocations` in the `IngressController` CR. The cluster ingress operator then creates a `IngressController` by:

The cluster admin sets the `eipAllocations` in the `IngressController` CR. The cluster ingress operator then creates a new `IngressController` by:

candita · 2024-11-06T17:37:14Z

enhancements/ingress/set_eip_nlb_ingress.md

+#### 2. Update the EIP for an existing IngressController:
+   ##### Effectuating EIP Updates using manual method:
+   When the admin changes the eipAllocations field in the `IngressController` CR, the ingress operator sets the Load Balancer
+   Progressing status as per [section](#cluster-administrator-updates-eipallocations-field-of-the-existing-ingress-controller).


Broken link.

candita · 2024-11-06T17:37:28Z

enhancements/ingress/set_eip_nlb_ingress.md

+  1. The cluster admin manually configured the `service.beta.kubernetes.io/aws-load-balancer-eip-allocations`
+   annotation on the service, which is not supported and likely to cause issues.
+  2. The cluster admin configured the IngressController's `EIPAllocations` spec, but hasn't
+   effectuated the change by deleting the service (see [section](#cluster-administrator-updates-eipallocations-field-of-the-existing-ingress-controller)).


Broken link.

candita · 2024-11-06T17:39:00Z

enhancements/ingress/set_eip_nlb_ingress.md

+   2. then creating a service of service type load balancer with the annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations`,
+      which uses the value from the field `eipAllocations` of `IngressController` CR.  
+
+#### 2. Update the EIP for an existing IngressController:


Suggested change

#### 2. Update the EIP for an existing IngressController:

#### 2. When changing the EIPs for an existing IngressController:

candita · 2024-11-06T17:40:22Z

enhancements/ingress/set_eip_nlb_ingress.md

+      which uses the value from the field `eipAllocations` of `IngressController` CR.  
+
+#### 2. Update the EIP for an existing IngressController:
+   ##### Effectuating EIP Updates using manual method:


A better name for this is "managed", which implies that the cluster admin is managing the process.

Suggested change

##### Effectuating EIP Updates using manual method:

In order to provide the most flexibility to cluster admin operations, there are two methods that cover the modification of EIPs in an existing IngressController, managed and automatic.

##### Managed

candita · 2024-11-06T17:41:22Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+#### 2. Update the EIP for an existing IngressController:
+   ##### Effectuating EIP Updates using manual method:
+   When the admin changes the eipAllocations field in the `IngressController` CR, the ingress operator sets the Load Balancer


Suggested change

When the admin changes the eipAllocations field in the `IngressController` CR, the ingress operator sets the Load Balancer

When the cluster admin changes the eipAllocations field in the `IngressController` specification, the ingress operator sets the Load Balancer

candita · 2024-11-06T17:47:41Z

enhancements/ingress/set_eip_nlb_ingress.md

+   The cluster admin has to manually delete service in order to effectuate the change.
+   Then step 1.2 is followed by the ingress operator.


Suggested change

The cluster admin has to manually delete service in order to effectuate the change.

Then step 1.2 is followed by the ingress operator.

The cluster admin manually deletes the service when they are ready for the ensuing service interruption.

The rest of the service configuration will happen automatically.

candita · 2024-11-06T17:49:35Z

enhancements/ingress/set_eip_nlb_ingress.md

+   Progressing status as per [section](#cluster-administrator-updates-eipallocations-field-of-the-existing-ingress-controller).
+   The cluster admin has to manually delete service in order to effectuate the change.
+   Then step 1.2 is followed by the ingress operator.
+   We have intentionally designed this feature to require manual deletion of the service for the following reasons:


Make a new paragraph.

Suggested change

We have intentionally designed this feature to require manual deletion of the service for the following reasons:

We have intentionally designed this feature to require manual deletion of the service for the following three reasons:

candita · 2024-11-06T17:51:42Z

enhancements/ingress/set_eip_nlb_ingress.md

+   While the AWS Cloud Controller Manager (CCM) handles invalid annotations by producing errors and events, it is difficult 
+   for the Ingress Operator to decipher these CCM events after the load balancer has been provisioned. 


"it is difficult" doesn't say much, and stands out as a flaw. Can it be reworded as:

Suggested change

While the AWS Cloud Controller Manager (CCM) handles invalid annotations by producing errors and events, it is difficult

for the Ingress Operator to decipher these CCM events after the load balancer has been provisioned.

While the AWS Cloud Controller Manager (CCM) handles invalid annotations by producing errors and events, it doesn't provide enough information

for the Ingress Operator to decipher these CCM events after the load balancer has been provisioned.

Suggested change

While the AWS Cloud Controller Manager (CCM) handles invalid annotations by producing errors and events, it is difficult

for the Ingress Operator to decipher these CCM events after the load balancer has been provisioned.

While the AWS Cloud Controller Manager (CCM) handles invalid annotations by producing errors and events, it is difficult

for the Ingress Operator to decipher these CCM events after the load balancer has been provisioned.

candita · 2024-11-06T17:52:31Z

enhancements/ingress/set_eip_nlb_ingress.md

+   While the AWS Cloud Controller Manager (CCM) handles invalid annotations by producing errors and events, it is difficult 
+   for the Ingress Operator to decipher these CCM events after the load balancer has been provisioned. 
+   Therefore, the Ingress Operator can't indicate to a cluster admin that an IngressController's load balancer is in a 
+   malfunctioning state (i.e., the service is not getting reconciled) while the service is already provisioned.


Suggested change

malfunctioning state (i.e., the service is not getting reconciled) while the service is already provisioned.

malfunctioning state (i.e., the service is not getting reconciled) while the service is being provisioned.

candita · 2024-11-06T17:56:33Z

enhancements/ingress/set_eip_nlb_ingress.md

+   unsupported configuration.
+
+   If the `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` annotation was previously configured and the cluster is upgraded,
+   the default value [] for EIPAllocations will clear the annotation for existing IngressControllers, which would break cluster ingress.


Is this what you mean here?

Suggested change

the default value [] for EIPAllocations will clear the annotation for existing IngressControllers, which would break cluster ingress.

the empty default value for EIPAllocations will clear the annotation for existing IngressControllers, which would break cluster ingress.

candita · 2024-11-06T17:58:38Z

enhancements/ingress/set_eip_nlb_ingress.md

+   Note: Cluster admins upgrading with an unmanaged EIP annotation don't need to delete the service to propagate the `eipAllocation` values, 
+   instead they can follow the [Unmanaged EIPAllocation Annotation Migration Workflow](#unmanaged-eipallocation-annotation-migration-workflow).
+
+   Reason #3: The CCM Doesn't Reconcile NLB `eipAllocation` Updates after creation.


Suggested change

Reason #3: The CCM Doesn't Reconcile NLB `eipAllocation` Updates after creation.

Reason #3: The CCM doesn't reconcile NLB `eipAllocation` updates after creation.

candita · 2024-11-06T18:00:54Z

enhancements/ingress/set_eip_nlb_ingress.md

+   once it has been created. However, requiring a service deletion will effectuate these changes.
+
+   See [CCM Doesn't Reconcile NLB EIPAllocation Updates](#ccm-doesnt-reconcile-nlb-eipallocation-updates) for more details on why the CCM doesn't reconcile these updates.
+


Isn't there a fourth reason? e.g. to allow the cluster admin to schedule service interruptions during a maintenance window.

candita · 2024-11-06T18:03:56Z

enhancements/ingress/set_eip_nlb_ingress.md

+   See [CCM Doesn't Reconcile NLB EIPAllocation Updates](#ccm-doesnt-reconcile-nlb-eipallocation-updates) for more details on why the CCM doesn't reconcile these updates.
+
+   #####  Effectuating EIP Updates using automatic method:
+   If the admin wants to detect the change in `eipAllocations` and update the LB service automatically,


Suggested change

If the admin wants to detect the change in `eipAllocations` and update the LB service automatically,

If the cluster admin doesn't need to manage the service interruption, they can configure the LB service to be deleted automatically,

candita · 2024-11-06T18:04:50Z

enhancements/ingress/set_eip_nlb_ingress.md

+
+   #####  Effectuating EIP Updates using automatic method:
+   If the admin wants to detect the change in `eipAllocations` and update the LB service automatically,
+   rather than explicitly deleting the Service after changes, they can add the  `ingress.operator.openshift.io/auto-delete-load-balancer` annotation.


Suggested change

rather than explicitly deleting the Service after changes, they can add the `ingress.operator.openshift.io/auto-delete-load-balancer` annotation.

rather than explicitly deleting the Service after changes, by adding the `ingress.operator.openshift.io/auto-delete-load-balancer` annotation.

candita · 2024-11-06T18:12:38Z

enhancements/ingress/set_eip_nlb_ingress.md

+   While simpler in design, this approach could result in several minutes of unexpected disruption to ingress traffic. A cluster admin might not anticipate that updating
+   the Subnets field could cause disruption, leading to an unwelcome surprise.
+
+### 3. Any change to the service annotation does not trigger the operator to manage the load balancer service.


Formatting needs another "#".

Suggested change

### 3. Any change to the service annotation does not trigger the operator to manage the load balancer service.

#### 3. Spec and status can diverge

candita · 2024-11-06T18:15:18Z

enhancements/ingress/set_eip_nlb_ingress.md

+- If the above two points are not followed, the Kubernetes LoadBalancer service for the IngressController
+  won't get assigned EIPs,  which will cause the LoadBalancer service to be in persistently pending state by the
+  Cloud Controller Manager (CCM). The reason for the persistently pending state is posted to the status of the IngressController.
+  As of now we are not able to identify any more risks which can affect other components of the EP.


Remove. Or add after the last risk statement.

Suggested change

As of now we are not able to identify any more risks which can affect other components of the EP.

candita · 2024-11-06T18:17:39Z

enhancements/ingress/set_eip_nlb_ingress.md

+###  Effectuating EIP Updates using automatic method
+Please refer [subsection](#effectuating-eip-updates-using-automatic-method) for more details.


I think this is redundant unless you explain the risk explicitly.

candita · 2024-11-06T18:21:22Z

enhancements/ingress/set_eip_nlb_ingress.md

+1. Shall we support updating EIP in the default Classic `IngressController` after installation ?
+   Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).


Suggested change

1. Shall we support updating EIP in the default Classic `IngressController` after installation ?

Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).

1. Shall we support updating EIP in the default Classic `IngressController` after installation ?

Yes. Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).

candita · 2024-11-06T18:22:35Z

enhancements/ingress/set_eip_nlb_ingress.md

+   The load balancer would be created with no eipAllocations and the service annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` will be removed.
+   For more check [this](#cluster-admin-wants-operator-to-automatically-effectuate-the-load-balancer-service-for-updated-ingress-controller-eipallocations-)
+   AWS will randomly assign the NLB's IP addresses from the IP address pools available.


Rearrange the sentences:

Suggested change

The load balancer would be created with no eipAllocations and the service annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` will be removed.

For more check [this](#cluster-admin-wants-operator-to-automatically-effectuate-the-load-balancer-service-for-updated-ingress-controller-eipallocations-)

AWS will randomly assign the NLB's IP addresses from the IP address pools available.

The load balancer would be created with no eipAllocations and the service annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` will be removed. AWS will randomly assign the NLB's IP addresses from the IP address pools available.

For more details, see [this](#cluster-admin-wants-operator-to-automatically-effectuate-the-load-balancer-service-for-updated-ingress-controller-eipallocations-)

candita · 2024-11-06T18:23:56Z

enhancements/ingress/set_eip_nlb_ingress.md

+5. Can you assign `eipAllocations` to an `IngressController` with scope `Internal` ?
+
+    No, eipAllocations can be provided only for an `IngressController` with scope `External`.
+    The default IC is set to [External](https://github.com/openshift/cluster-ingress-operator/blob/master/pkg/operator/operator.go#L476)


This link isn't correct, it doesn't point to scope.

candita · 2024-11-06T18:25:21Z

enhancements/ingress/set_eip_nlb_ingress.md

+    However, we will validate the `IngressController` CR at API using CEL to check if `eipAllocations` are provided only when `scope` is 
+    set to `External`.


Suggested change

However, we will validate the `IngressController` CR at API using CEL to check if `eipAllocations` are provided only when `scope` is

set to `External`.

The `IngressController` CR cannot be saved if `eipAllocations` are provided and `scope` is not

set to `External`.

candita · 2024-11-06T18:35:25Z

enhancements/ingress/set_eip_nlb_ingress.md

+3. **Future Possibilities**: Should the AWS CCM eventually supports reliably modifying subnets without deleting the
+   LB-type service, we can then easily adapt the Ingress Operator to immediately effectuate the `eipAllocations` and eliminate
+   the need for a cluster admin to delete the LB-type service.


This is not a drawback, and isn't related to immutability. It is really its own Alternative.

Suggested change

3. **Future Possibilities**: Should the AWS CCM eventually supports reliably modifying subnets without deleting the

LB-type service, we can then easily adapt the Ingress Operator to immediately effectuate the `eipAllocations` and eliminate

the need for a cluster admin to delete the LB-type service.

### Future Possibility based on changes to AWS CCM.

Should the AWS CCM eventually supports reliably modifying subnets without deleting the

LB-type service, we can then easily adapt the Ingress Operator to immediately effectuate the `eipAllocations` and eliminate the need for a cluster admin to delete the LB-type service.

candita · 2024-11-06T18:39:26Z

/remove-lifecycle-rotten

Hi @miheer, I have completed a final review on this EP. It's mostly nits, but some areas need bigger changes. Afaik none of my suggestions would need to be propagated to the implementation. If/when you address my suggestions, can you please either put a 👍🏼 on each, or explain why you didn't accept the suggestion. That will make it easier for me to add the lgtm.

candita · 2024-11-06T18:55:46Z

/remove-lifecycle rotten

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 12, 2024

openshift-ci bot requested review from candita and frobware March 12, 2024 11:04

miheer force-pushed the eip branch 7 times, most recently from bc40070 to 4b14bb9 Compare March 22, 2024 09:46

miheer changed the title ~~WIP: Set EIP for NLB Ingress controller.~~ Set EIP for NLB Ingress controller. Mar 22, 2024

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 22, 2024

miheer changed the title ~~Set EIP for NLB Ingress controller.~~ NE-1517: Set EIP for NLB Ingress controller. Mar 22, 2024

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 22, 2024

miheer force-pushed the eip branch 3 times, most recently from 9e62c06 to 45ad01d Compare March 24, 2024 20:52

openshift-ci bot assigned candita, frobware and Miciah Mar 27, 2024

frobware reviewed Mar 28, 2024

View reviewed changes

Miciah reviewed Mar 28, 2024

View reviewed changes

candita reviewed Apr 1, 2024

View reviewed changes

candita reviewed Nov 5, 2024

View reviewed changes

candita reviewed Nov 6, 2024

View reviewed changes

openshift-ci bot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Nov 6, 2024


		## Upgrade / Downgrade Strategy

		During upgrade, user should be able to populate the eip_allocations_field via installer

	1. Cluster administrator creates adds the following in the install-config.yaml
	1. Cluster administrator creates or adds the following in the install-config.yaml

	After the changes are saved, the operator will delete and create the load balancer service automatically.
	After the changes are saved, the operator will automatically delete and create the load balancer service.


		#### Updating EIP in the default Classic IngressController after installation

		We support updating the EIP in the default Classic `IngressController`.

	#### 1. Create a new IngressController with EIPs:
	#### 1. When creating a customer IngressController with EIPs:


		#### 1. Create a new IngressController with EIPs:

		The admin sets the `eipAllocations` in the `IngressController` CR. The cluster ingress operator then creates a `IngressController` by:

	The admin sets the `eipAllocations` in the `IngressController` CR. The cluster ingress operator then creates a `IngressController` by:
	The cluster admin sets the `eipAllocations` in the `IngressController` CR. The cluster ingress operator then creates a new `IngressController` by:

	#### 2. Update the EIP for an existing IngressController:
	#### 2. When changing the EIPs for an existing IngressController:

-   ##### Effectuating EIP Updates using manual method:
+In order to provide the most flexibility to cluster admin operations, there are two methods that cover the modification of EIPs in an existing IngressController, managed and automatic.
+   ##### Managed

	When the admin changes the eipAllocations field in the `IngressController` CR, the ingress operator sets the Load Balancer
	When the cluster admin changes the eipAllocations field in the `IngressController` specification, the ingress operator sets the Load Balancer

		The cluster admin has to manually delete service in order to effectuate the change.
		Then step 1.2 is followed by the ingress operator.

	We have intentionally designed this feature to require manual deletion of the service for the following reasons:

	We have intentionally designed this feature to require manual deletion of the service for the following three reasons:

		While the AWS Cloud Controller Manager (CCM) handles invalid annotations by producing errors and events, it is difficult
		for the Ingress Operator to decipher these CCM events after the load balancer has been provisioned.

	malfunctioning state (i.e., the service is not getting reconciled) while the service is already provisioned.
	malfunctioning state (i.e., the service is not getting reconciled) while the service is being provisioned.

	the default value [] for EIPAllocations will clear the annotation for existing IngressControllers, which would break cluster ingress.
	the empty default value for EIPAllocations will clear the annotation for existing IngressControllers, which would break cluster ingress.

	Reason #3: The CCM Doesn't Reconcile NLB `eipAllocation` Updates after creation.
	Reason #3: The CCM doesn't reconcile NLB `eipAllocation` updates after creation.

		once it has been created. However, requiring a service deletion will effectuate these changes.

		See [CCM Doesn't Reconcile NLB EIPAllocation Updates](#ccm-doesnt-reconcile-nlb-eipallocation-updates) for more details on why the CCM doesn't reconcile these updates.

	If the admin wants to detect the change in `eipAllocations` and update the LB service automatically,
	If the cluster admin doesn't need to manage the service interruption, they can configure the LB service to be deleted automatically,

	rather than explicitly deleting the Service after changes, they can add the `ingress.operator.openshift.io/auto-delete-load-balancer` annotation.
	rather than explicitly deleting the Service after changes, by adding the `ingress.operator.openshift.io/auto-delete-load-balancer` annotation.

	### 3. Any change to the service annotation does not trigger the operator to manage the load balancer service.
	#### 3. Spec and status can diverge

NE-1517: Set EIP for NLB Ingress controller. #1593

Are you sure you want to change the base?

NE-1517: Set EIP for NLB Ingress controller. #1593

Conversation

miheer commented Mar 12, 2024

openshift-ci-robot commented Mar 22, 2024 • edited by openshift-ci bot Loading

candita commented Mar 27, 2024

frobware left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frobware Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

miheer Mar 29, 2024 • edited Loading

Choose a reason for hiding this comment

candita Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miheer commented Mar 29, 2024 • edited Loading

candita Apr 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

candita Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

candita Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci-robot commented Mar 22, 2024 •

edited by openshift-ci bot

Loading

frobware left a comment •

edited

Loading

frobware Mar 28, 2024 •

edited

Loading

miheer Mar 29, 2024 •

edited

Loading

candita Apr 1, 2024 •

edited

Loading

miheer commented Mar 29, 2024 •

edited

Loading

candita Apr 1, 2024 •

edited

Loading

candita Nov 6, 2024 •

edited

Loading

candita Nov 6, 2024 •

edited

Loading

		### Effectuating EIP Updates using automatic method
		Please refer [subsection](#effectuating-eip-updates-using-automatic-method) for more details.

		1. Shall we support updating EIP in the default Classic `IngressController` after installation ?
		Please refer to [section](#updating-eip-in-the-default-classic-ingresscontroller-after-installation).

		However, we will validate the `IngressController` CR at API using CEL to check if `eipAllocations` are provided only when `scope` is
		set to `External`.