CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver #2520

tjungblu · 2025-10-07T14:56:49Z

This adds a minute/hour configuration duration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O.

Corresponding enhancement in
openshift/enhancements#1857

openshift-ci-robot · 2025-10-07T14:56:52Z

@tjungblu: This pull request references CNTRLPLANE-1576 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

This adds a minute/hour configuration duration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-10-07T14:57:50Z

Hello @tjungblu! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

openshift-ci · 2025-10-08T14:08:12Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joelspeed for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

JoelSpeed · 2025-10-08T14:15:28Z

operator/v1/types_kubeapiserver.go

+	// The value must be parseable as a time duration value;
+	// see <https://pkg.go.dev/time#ParseDuration>.


We prefer not to use duration values anymore. Instead, create a int32 type, with units in the name

For example, this should be eventTTLMinutes.

We do this because not all clients are built in Go, and building a Go compatible duration parsing in other languages is not going to be fun

thanks for reminding me. I've changed this to be configurable in minutes via int32.

JoelSpeed · 2025-10-08T14:18:27Z

operator/v1/types_kubeapiserver.go

+	// If configured, it must be a value of 1m (one minute) or greater, we only allow setting
+	// minute and hour durations (e.g. 5m or 5h).


We should set a sensible maximum value for this TTL. Can you suggest a sensible maximum value for this?

yep, made it 2x the default of 3h now

JoelSpeed · 2025-10-08T14:23:30Z

operator/v1/types_kubeapiserver.go

+	// +kubebuilder:validation:Format=duration
+	// +kubebuilder:validation:Pattern=^(0|([0-9]+(\.[0-9]+)?(m|h))+)$
+	// +kubebuilder:validation:Type:=string
+	// +default="3h"


Since this is a configuration API, we want to reserve the right to change this value over time. This will be easier if we don't set the value through openapi defaulting.

It's better here to set the value in the controller by detecting that it is omitted.

We generally add a comment to the godoc to explain this scenario

// When omitted this means no opinion, and the platform is left to choose a reasonable default, which is subject to change over time. // The current default value is 3h.

added this, great idea

JoelSpeed · 2025-10-08T15:11:46Z

operator/v1/types_kubeapiserver.go

 type KubeAPIServerSpec struct {
 	StaticPodOperatorSpec `json:",inline"`
+
+	// eventTTL specifies the amount of time that the events are stored before being deleted.


Why would a user choose either a larger or smaller value for this? Is there a performance benefit? What is too large? What are the impacts of choosing something very small?

How deep do we want to document this on the godoc vs. the actual openshift documentation?
Or is this just for your personal curiosity?

I think we should give at least some guidance as to what the effect is of choosing a smaller or larger value, doesn't have to be super deep, but if we can make the API self service that's preferable

My main concern is that someone sees they can change it to 1m in the oc explain, so does so, without us giving them any hint as to potential issues this short feedback loop could introduce

I've left a few words about lowering it, since in the other thread we left the 3h default as the maximum.

Let me know if a deeper explanation is required here, I'd rather not duplicate content too much and it diverges real quick as the docs team will not update the API along any docs bug coming in...

benluddy · 2025-10-08T15:39:20Z

operator/v1/types_kubeapiserver.go

+	//
+	// +kubebuilder:default:=180
+	// +kubebuilder:validation:Minimum=30
+	// +kubebuilder:validation:Maximum=360


Do we have users that need to increase the event TTL above 3h? Choosing 3h as the maximum would preserve the upper bound when it comes to benchmarking and publishing supported cluster sizes.

For reference, upstream default TTL is 1h and it is already considering very high for today's cluster sizes.

The only reason we have 3h downstream as the default value is because origin jobs have a 3h timeout and this allows to retain the events for the entire duration of the tests.

that would also be a case-in-point to have it configurable for longer periods in our CI. I've changed the upper bound to 3h again.

JoelSpeed · 2025-10-08T15:45:49Z

operator/v1/types_kubeapiserver.go

 	StaticPodOperatorSpec `json:",inline"`
+
+	// eventTTLMinutes specifies the amount of time that the events are stored before being deleted.
+	// This setting is allowed between 30 minutes minimum up to 6h (360 minutes).


Because the field value is an integer, it would be better to reverse these

Suggested change

// This setting is allowed between 30 minutes minimum up to 6h (360 minutes).

// The TTL is allowed between 30 minutes minimum up to a maximum of 360 minutes (6 hours).

JoelSpeed · 2025-10-08T15:46:06Z

operator/v1/types_kubeapiserver.go

+	// When omitted this means no opinion, and the platform is left to choose a reasonable default, which is subject to change over time.
+	// The current default value is 3h (180 minutes).
+	//
+	// +kubebuilder:default:=180


I left a comment on the old default, we shouldn't have the openapi default like this

so removing ALL defaults or leaving this default and adding the other default back?

removed all defaults now

JoelSpeed · 2025-10-08T15:46:37Z

Is there an EP for this new field? It should be added behind a feature gate

dgrisonnet · 2025-10-08T15:48:13Z

operator/v1/types_kubeapiserver.go

+	// The current default value is 3h (180 minutes).
+	//
+	// +kubebuilder:default:=180
+	// +kubebuilder:validation:Minimum=30


I think we can go as far down as 5 minutes. This would allow customers that persists events externally to reduce the amount of events stored in their cluster as much as possible while still being able to know what is happening right now in their cluster.

Also, when there is an event spew, the only way to recover the cluster quickly is to reduce the event-ttl as much as possible. For that some people set the ttl to 0 in vanilla Kubernetes, but I think that for debug purposes, it would still be useful to have some events around, so 5 minutes is better IMO.

For that some people set the ttl to 0 in vanilla Kubernetes

That would be bad, because 0 will not attach a lease on the object, so it will never be deleted :)

But I'm cool with 5 minutes, updating the api and enhancement accordingly.

I assume that we’ll assess the trade-off of a lower TTL on resources such as CPU before promoting this setting to GA. If we get it wrong, we can always bump the value later.

yep, I'm looking into this right now, there's going to be a section in the enhancement. There are also some data points already, if you want to take a look

tjungblu · 2025-10-09T07:43:25Z

@JoelSpeed yes, there is: openshift/enhancements#1857

We can add it behind a feature gate, if required.

This adds a minute based configuration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O. Signed-off-by: Thomas Jungblut <[email protected]>

openshift-ci-robot · 2025-10-09T08:08:49Z

@tjungblu: This pull request references CNTRLPLANE-1576 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

This adds a minute/hour configuration duration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O.

Corresponding enhancement in
openshift/enhancements#1857

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

JoelSpeed · 2025-10-09T10:01:22Z

Ok, API changes LGTM, but lets get this behind a feature gate and go through that process

tjungblu · 2025-10-09T10:11:43Z

Cool thanks, I'll open a separate one for the feature gate.

/hold
until then

tjungblu · 2025-10-09T10:24:20Z

added the gate in here: #2525

openshift-ci · 2025-10-09T10:57:47Z

@tjungblu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`61c6605`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

API PR in openshift/api#2520 Feature Gate PR in openshift/api#2525 Signed-off-by: Thomas Jungblut <[email protected]>

p0lyn0mial · 2025-10-10T12:15:04Z

operator/v1/types_kubeapiserver.go

+	// +kubebuilder:validation:Minimum=5
+	// +kubebuilder:validation:Maximum=180
+	// +optional
+	EventTTLMinutes int32 `json:"eventTTLMinutes,omitempty"`


mhm, what happens if an admin sets that field, but there is a controller that is using an old generated client that doesn’t know about it and that controller updates the spec?

is this something we should/usually take into account?

This is a general issue with adding new fields to any API. In this case, depending on the type of request they make (update vs patch), they would likely clear the field and then the behaviour would be reverted, it's not a catastrophic change of the behaviour so isn't going to burn the cluster to the ground.

If you wanted to, you could make this field un-removable, but I don't think that's really the right use case for this field

out of curiosity, how could one make a field unremovable?

You need a chain of "un-removability" from the nearest ancestor that is always required

// +kubebuilder:validation:XValidation:rule="!oldSelf.?child.hasValue() || self.?child.hasValue()",optionalOldSelf=true,message="child may not be removed once set"

This would go on the struct definition of each ancestor til you find the nearest always required ancestor

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 7, 2025

openshift-ci bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 7, 2025

openshift-ci bot requested review from JoelSpeed and everettraven October 7, 2025 15:01

tjungblu mentioned this pull request Oct 8, 2025

CNTRLPLANE-1575: Add support for event-ttl in Kube API Server Operator openshift/enhancements#1857

Open

tjungblu force-pushed the CNTRLPLANE-1576 branch from b24d8da to e816e63 Compare October 8, 2025 14:07

JoelSpeed reviewed Oct 8, 2025

View reviewed changes

tjungblu force-pushed the CNTRLPLANE-1576 branch from e816e63 to 6054d58 Compare October 8, 2025 15:22

benluddy reviewed Oct 8, 2025

View reviewed changes

JoelSpeed reviewed Oct 8, 2025

View reviewed changes

dgrisonnet reviewed Oct 8, 2025

View reviewed changes

CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver

61c6605

This adds a minute based configuration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O. Signed-off-by: Thomas Jungblut <[email protected]>

tjungblu force-pushed the CNTRLPLANE-1576 branch from 6054d58 to 61c6605 Compare October 9, 2025 08:08

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 9, 2025

tjungblu added a commit to tjungblu/enhancements that referenced this pull request Oct 9, 2025

CNTRLPLANE-1575: Add support for event-ttl in Kube API Server Operator

3fb6e12

API PR in openshift/api#2520 Feature Gate PR in openshift/api#2525 Signed-off-by: Thomas Jungblut <[email protected]>

tjungblu added a commit to tjungblu/enhancements that referenced this pull request Oct 10, 2025

CNTRLPLANE-1575: Add support for event-ttl in Kube API Server Operator

3a99db9

API PR in openshift/api#2520 Feature Gate PR in openshift/api#2525 Signed-off-by: Thomas Jungblut <[email protected]>

p0lyn0mial reviewed Oct 10, 2025

View reviewed changes

		// The value must be parseable as a time duration value;
		// see <https://pkg.go.dev/time#ParseDuration>.

		// If configured, it must be a value of 1m (one minute) or greater, we only allow setting
		// minute and hour durations (e.g. 5m or 5h).

	// This setting is allowed between 30 minutes minimum up to 6h (360 minutes).
	// The TTL is allowed between 30 minutes minimum up to a maximum of 360 minutes (6 hours).

CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver #2520

Are you sure you want to change the base?

CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver #2520

Uh oh!

Conversation

tjungblu commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 7, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Oct 7, 2025

Uh oh!

openshift-ci bot commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoelSpeed commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjungblu commented Oct 9, 2025

Uh oh!

openshift-ci-robot commented Oct 9, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoelSpeed commented Oct 9, 2025

Uh oh!

tjungblu commented Oct 9, 2025

Uh oh!

tjungblu commented Oct 9, 2025

Uh oh!

openshift-ci bot commented Oct 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tjungblu commented Oct 7, 2025 •

edited

Loading

openshift-ci-robot commented Oct 7, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Oct 9, 2025 •

edited by openshift-ci bot

Loading