Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ResourceLimits for test pods #253

Merged
merged 1 commit into from
Jan 13, 2025
Merged

Add ResourceLimits for test pods #253

merged 1 commit into from
Jan 13, 2025
+1,006 −4

Conversation

lpiwowar
Copy link
Collaborator

@lpiwowar lpiwowar commented Nov 25, 2024 •
edited
Loading

The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617

@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Nov 25, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

lpiwowar
lpiwowar commented Nov 25, 2024
View reviewed changes
Copy link
Collaborator Author

@lpiwowar lpiwowar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eshulman2, @eduolivares, @ashu-011: I would be really grateful ❤️ for your input here. This PR is about setting sane default resource limits [1] for the test pods. The limits are going to be different for each framework therefore I need an estimation from each of you, if possible:).

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

api/v1beta1/horizontest_types.go Show resolved Hide resolved
api/v1beta1/tobiko_types.go Show resolved Hide resolved
api/v1beta1/ansibletest_types.go Show resolved Hide resolved
@lpiwowar lpiwowar force-pushed the feature/resource-limit branch 2 times, most recently from 8827409 to 5147c15 Compare November 26, 2024 16:50
lpiwowar added a commit to openstack-k8s-operators/ci-framework that referenced this pull request Dec 18, 2024
@lpiwowar
[Test-operator] Set resource limits
cdcfd6c 
This PR exposes the resources parameter in for all test-operator
related CRs (Tempest, Tobiko, AnsibleTest, HorizonTest). This parameter
can be used to specify amount of resources the test pods spawned by the
test-operator should consume [1].

This commit intentionally unsets the value for Tempest test pods as
the propagation of the fix for Tempest memory leak bug [1] did not
reach the upstream openstack-tempest-all image yet. Once the fix is
in place the default value of cifmw_test_operator_tempest_resources
can be changed to {}.

[1] openstack-k8s-operators/test-operator#253
@lpiwowar lpiwowar mentioned this pull request Dec 18, 2024
Merged
@lpiwowar lpiwowar force-pushed the feature/resource-limit branch from 5147c15 to 8e66c54 Compare December 18, 2024 14:51
@softwarefactory-project-zuul softwarefactory-project-zuul
Copy link

This change depends on a change that failed to merge.

Change openstack-k8s-operators/ci-framework#2617 is needed.

@lpiwowar
Copy link
Collaborator Author

recheck

@lpiwowar lpiwowar mentioned this pull request Dec 18, 2024
Closed
lpiwowar added a commit to openstack-k8s-operators/ci-framework that referenced this pull request Dec 19, 2024
@lpiwowar
[Test-operator] Set resource limits
356c19b 
This PR exposes the resources parameter in for all test-operator
related CRs (Tempest, Tobiko, AnsibleTest, HorizonTest). This parameter
can be used to specify amount of resources the test pods spawned by the
test-operator should consume [1].

This commit intentionally unsets the value for Tempest test pods as
the propagation of the fix for Tempest memory leak bug [1] did not
reach the upstream openstack-tempest-all image yet. Once the fix is
in place the default value of cifmw_test_operator_tempest_resources
can be changed to {}.

[1] openstack-k8s-operators/test-operator#253
@lpiwowar
Copy link
Collaborator Author

/test all

@lpiwowar lpiwowar force-pushed the feature/resource-limit branch 2 times, most recently from 9a14c77 to b157f44 Compare December 19, 2024 11:04
@ashu-011 ashu-011 mentioned this pull request Jan 6, 2025
Closed
@lpiwowar
Copy link
Collaborator Author

lpiwowar commented Jan 9, 2025

/test all

openshift-merge-bot bot pushed a commit to openstack-k8s-operators/ci-framework that referenced this pull request Jan 9, 2025
@lpiwowar @openshift-merge-bot
[Test-operator] Set resource limits
deb0100 
This PR exposes the resources parameter in for all test-operator
related CRs (Tempest, Tobiko, AnsibleTest, HorizonTest). This parameter
can be used to specify amount of resources the test pods spawned by the
test-operator should consume [1].

This commit intentionally unsets the value for Tempest test pods as
the propagation of the fix for Tempest memory leak bug [1] did not
reach the upstream openstack-tempest-all image yet. Once the fix is
in place the default value of cifmw_test_operator_tempest_resources
can be changed to {}.

[1] openstack-k8s-operators/test-operator#253
@lpiwowar lpiwowar marked this pull request as ready for review January 10, 2025 10:08
@lpiwowar lpiwowar requested review from kstrenkova and removed request for lewisdenny and kopecmartin January 10, 2025 10:08
kstrenkova
kstrenkova requested changes Jan 10, 2025
View reviewed changes
Copy link
Contributor

@kstrenkova kstrenkova left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested all 4 CRs and they seem to be working. The default values are corresponding to the ones set in code and when changing resource values in CR, the change is propagated into the pod itself.

There is only one request I have regarding this patch and that is adding examples into all 4 CRs of the new parameter Resources. Like so:

# resources:
#  limits:
#    cpu: 8000m
#    memory: 4Gi
#  requests:
#    cpu: 4000m
#    memory: 2Gi

It lets users know there is an option to change the default resources and gives them overview what the default values are. In my opinion at least 😄

@lpiwowar
Copy link
Collaborator Author

@kstrenkova that's a good idea. Keeping the docs up to date is definitely helpful! Thanks for the review!

I will update the documentation.

@lpiwowar
Add ResourceLimits for test pods
bc47b78 
The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617
@lpiwowar lpiwowar force-pushed the feature/resource-limit branch from b157f44 to bc47b78 Compare January 13, 2025 08:33
kstrenkova
kstrenkova approved these changes Jan 13, 2025
View reviewed changes
Copy link
Contributor

@kstrenkova kstrenkova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding the examples to CRs and documentation! I think now the patch is complete and working, so lgtm.

@kstrenkova
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm label Jan 13, 2025
@openshift-ci OpenShift CI
Copy link

openshift-ci bot commented Jan 13, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kstrenkova, lpiwowar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit be35b07 into openstack-k8s-operators:main Jan 13, 2025
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ashu-011 ashu-011 ashu-011 left review comments

@kstrenkova kstrenkova kstrenkova approved these changes

Assignees

@kstrenkova kstrenkova

Labels
approved lgtm
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@lpiwowar @kstrenkova @ashu-011