Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Test-operator] Set resource limits #2617

Merged
merged 1 commit into from
Jan 9, 2025
Merged

Conversation

lpiwowar
Copy link
Contributor

This PR exposes the resources parameter in for all test-operator related CRs (Tempest, Tobiko, AnsibleTest, HorizonTest). This parameter can be used to specify amount of resources the test pods spawned by the test-operator should consume [1].

This commit intentionally unsets the value for Tempest test pods as the propagation of the fix for Tempest memory leak bug [1] did not reach the upstream openstack-tempest-all image yet. Once the fix is in place the default value of cifmw_test_operator_tempest_resources can be changed to {}.

[1] openstack-k8s-operators/test-operator#253

Copy link
Contributor

openshift-ci bot commented Dec 18, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

lpiwowar added a commit to lpiwowar/test-operator that referenced this pull request Dec 18, 2024
The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/fc5f70affacb4ec580c48b2da4f76f7d

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 58m 22s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 06s
cifmw-crc-podified-edpm-baremetal FAILURE in 56m 24s
✔️ cifmw-multinode-tempest SUCCESS in 1h 47m 27s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 13s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 10s
✔️ build-push-container-cifmw-client SUCCESS in 20m 48s
✔️ cifmw-molecule-test_operator SUCCESS in 4m 07s

This PR exposes the resources parameter in for all test-operator
related CRs (Tempest, Tobiko, AnsibleTest, HorizonTest). This parameter
can be used to specify amount of resources the test pods spawned by the
test-operator should consume [1].

This commit intentionally unsets the value for Tempest test pods as
the propagation of the fix for Tempest memory leak bug [1] did not
reach the upstream openstack-tempest-all image yet. Once the fix is
in place the default value of cifmw_test_operator_tempest_resources
can be changed to {}.

[1] openstack-k8s-operators/test-operator#253
lpiwowar added a commit to lpiwowar/test-operator that referenced this pull request Dec 19, 2024
The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617
lpiwowar added a commit to lpiwowar/test-operator that referenced this pull request Dec 19, 2024
The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/181fda8a469f465cb893859277aeed63

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 53m 28s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 18m 15s
cifmw-crc-podified-edpm-baremetal FAILURE in 51m 52s
✔️ cifmw-multinode-tempest SUCCESS in 1h 41m 26s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 7m 52s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 41s
✔️ build-push-container-cifmw-client SUCCESS in 37m 03s
✔️ cifmw-molecule-test_operator SUCCESS in 3m 56s

@lpiwowar
Copy link
Contributor Author

recheck

Copy link

github-actions bot commented Jan 4, 2025

This PR is stale because it has been for over 15 days with no activity.
Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the Stale label Jan 4, 2025
ashu-011 pushed a commit to ashu-011/test-operator that referenced this pull request Jan 6, 2025
The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617
@lpiwowar lpiwowar marked this pull request as ready for review January 8, 2025 10:30
@lpiwowar
Copy link
Contributor Author

lpiwowar commented Jan 8, 2025

recheck

Let's see that nothing got broken since the PR's creation.

Copy link

@evallesp evallesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One non-blocking minor suggestion. LGTM in general.

@@ -65,6 +65,7 @@ cifmw_test_operator_stages:
* `cifmw_test_operator_tempest_extra_rpms`: (List) . A list of URLs that point to RPMs that should be installed before the execution of tempest. Note that this parameter has no effect when `cifmw_test_operator_tempest_external_plugin` is used. Default value: `[]`
* `cifmw_test_operator_tempest_extra_configmaps_mounts`: (List) A list of configmaps that should be mounted into the tempest test pods. Default value: `[]`
* `cifmw_test_operator_tempest_debug`: (Bool) Run Tempest in debug mode, it keeps the operator pod sleeping infinity (it must only set to `true`only for debugging purposes). Default value: `false`
* `cifmw_test_operator_tempest_resources`: (Dict) A dictionary that specifies resources (cpu, memory) for the test pods. When untouched it clears the default values set on the test-operator side. This means that the tempest test pods run with unspecified resource limits. Default value: `{requests: {}, limits: {}}`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(non-blocking) suggestion: WDYT about adding the service before "test pods": operator test pods, tobiko test pods....

@evallesp
Copy link

evallesp commented Jan 9, 2025

/lgtm

@frenzyfriday
Copy link
Collaborator

/lgtm

@pablintino
Copy link
Collaborator

/approve

Copy link
Contributor

openshift-ci bot commented Jan 9, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pablintino

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved label Jan 9, 2025
@openshift-merge-bot openshift-merge-bot bot merged commit deb0100 into main Jan 9, 2025
6 checks passed
@openshift-merge-bot openshift-merge-bot bot deleted the resources-param branch January 9, 2025 16:26
lpiwowar added a commit to lpiwowar/test-operator that referenced this pull request Jan 13, 2025
The test pods spawned by the test operator prior to this change were
executed without any resource limits. This ultimately meant that the
pods could consume an unlimited amount of resources until the limit for
a worker node was reached.

This commit introduces default resource limits for each test pod
spawned by the test operator (tempest, tobiko, ansibletest, horizontest
pod). The default value can be overridden using .Spec.Resources
parameter which is introduced as part of this commit as well [1]:

spec:
  resources:
    requests:
      memory: 1Gi
      cpu:    250m
    limits:
      memory: 2Gi
      cpu:    500M

[1] https://pkg.go.dev/k8s.io/api/core/v1#ResourceRequirements

Depends-On: openstack-k8s-operators/ci-framework#2617
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants