Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix IPU being blocked by resource limitations #1256

Merged
merged 1 commit into from
Nov 7, 2024

Conversation

tomasfratrik
Copy link
Member

First resource limit is maximum number of open file descriptors limit, second one being limit for maximum writable file size.

With resource lib we set the limit for the current process (tested with my testing scripts) and its children processes. By testing pre-upgrade and upgrade I found a threshold of max. open file descriptors to be around 512, (the default value for ulimit -n is 1024) so a small multiplier of the default value shouldn't be a problem to set. With writing size there shouldn't be a problem to set it to "infinity".

Both problems from the tickets were solved, but there could be other side effects.

Resolves: RHEL-26459 and RHEL-16881

Copy link

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

  • review please @oamg/developers to notify leapp developers of the review request
  • /packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build.
However, here are additional useful commands for packit:

  • /packit test to re-run manually the default tests
  • /packit retest-failed to re-run failed tests manually
  • /packit test oamg/leapp#42 to run tests with leapp builds for the leapp PR#42 (default is latest upstream - master - build)

Note that first time contributors cannot run tests automatically - they need to be started by a reviewer.

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones.
To launch on-demand tests with packit:

  • /packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
  • /packit test --labels beaker-minimal-8.10to9.4,kernel-rt-8.10to9.4 to schedule kernel-rt and beaker-minimal test sets for 8.10->9.4 upgrade path

See other labels for particular jobs defined in the .packit.yaml file.

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

@tomasfratrik tomasfratrik marked this pull request as draft June 20, 2024 08:22
@tomasfratrik tomasfratrik changed the title Fix IPU being blocked by resource limitations [DRAFT] Fix IPU being blocked by resource limitations Jun 20, 2024
@tomasfratrik tomasfratrik force-pushed the resource-fix branch 2 times, most recently from ae14f2e to 1bfaa9b Compare June 24, 2024 08:50
@tomasfratrik tomasfratrik changed the title [DRAFT] Fix IPU being blocked by resource limitations Fix IPU being blocked by resource limitations Jun 24, 2024
@tomasfratrik tomasfratrik marked this pull request as ready for review June 24, 2024 10:55
@matejmatuska
Copy link
Member

/packit copr-build

@fernflower
Copy link
Member

/packit test

@tomasfratrik
Copy link
Member Author

I don't really like the redundancy of code here, but I'm not sure where should the common logic be placed.

@matejmatuska
Copy link
Member

I don't really like the redundancy of code here, but I'm not sure where should the common logic be placed.

I think commands/command_utils.py is an option. There is already some code the commands share.

@matejmatuska
Copy link
Member

/packit retest-failed

@tomasfratrik tomasfratrik force-pushed the resource-fix branch 2 times, most recently from 87f52e4 to e4afdea Compare July 2, 2024 09:16
@tomasfratrik
Copy link
Member Author

I don't really like the redundancy of code here, but I'm not sure where should the common logic be placed.

I think commands/command_utils.py is an option. There is already some code the commands share.

Thanks, I put it there

@pirat89
Copy link
Member

pirat89 commented Jul 2, 2024

/packit copr-build

@pirat89 pirat89 added this to the 8.10/9.6 milestone Jul 2, 2024
@pirat89
Copy link
Member

pirat89 commented Jul 4, 2024

all standard tests passed. executing extended tests

@pirat89
Copy link
Member

pirat89 commented Jul 4, 2024

/packit test

@pirat89
Copy link
Member

pirat89 commented Jul 4, 2024

ah.. I thought it's different PR :D mama mia!

@pirat89 pirat89 removed this from the 8.10/9.6 milestone Sep 21, 2024
@pirat89 pirat89 added the bug Something isn't working label Oct 25, 2024
@pirat89 pirat89 added this to the 8.10/9.6 milestone Oct 25, 2024
@tomasfratrik tomasfratrik force-pushed the resource-fix branch 2 times, most recently from 67a4af0 to 1b9ecbe Compare October 30, 2024 12:33
@matejmatuska
Copy link
Member

/packit retest-failed

Copy link
Member

@matejmatuska matejmatuska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested for both maximum file size and maximum open fd's, works as expected.
Tested by setting a limit low enough that leapp fails without this patch and confirming it then works with it.

However unit tests are missing.

First resource limit is maximum number of open file descriptors limit,
second one being limit for maximum writable file size. Plus add unit
tests.

Resolves: RHEL-26459 and RHEL-16881
@tomasfratrik
Copy link
Member Author

Latest push squashes commits.

@tomasfratrik
Copy link
Member Author

Unit tests were added.

Copy link
Member

@matejmatuska matejmatuska left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

NOTE: If this breaks local tests in containers for you, just rebuild them (make clean_containers and then it will be rebuilt the next time you run make test_container)

@matejmatuska matejmatuska merged commit 5e6d176 into oamg:main Nov 7, 2024
25 of 26 checks passed
@matejmatuska matejmatuska added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants