Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

github: run system-tests on larger runners (8cores+32G) #334

Merged
merged 15 commits into from
Jul 10, 2024

Conversation

simondeziel
Copy link
Member

@simondeziel simondeziel commented Jun 26, 2024

Despite the additional HW resources, the test run-times didn't improved drastically but hopefully this should be more robust. This should also open the door to doing heavier tests without running into space or RAM limits. The CPU overcommit ratio is also better thus reducing the steal time.

Test run-times can be compared with https://github.com/canonical/microcloud/actions/runs/9783672548/job/27012801592

Note: the System (1.22.x, interactive) still takes ~3 hours and sometimes fail, same as with the regular runners :/

@simondeziel
Copy link
Member Author

Things are way more flaky on 24.04 it seems. The wrong/unexpected disk is often picked causing multiple failures. I'm a bit unsure how to improve that so if anyone has an idea, please let me know.

@simondeziel
Copy link
Member Author

A new failure reason:

10.223.175.0/24
                                                 # workaround for set -e
'
+ sed '/^\s*#/d; s/\s*#.*//; /^$/d'
Error: Failed to bootstrap local MicroCeph: Failed to run post-bootstrap actions: Failed to start monitor: Failed to run: snapctl start microceph.mon --enable: exit status 1 (error: error running snapctl: snap "microceph" has "auto-refresh" change in progress)
+ cleanup

@simondeziel simondeziel force-pushed the large-runners branch 2 times, most recently from eb06085 to 710531c Compare July 10, 2024 14:45
simondeziel and others added 12 commits July 10, 2024 13:55
Signed-off-by: Max Asnaashari <[email protected]>
Signed-off-by: Simon Deziel <[email protected]>
Ceph encryption will need this.

Signed-off-by: Simon Deziel <[email protected]>
This will be a requirement for Ceph encryption as it needs the `dm_crypt` module that's not enabled in 22.04's -kvm kernel.

Signed-off-by: Simon Deziel <[email protected]>
This also avoids any failure during the `setup` phase to trigger the cleanup which runs:

```
  + lxc list --all-projects
...
   + lxc exec micro01 -- lxc list
  Installing LXD snap, please be patient.
  If this is your first time running LXD on this machine, you should also run: lxd init
  To start your first container, try: lxc launch ubuntu:22.04
  Or for a virtual machine: lxc launch ubuntu:22.04 --vm

  +------+-------+------+------+------+-----------+
  | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
  +------+-------+------+------+------+-----------+
```

Signed-off-by: Simon Deziel <[email protected]>
@simondeziel simondeziel marked this pull request as ready for review July 10, 2024 21:20
@simondeziel
Copy link
Member Author

@masnax thanks for #341 it made it reliable again, apparently even with 24.04 as all tests succeeded on their first try.

@masnax masnax merged commit 7af979f into canonical:main Jul 10, 2024
15 checks passed
@simondeziel simondeziel deleted the large-runners branch July 10, 2024 21:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants