Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Add 2024 versions to tests #660

Conversation

laurent-indermuehle
Copy link
Collaborator

@laurent-indermuehle laurent-indermuehle commented Jul 17, 2024

SUMMARY

fixes #624
fixes #625

Removes tests for specific version of Python

why:

  • Bug for a specific version of Python are rare in Integration tests
  • This allow us to use ubuntu2204 instead of unbun2004. This is a limitation of ansible-test. The alternative would have been to use custom container, but we removes ours 3 weeks ago because it was too much maintenance.
  • Ubuntu 22.04 can connect to MySQL 8.4. This is not the case for ubuntu 20.04.
  • Reduce the number of tests
  • It means we only use Python 3.10 (until ansible-test update its images).
  • Fix CI for MySQL 8.0 that occured due to an update in ubuntu 20.04 last week (see bellow)

Remove tests for

  • MySQL 5.7.40 that is EoL
  • MariaDB 10.4.27 that is EoL
  • MariaDB 10.6.11, reason being that usually breaking changes introduced in the versions in between are also present in the latest version.
  • PyMySQL 0.7.11 (it is very very old and we can't have too many tests)

Add tests for

  • RHEL8 context: pymysql:0.10.1, ansible:2.16
  • MariaDB 10.11.8
  • PyMySQL 1.1.1

Update tests

  • MySQL 8.0.31 -> 8.0.38
  • MariaDB 10.5.18 -> 10.5.25

Number of VM

  • Before this PR; integration tests used 34 VMs.
  • After this PR; integration tests use 36 VMs.

Note for MySQL 8.4

Since MySQL 8.2, the mysql_native_plugin is absent. I modified the Makefile and GHA workflow to add the option to enable it back. Otherwise our tests fails to connect to the database.

Other tests are now failing due to deprecated option between 8.0 and 8.4 but I intend to let the tests red here and open a new PR to fix issues with MySQL 8.4 later.

Note for MySQL 8.0

One week ago, our CI stopped working. This is because Ubuntu 20.04 got updated for security reasons and now mysql-client complains that 'cryptography' package is missing. The upgrade to ubuntu 22.04 fix that.

Note for MySQL 9 and MariaDB 11

None of the latest db engine have a LTS release. So we will wait a little longer before testing those versions.

ISSUE TYPE

  • Tests

COMPONENT NAME

  • CI

Copy link

codecov bot commented Jul 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.18%. Comparing base (4912f1a) to head (751830c).
Report is 1 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #660       +/-   ##
===========================================
+ Coverage   65.51%   76.18%   +10.66%     
===========================================
  Files          20       19        -1     
  Lines        2627     2624        -3     
  Branches      673      673               
===========================================
+ Hits         1721     1999      +278     
+ Misses        697      426      -271     
+ Partials      209      199       -10     
Flag Coverage Δ
integration 75.49% <ø> (+10.13%) ⬆️
sanity ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

This is puzzling me. Why when using ansible devel the python command
changes? I know ansible-test install python after starting ubuntu22.04
so the way python is install must changes.
Comment on lines +63 to +68
- '0.9.3'
- '1.0.2'
- '1.1.1'
- '2.0.1'
- '2.0.3'
- '2.1.1'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- '0.9.3'
- '1.0.2'
- '1.1.1'
- '2.0.1'
- '2.0.3'
- '2.1.1'
- '0.9.3'
- '1.0.2'
- '1.1.1'
- '2.0.1'
- '2.0.3'
- '2.1.1'

can we also apply same oldest-supported-and-newest strategy here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Best way to install PyMySQL is to use the distribution's package manager.
The version depends on the will of the APT/RPM maintainers:

  • Ubuntu 22.04 : 1.0.2
  • RHEL8 : 0.10.1
  • RHEL9 : 1.1.1
  • Fedora39 : 1.1.0-3 and 1.1.1-1

So we can't say: "This 1.0.2 bug was fixed in 1.1.1, please update", if their distribution doesn't provide such package.

We won't save that many VMs anyway. And PyMySQL is not updated that often: https://pypi.org/project/PyMySQL/#history

When reading log we tend to look at the bottom, but doing so we find
often a idempotent error that are nothing to do with the first error.
Disabling this can greatly speedup tests and makes logs more readable.

Plus, now GHA jumps automatically at the latest error message. So with
this modification, we will always jump to the latest real error message.
We can't expand the left column on GHA, so the shorter, the better.
Use Ⓐ instead of Ansible.
@laurent-indermuehle laurent-indermuehle marked this pull request as ready for review July 18, 2024 12:59
@laurent-indermuehle
Copy link
Collaborator Author

Ready for review please.
The only failed test are for MySQL 8.4 and will be fixed later.

@laurent-indermuehle laurent-indermuehle merged commit c503dc5 into ansible-collections:main Jul 19, 2024
40 of 46 checks passed
@laurent-indermuehle laurent-indermuehle deleted the lie_add_2024_versions_to_tests branch July 19, 2024 09:04
laurent-indermuehle added a commit to laurent-indermuehle/community.mysql that referenced this pull request Jul 19, 2024
* Enable mysql_native_password for MySQL 8.2+
* Fix connection to MySQL 8 since Ubuntu 20.04 update
* Cut mysqlclient form the documentation
* Cut tests for Python 3.12 not supported by ansible-test
* Upgrade integration controller to ubuntu2204 by removing python

ansible-test uses python 3.10 if we specify ubuntu2204. Thus we lose the
ability to chose specific version of python to test. But integrations
tests are optional for a collection. And we don't catch a issue with
Python that often (ever ? I don't recall seen one).

This allow us to test MySQL 8.4, so it's a win.

* Cut tests for EoL MariaDB 10.4
* Reduce number of test in the matrix
* Cut support for intermediate LTS
* Fix python command not found with ansible-devel and add the debug

This is puzzling me. Why when using ansible devel the python command
changes? I know ansible-test install python after starting ubuntu22.04
so the way python is install must changes.

* Disable retry-on-error

When reading log we tend to look at the bottom, but doing so we find
often a idempotent error that are nothing to do with the first error.
Disabling this can greatly speedup tests and makes logs more readable.

Plus, now GHA jumps automatically at the latest error message. So with
this modification, we will always jump to the latest real error message.

* Enhance jobs title readability

We can't expand the left column on GHA, so the shorter, the better.
Use Ⓐ instead of Ansible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants