[Ubuntu] Stage 1 and Stage 2 errors due to python 3.12 being introduced

[dnegreira]

Now Ubuntu Noble (24.04) ships with Python 3.12 and we are seeing a bunch of errors on Stage 1 and Stage 2 testing.

Stage 1 full logs

Stage 1 Errors:

(066/161) tests/unittests/option_tests.py:GlobalOptionTest.test_cascade:  ERROR: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) (28.89 s)
 (065/161) tests/unittests/option_tests.py:GlobalOptionTest.test_simple_lookup:  ERROR: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) (29.03 s)

and

 (115/161) tests/unittests/policy_tests.py:DpkgPackageManagerTests.test_load_all_packages:  ERROR: AttributeError: 'DpkgPackageManagerTests' object has no attribute 'assertNotEquals'. Did you mean: 'assertNotEqual'? (0.20 s)
 (117/161) tests/unittests/policy_tests.py:MultiPackageManagerTests.test_load_all_packages:  ERROR: AttributeError: 'MultiPackageManagerTests' object has no attribute 'assertNotEquals'. Did you mean: 'assertNotEqual'? (0.20 s)

Stage 2 full logs
Stage 2 Errors:

PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --test-runner=runner -t stagetwo tests/{cleaner,collect,report,vendor}_tests
/tmp/cirrus-ci-build/tests/sos_tests.py:427: SyntaxWarning: invalid escape sequence '\.'
  return re.findall('/.*sosreport-.*tar.*\.gpg', self.cmd_output.stdout)[-1]
[..... lots of similar errors removed for brevity's sake .... ]

and

Avocado crashed: TestSuiteError: Runner not implemented.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/avocado/core/suite.py", line 262, in runner
    runner_extension = RunnerDispatcher()[runner_name]
                       ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/avocado/core/extension_manager.py", line 181, in __getitem__
    raise KeyError
KeyError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/avocado/core/job.py", line 636, in run_tests
    summary |= suite.run(self)
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/avocado/core/suite.py", line 338, in run
    return self.runner.run_suite(job, self)
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/avocado/core/suite.py", line 265, in runner
    raise TestSuiteError("Runner not implemented.")
avocado.core.suite.TestSuiteError: Runner not implemented.
Please include the traceback info and command line used on your bug report
Report bugs visiting https://github.com/avocado-framework/avocado/issues/new
Exit status: 4

Me and @arif-ali are currently looking into this.

[arif-ali]

The first 3 items in the list have been resolved in the latest PR, just need to work out the avocado issue

on another note, this isn't the first time we've had new python version issues below are list of issues and PRs that have been implemented due to people finding issues and then resolving retrospectively. I think we need a better way to find these issues so that we are more pro-active

#3308 #3319 #3467 #3330 #3301

I will be proposing something after we get this one resolved, as this fix will depends now on our release of the package for 4.7.0

[dnegreira]

Me and @arif-ali investigated this, and @arif-ali has a bunch of code changes on PR #3587 which have the fixes for the regexp changes.
As for the issue with the 'Runner not implemented' we figured out it was because imp is now deprecated on python3.12.

The avocado-framework [pushed a fix for avocado-framework 92] to be compatible with Python3.12 but it is not pushed to Pypi yet, so we would need that in order to be able to run tests on Python3.12, I have raised a issue on the avocado-framework project to kindly ask them to push a new 92 LTS version to Pypi.

[TurboTurtle]

I just mentioned this in #3587 but commenting here for tracking.

There is now a new avocado LTS, version 103.0, which formally adds python 3.12 support. We should look at moving towards that and off of our current version of avocado. I can set some time aside this weekend to start looking into that, but if anyone else would like to start that venture before then please feel free to; no need to wait for me on this :).

[arif-ali]

I did briefly look at it on how we can switch to nrunner and was pulling my hair out on why it was failing. Now that I read your comment in #3587, it just clicked on the parallelism, and there is an option --nrunner-max-parallel-tasks=X, if we set this t 1, it disables that, and hence work in the same way as before.

This could have been the the key ;). I have stageone currently running with nrunner and so far it is passing with 94.0. It should be a simple change imo.

Once both #3587 and #3591 are merged, it would be easier to get this into action, as these would all conflict each other.

I'll see if the same works with the 103.0, and will propose something after the above 2 PRs land

[TurboTurtle]

Yeah, I'm giving it a once-over now on my end. There's some minor fixes we'll need to do for regex matching (looks like the bulk of it can just convert to raw strings), but for the most part this should actually be pretty straight-forward (knock on wood).

I'll keep plugging away at this, but we should be able to move to 103 without too much fuss. The option does change to --max-parallel-tasks, trying to see if there are any other small tweaks to make.

[arif-ali]

So, based on my testing, on top of #3587 we should only need the following diff, and all should go to plan

https://paste.ubuntu.com/p/PDxvFtdM4j/

diff --git a/.cirrus.yml b/.cirrus.yml
index 7bd019bd..891e42e8 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -134,7 +134,7 @@ deb_build_task:
         apt -y install devscripts equivs python3-pip
         mk-build-deps
         apt -y install ./sosreport-build-deps*.deb
-        pip3 install avocado-framework==94.0 --break-system-packages
+        pip3 install "avocado-framework<104.0" --break-system-packages
     main_script: |
         dpkg-buildpackage -b -us -uc -rfakeroot -m --build-by="[email protected]"
     prep_artifacts_script: mv ../*.deb ./sos_cirrus.deb
@@ -226,13 +226,13 @@ report_stageone_task:
         fi
         PIP_EXTRA=""
         [[ $(pip3 install --help | grep break-system) ]] && PIP_EXTRA="--break-system-packages"
-        pip3 install avocado-framework==94.0 ${PIP_EXTRA}
+        pip3 install "avocado-framework<104.0" ${PIP_EXTRA}
     # run the unittests separately as they require a different PYTHONPATH in
     # order for the imports to work properly under avocado
     unittest_script: &unit_test |
         PYTHONPATH=. avocado run tests/unittests/
     main_script: &stageone_test
-        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --test-runner=runner -t stageone tests/{cleaner,collect,report,vendor}_tests
+        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --max-parallel-tasks=1 -t stageone tests/{cleaner,collect,report,vendor}_tests
     on_failure:
         fail_script: &faillogs |
             ls -d /var/tmp/avocado* /root/avocado* 2> /dev/null | xargs tar cf sos-fail-logs.tar
@@ -252,11 +252,6 @@ report_stageone_daily_task:
     environment:
       PKG: "snap"
     setup_script: *setup
-    avocado_install_script: &avocado_setup |
-        pip3 uninstall -y avocado-framework --break-system-packages
-        git clone --single-branch -b 92lts https://github.com/avocado-framework/avocado.git
-        cd avocado
-        make install
     unittest_script: *unit_test
     main_script: *stageone_test
     on_failure:
@@ -285,7 +280,7 @@ report_stagetwo_task:
             dnf -y install python3-pexpect
         fi
     main_script: &stagetwo_test |
-        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --test-runner=runner -t stagetwo tests/{cleaner,collect,report,vendor}_tests
+        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --max-parallel-tasks=1 -t stagetwo tests/{cleaner,collect,report,vendor}_tests
     on_failure:
         fail_script: *faillogs
         log_artifacts: *logs
@@ -303,7 +298,6 @@ report_stagetwo_daily_task:
     environment:
       PKG: "snap"
     setup_script: *setup
-    avocado_install_script: *avocado_setup
     main_script: *stagetwo_test
     on_failure:
         fail_script: *faillogs
diff --git a/tests/collect_tests/help_output_tests.py b/tests/collect_tests/help_output_tests.py
index 584ad498..f53c80a1 100644
--- a/tests/collect_tests/help_output_tests.py
+++ b/tests/collect_tests/help_output_tests.py
@@ -14,7 +14,7 @@ from avocado.utils import software_manager
 from sos_tests import StageOneOutputTest, SOS_REPO_ROOT, skipIf
 
 installer = software_manager
-sm = installer.SoftwareManager()
+sm = installer.manager.SoftwareManager()
 
 PEXPECT_PRESENT = sm.check_installed('python3-pexpect')
 
diff --git a/tests/sos_tests.py b/tests/sos_tests.py
index 1fb31875..25d5f4e9 100644
--- a/tests/sos_tests.py
+++ b/tests/sos_tests.py
@@ -772,7 +772,7 @@ class StageTwoReportTest(BaseSoSReportTest):
         # seems awkward, but check_installed() and remove() are not exposed
         # together with install_distro_packages()
         self.installer = software_manager
-        self.sm = self.installer.SoftwareManager()
+        self.sm = self.installer.manager.SoftwareManager()
 
         for dist in self.packages:
             if isinstance(self.packages[dist], str):
@@ -842,7 +842,7 @@ class StageTwoReportTest(BaseSoSReportTest):
             self._strip_installed_packages()
             if not self.packages[self.local_distro]:
                 return
-            installed = self.installer.install_distro_packages(self.packages)
+            installed = self.installer.distro_packages.install_distro_packages(self.packages)
             if not installed:
                 raise Exception(
                     "Unable to install requested packages %s"

[TurboTurtle]

Of course I say that, then jinx it. ;)

I'm looking at the SoftwareManager changes that allow our StageTwo tests to temporarily install packages and then remove them after the test is done.

For the check_installed and remove methods, this is a change of module structure but to be honest the way the inheritance of backends work, it took me a bit to figure that out >.>;. We should be good to just do this:

# instead of this
self.sm = avocado.utils.software_manager.SoftwareManager() 

# do this now
self.sm = avocado.utils.software_manager.manager.SoftwareManager()

However, the install_distro_packages() method seems to be completely missing, despite the 103 documentation showing it also just changed location in the module...

>>> pkgs = {'rhel': 'sos'}
>>> avocado.utils.software_manager.distro_packages.install_distro_packages(pkgs)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'avocado.utils.software_manager' has no attribute 'distro_packages' 

>>> sm = avocado.utils.software_manager.manager.SoftwareManager()
>>> sm.install_distro_packages(pkgs)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jake/.local/lib/python3.12/site-packages/avocado/utils/software_manager/manager.py", line 47, in __getattr__
    return self.backend.__getattribute__(name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DnfBackend' object has no attribute 'install_distro_packages'

@clebergnu Sorry for the out-of-the-blue tag, but you're the main contact I remember from the avocado team. Is the above just a doc bug in that install_distro_packages() doesn't exist anymore, or am I blind to something obvious?

[arif-ali]

@TurboTurtle I just checked this page, which shows the the location of install_distro_packages now

[TurboTurtle]

Yeah, that's where I linked to as well:

>>> avocado.VERSION
'103.0'
>>> pkgs = {'rhel': ['sos']}
>>> avocado.utils.software_manager.distro_packages.install_distro_packages(pkgs)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'avocado.utils.software_manager' has no attribute 'distro_packages'
>>> 

[arif-ali]

yup, you're right, didn't catch it after my mods. latest run has the following, so the new distro_packages is not working

 (229/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_plugin_ran: STARTED
 (229/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_plugin_ran: ERROR: module 'avocado.utils.software_manager' has no attribute 'distro_packages' (0.10 s)
 (230/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_conf_collected: STARTED
 (230/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_conf_collected: ERROR: [Errno 2] No such file or directory: '/var/tmp/avocado_l16c5461sos_tests.py.Krb5PluginTest/output' (0.03 s)
 (231/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_ubuntu_kdcdir_collected: STARTED
 (231/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_ubuntu_kdcdir_collected: ERROR: [Errno 2] No such file or directory: '/var/tmp/avocado_l16c5461sos_tests.py.Krb5PluginTest/output' (0.04 s)
 (232/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_redhat_kdcdir_collected: STARTED
 (232/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_redhat_kdcdir_collected: ERROR: [Errno 2] No such file or directory: '/var/tmp/avocado_l16c5461sos_tests.py.Krb5PluginTest/output' (0.03 s)
 (233/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_archive_created: STARTED
 (233/301) tests/report_tests/plugin_tests/krb5.py:Krb5PluginTest.test_archive_created: ERROR: [Errno 2] No such file or directory: '/var/tmp/avocado_l16c5461sos_tests.py.Krb5PluginTest/output' (0.04 s)

[TurboTurtle]

I'll open a PR shortly with the raw string changes as well as the, to date, found avocado changes. Once we figure out a workaround there, we can move forward on this rather quickly. At worst, we can loop over SoftwareManager().install() ourselves, rather than relying on a potential install_distro_packages().

[arif-ali]

Found it :)

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import avocado
>>> pkgs = {'ubuntu': ['sos']}
>>> from avocado.utils.software_manager import distro_packages
>>> distro_packages.install_distro_packages(pkgs)
False
>>> avocado.version
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'avocado' has no attribute 'version'
>>> avocado.VERSION
'103.0'
>>> pkgs = {'ubuntu': ['sosreport']}
>>> distro_packages.install_distro_packages(pkgs)
True

[arif-ali]

updated diff below, which should work

diff --git a/.cirrus.yml b/.cirrus.yml
index 7bd019bd..891e42e8 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -134,7 +134,7 @@ deb_build_task:
         apt -y install devscripts equivs python3-pip
         mk-build-deps
         apt -y install ./sosreport-build-deps*.deb
-        pip3 install avocado-framework==94.0 --break-system-packages
+        pip3 install "avocado-framework<104.0" --break-system-packages
     main_script: |
         dpkg-buildpackage -b -us -uc -rfakeroot -m --build-by="[email protected]"
     prep_artifacts_script: mv ../*.deb ./sos_cirrus.deb
@@ -226,13 +226,13 @@ report_stageone_task:
         fi
         PIP_EXTRA=""
         [[ $(pip3 install --help | grep break-system) ]] && PIP_EXTRA="--break-system-packages"
-        pip3 install avocado-framework==94.0 ${PIP_EXTRA}
+        pip3 install "avocado-framework<104.0" ${PIP_EXTRA}
     # run the unittests separately as they require a different PYTHONPATH in
     # order for the imports to work properly under avocado
     unittest_script: &unit_test |
         PYTHONPATH=. avocado run tests/unittests/
     main_script: &stageone_test
-        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --test-runner=runner -t stageone tests/{cleaner,collect,report,vendor}_tests
+        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --max-parallel-tasks=1 -t stageone tests/{cleaner,collect,report,vendor}_tests
     on_failure:
         fail_script: &faillogs |
             ls -d /var/tmp/avocado* /root/avocado* 2> /dev/null | xargs tar cf sos-fail-logs.tar
@@ -252,11 +252,6 @@ report_stageone_daily_task:
     environment:
       PKG: "snap"
     setup_script: *setup
-    avocado_install_script: &avocado_setup |
-        pip3 uninstall -y avocado-framework --break-system-packages
-        git clone --single-branch -b 92lts https://github.com/avocado-framework/avocado.git
-        cd avocado
-        make install
     unittest_script: *unit_test
     main_script: *stageone_test
     on_failure:
@@ -285,7 +280,7 @@ report_stagetwo_task:
             dnf -y install python3-pexpect
         fi
     main_script: &stagetwo_test |
-        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --test-runner=runner -t stagetwo tests/{cleaner,collect,report,vendor}_tests
+        PYTHONPATH=tests/ avocado run -p TESTLOCAL=true --max-parallel-tasks=1 -t stagetwo tests/{cleaner,collect,report,vendor}_tests
     on_failure:
         fail_script: *faillogs
         log_artifacts: *logs
@@ -303,7 +298,6 @@ report_stagetwo_daily_task:
     environment:
       PKG: "snap"
     setup_script: *setup
-    avocado_install_script: *avocado_setup
     main_script: *stagetwo_test
     on_failure:
         fail_script: *faillogs
diff --git a/tests/collect_tests/help_output_tests.py b/tests/collect_tests/help_output_tests.py
index 584ad498..f53c80a1 100644
--- a/tests/collect_tests/help_output_tests.py
+++ b/tests/collect_tests/help_output_tests.py
@@ -14,7 +14,7 @@ from avocado.utils import software_manager
 from sos_tests import StageOneOutputTest, SOS_REPO_ROOT, skipIf
 
 installer = software_manager
-sm = installer.SoftwareManager()
+sm = installer.manager.SoftwareManager()
 
 PEXPECT_PRESENT = sm.check_installed('python3-pexpect')
 
diff --git a/tests/sos_tests.py b/tests/sos_tests.py
index 1fb31875..4cd6822f 100644
--- a/tests/sos_tests.py
+++ b/tests/sos_tests.py
@@ -12,6 +12,7 @@ from avocado.core.output import LOG_UI
 from avocado import Test
 from avocado.utils import archive, process, distro, software_manager
 from avocado.utils.cpu import get_arch
+from avocado.utils.software_manager import distro_packages
 from fnmatch import fnmatch
 
 import glob
@@ -772,7 +773,7 @@ class StageTwoReportTest(BaseSoSReportTest):
         # seems awkward, but check_installed() and remove() are not exposed
         # together with install_distro_packages()
         self.installer = software_manager
-        self.sm = self.installer.SoftwareManager()
+        self.sm = self.installer.manager.SoftwareManager()
 
         for dist in self.packages:
             if isinstance(self.packages[dist], str):
@@ -842,7 +843,7 @@ class StageTwoReportTest(BaseSoSReportTest):
             self._strip_installed_packages()
             if not self.packages[self.local_distro]:
                 return
-            installed = self.installer.install_distro_packages(self.packages)
+            installed = distro_packages.install_distro_packages(self.packages)
             if not installed:
                 raise Exception(
                     "Unable to install requested packages %s"
                    

[TurboTurtle]

Reopening after merging #3587.

@arif-ali didn't notice that your #3591 added the raw string changes. There's now a merge conflict after merging #3587, so let's get that wrapped up and then we can move forward to 103.