Fixes #36902 - Support Pulpcore 3.39, drop older versions

[ianballou]

Assumes that pulp/pulpcore#4679 is in Pulpcore 3.39.

Also assumes that pulp/pulpcore#4681 is going to be fixed for Pulpcore 3.39 in time. This doesn't affect the installer but we shouldn't release Katello with that Pulpcore bug.

I had to assume that the entry point will live in /usr/bin since, when installed via pip, it is installed to /usr/local/bin.

I verified these work (minus the jitter for now):

ExecStart=/usr/local/bin/pulpcore-api --preload --timeout 90 --workers 5 --max-requests 800 --access-logfile - --access-logformat 'pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'

and

ExecStart=/usr/local/bin/pulpcore-content  --preload --timeout 90  --workers 5 --access-logfile -

[ianballou]

I haven't touched the tests yet.

[ianballou]

I suppose this is going to break all of the older Pulp tests?

[ekohl]

I agree this will break older releases, which is fine. Last time we did that was 5db8f48 but it changed a bit since then. To update the supported versions:

puppet-pulpcore/.sync.yml

Line 3 in 101c325

beaker_facter: 'pulpcore_version:Pulp:3.28,3.22,3.21'

puppet-pulpcore/.github/workflows/ci.yml

Line 18 in 101c325

beaker_facter: 'pulpcore_version:Pulp:3.28,3.22,3.21'

puppet-pulpcore/manifests/repo.pp

Line 6 in 101c325

Pattern['^\d+\.\d+$'] $version = '3.28',

https://github.com/theforeman/puppet-pulpcore/blob/master/README.md#support-policy

Of course it will fail until we have repositories. We can also add nightly since that now exists.

[ianballou]

@ekohl I updated all of those files including the README.

[ianballou]

[test]

[ianballou]

Actually I don't think we can test just yet, the packages are available but as nightly https://yum.theforeman.org/pulpcore/nightly/el8/

[evgeni]

I've just merged #318, if you rebase and add "nightly" to the matrix, at least that we should see working

[ianballou]

@evgeni I've added nightly

[evgeni]

  missing groups or modules: pulpcore:el8

this seems odd… cc @ehelms @Odilhao

[evgeni]

and on el9 it tries to fetch gpg keys, fails (obviously) and bails out

  Error: /Stage[main]/Pulpcore::Install/Package[pulpcore]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/dnf -d 0 -e 1 -y install pulpcore' returned 1: Status code: 404 for https://yum.theforeman.org/pulpcore/nightly/GPG-RPM-KEY-pulpcore (IP: 151.101.42.49)

[ianballou]

I just found out that Python 11 doesn't work with module streams, so there will be no modules. This might cause some bigger issues, at the very least our docs will need updating...

[evgeni]

Oooh, you're saying Pulpcore is also demodularized now?!

[ianballou]

Oooh, you're saying Pulpcore is also demodularized now?!

Seems like it, I just found out a couple minutes ago. Sounds like it's not by choice, but by how Python 3.11 works. Sounds like we could still ship the modular metadata if removing it is too troublesome.

[evgeni]

Well, just because Python is not shipped in a module anymore doesn't mean we can't ship (no Android, not shit, I mean sure, in a way, but no) Pulpcore as a module. That being said, there is no reason for us to now. I think this is fine and we can just drop the module stuff from repo.pp.

[ekohl]

I didn't review the last version, but I opened the question whether pulpcore needs modularity given Python 3.11 is just a regular package. That would needs updates, which makes the whole Pulp upgrade from a platform perspective an even larger risk than it already was, but cleaner

[ianballou]

Would it only be the docs and this puppet-pulpcore repo that will need updates to remove the modular expectation?

[evgeni]

That would be my naive assumption.

Maybe foreman operations collection, don't recall how we set up Pulpcore there.

[ianballou]

I've pushed a change to drop the modules, let's see if I did it right.

[ianballou]

Nightly GPG key is an issue again: Error: /Stage[main]/Pulpcore::Install/Package[pulpcore]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/dnf -d 0 -e 1 -y install pulpcore' returned 1: Status code: 404 for https://yum.theforeman.org/pulpcore/nightly/GPG-RPM-KEY-pulpcore (IP: 146.75.94.49)

[ianballou]

I'm attempting to remove the GPG key requirement for nightly...

[ianballou]

Hmm still erroring Error: /Stage[main]/Pulpcore::Install/Package[pulpcore]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/dnf -d 0 -e 1 -y install pulpcore' returned 1: Status code: 404 for https://yum.theforeman.org/pulpcore/nightly/GPG-RPM-KEY-pulpcore

[ianballou]

Test-wise I think this is good to go -- the only failing bits are Pulp-CLI on EL9 with the 3.39 repo. Nightly is working on EL9, and everything EL8 works.

[evgeni]

So I tried an upgrade by installing 3.28 with the module currently on Forge and then upgrading to 3.39 with this PR here.

Generally, things seem to work (well, a pulp status is happy, this was my sole test :D) after applying pulpcore::repo from this PR, running dnf upgrade and applying pulpcore and pulcore::cli afterwards.

[evgeni]

Removing this here, leaves a "failsafe" module in DNF, but seems not hinder the upgrade:

[root@centos8-stream ~]# dnf module list |grep -B5 pulpcore
Last metadata expiration check: 0:04:07 ago on Thu 23 Nov 2023 07:26:27 AM UTC.
@modulefailsafe
Name                 Stream          Profiles                                 Summary                                                                                                                                                                                                                                   
pulpcore             el8 [e]                                                  Pulpcore module                                                                                                                                          

The only cleanup needed is to remove the /etc/dnf/modules.d/pulpcore.module file, which I think can be done in here?

(People who do not use pulpcore::repo still need to be informed via docs to either delete that file or call dnf module reset pulpcore, but I think that's fine)

[ekohl]

(People who do not use pulpcore::repo still need to be informed via docs to either delete that file or call dnf module reset pulpcore, but I think that's fine)

theforeman/foreman-documentation#2597

[ekohl]

This is now green, so I'm going to merge this now.

[ekohl]

I realize now that I only merged this part, which will end up in nightly but I can't merge Katello/katello#10791 because I don't have permissions.

[ekohl]

Everything related is now merged

[Odilhao]

The Katello release failed for nightly, and it's related to the installer, probably it was not released yet on the nightly package:

[2023-11-24T20:42:14.474Z]     Error 1: Puppet Service resource 'pulpcore-api.service' failed. Logs:

[2023-11-24T20:42:14.474Z]       /Service[pulpcore-api.service]

[2023-11-24T20:42:14.474Z]         Starting to evaluate the resource (1386 of 1453)

[2023-11-24T20:42:14.474Z]         Skipping restart; service is not running

[2023-11-24T20:42:14.474Z]         Triggered 'refresh' from 3 events

[2023-11-24T20:42:14.474Z]         The container Systemd::Unit_file[pulpcore-api.service] will propagate my refresh event

[2023-11-24T20:42:14.474Z]         Evaluated in 1.14 seconds

[2023-11-24T20:42:14.474Z]       /Stage[main]/Pulpcore::Service/Pulpcore::Socket_service[pulpcore-api]/Systemd::Unit_file[pulpcore-api.service]/Service[pulpcore-api.service]/ensure

[2023-11-24T20:42:14.474Z]         change from 'stopped' to 'running' failed: Systemd start for pulpcore-api.service failed!

[2023-11-24T20:42:14.474Z]     journalctl log for pulpcore-api.service:

[2023-11-24T20:42:14.474Z]     -- Logs begin at Fri 2023-11-24 20:25:45 UTC, end at Fri 2023-11-24 20:42:05 UTC. --

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:04 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: Starting Pulp API Server...

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com pulpcore-api[44109]: Error: This app must be executed using pulpcore-api entrypoint.

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: pulpcore-api.service: Main process exited, code=exited, status=1/FAILURE

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: pulpcore-api.service: Failed with result 'exit-code'.

[2023-11-24T20:42:14.474Z]     Nov 24 20:42:05 pipe-katello-server-nightly-centos8-stream.example.com systemd[1]: Failed to start Pulp API Server.

[ekohl]

I investigated this and wrote it down in theforeman/jenkins-jobs#382 (comment). My initial theory was wrong, and it appears things aren't synced from staging to release.

[evgeni]

But the pipes should be using staging anyway. So it shouldn't matter if staging>prod sync is broken?

And https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/2099/ was ree, which explains not synced prod?

[evgeni]

https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/2099/consoleFull ran at 9:50PM and pushed the new installer but https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/1837/ ran at 8:18PM and thus just didn't have the new installer (katello doesn't generate foreman staging, but the installer is in foreman).

but when you looked, staging was updated already and confused you :)

[evgeni]

and now installs passed fine.

tests are failing tho, see https://community.theforeman.org/t/katello-nightly-rpm-pipeline-1838-failed/35929/2