Pro client apt pinning prevents installing newer package versions from all other repositories (PPA, Ubuntu Cloud Archive, etc)

[lathiat]

Description of the bug

The pro client installs an apt pinning rule to give Pin-Priority 510 to packages in the UbuntuESM/UbuntuESMApps repositories, in the following files: /etc/apt/preferences.d/ubuntu-pro-esm-apps /etc/apt/preferences.d/ubuntu-pro-esm-infra

This causes apt not to select a package from any other repository, even if it has a newer version number. This applies to packages both in the main archive, as well as any third-party repository such as a Launchpad PPA, Ubuntu Cloud Archive, etc.

It seems the intent was likely to ensure we use the ESM version of a package in preference to a different version in the main archive, perhaps to prevent upgrading to a non-fixed version after an SRU to the main archive without the security fix was released for example, but instead users are unable to install newer versions of software from all third party repositories. This was done as part of #2580, #2681.

I discovered this when trying to install a newer version of 'fish' from their Launchpad PPA. The same would apply for users using any kind of third party PPA to get a newer version of a package.

This also has implications for other Ubuntu services such as the Ubuntu Cloud archive which may make newer versions of packages available for OpenStack, OVN, Ceph, etc. If such a package also gets an ESM update, the wrong version may unexpectedly be installed.

Expected behavior

Pro client should not prevent newer versions of packages being installed from other repositories. It perhaps could prefer ESM packages over the regular Ubuntu archive, if it did not also make the same change for other archives.

We can match the main archive with o=Ubuntu, many other archives have an origin that is different (e.g. PPAs having something like o=LP-PPA-fish-shell-release-3), however modifying o=Ubuntu to 499 for example (or setting everything other than o=Ubuntu to 510) also means any even old outdated PPA for some bit of software would superseed the main Ubuntu archive, which is also likely not what we want.

I am not sure it's possible to implement the current intent without some kind of rule that would explicitly say "only prefer UbuntuESM to Ubuntu, and not any other archive", from what I can see that option doesn't currently exist. But I think we need to consult with someone that really understands apt much better than even me.

Current behavior

root@zlab:~# apt-cache policy fish
fish:
  Installed: 3.3.1+ds-3ubuntu0.1~esm1
  Candidate: 3.3.1+ds-3ubuntu0.1~esm1
  Version table:
     3.7.1-1~jammy 500
        500 https://ppa.launchpadcontent.net/fish-shell/release-3/ubuntu jammy/main amd64 Packages
 *** 3.3.1+ds-3ubuntu0.1~esm1 510
        510 https://esm.ubuntu.com/apps/ubuntu jammy-apps-security/main amd64 Packages
        100 /var/lib/dpkg/status
     3.3.1+ds-3 500
        500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

root@zlab:~# apt-cache policy cargo
cargo:
  Installed: 0.67.1+ds0ubuntu0.libgit2-0ubuntu0.22.04.2+esm1
  Candidate: 0.67.1+ds0ubuntu0.libgit2-0ubuntu0.22.04.2+esm1
  Version table:
     1.75.0+dfsg0ubuntu1~bpo0-0ubuntu0.22.04 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
        500 http://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages
 *** 0.67.1+ds0ubuntu0.libgit2-0ubuntu0.22.04.2+esm1 510
        510 https://esm.ubuntu.com/apps/ubuntu jammy-apps-security/main amd64 Packages
        100 /var/lib/dpkg/status
     0.58.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu jammy/universe amd64 Packages

To Reproduce

Please include details on how to reproduce the bug.

lxc launch ubuntu:22.04 jammy
lxc shell jammy
pro attach TOKEN
apt update
apt install fish
apt-add-repository ppa:fish-shell/release-3
apt-cache policy fish

You can also see on a default install that cargo has a much newer package version in jammy-updates than ESM.

System information:

• Ubuntu release: VERSION="22.04.5 LTS (Jammy Jellyfish)"
• Pro Client version: 34~22.04

[renanrodrigo]

Hello @lathiat, thanks for raising this.
What you describe here is not a bug, but a feature (:
There is a product decision to prefer esm packages when those are installed, unless the users actively configure it diferently for any PPA.

As you pointed out, the intention is that any user who has opted in to ESM never automatically gets an update that was not released by our secuity team, for the risks that may be implied by these upgrades.

but instead users are unable to install newer versions of software from all third party repositories

Well that is only true for apt upgrade and variants, as unattended upgrades and update manager.
Users can install unpreferred packages by specifying the version when running apt install:

sudo  apt install fish=3.7.1-1~jammy

Alternatively, if users want any package in a PPA to have preference over ESM, they must manually configure apt to do so when the PPA is added, by creating a preferences file pinning the PPA to a higher version than ESM. We can't configure this in the user's behalf (I mean the Pro Client itself could of course, but that is again a product decision).
You also mention that this would mean that if the PPA gets stale, it will be preferred over the archive or ESM itself - again, the user must decide about this and configure it accordingly.

I see this is not documented anywhere. What our team can do is to document that you need to pin your PPAs - maybe add it to the page where we mention the pins. Do you think documenting it helps? Is that a feasible answer to your request?

[renanrodrigo]

On a separate thread, when it comes to the Ubuntu archive, it should be fine to have the ESM version pinned because it should never run too far from the versions in -updates for instance.
@dodys, do you see how the cargo version is outdated there on ESM in this example here? Is that expected? Is that part of that monitoring of universe updates so ESM does not get too far behind?

[lathiat]

Let's ignore the cargo situation for a moment within the context of the bug, since I just happened to notice that discrepancy, I think that is an unrelated issue that can be resolved, since as you state the ESM and Main archive packages should be closely tracking each other normally. But I'll discuss the rest.

Users of Ubuntu can and do install software from PPAs and other third party apt repositories very frequently, often those PPAs are added explicitly to get a newer version of a very specific piece of software they are interested in, in most cases that same piece of software is also in the Ubuntu archives, with an older version. Instructions to install software like this are everywhere. This is VERY very common, almost universal.

On that basis, I think that preventing installation of newer versions of software from other apt sources is a bug. This can and will break user expectations not only of "third party" PPAs, but even many apt sources or PPAs of official Canonical products such as Charmed OpenStack, COS, etc.

I agree that having a slightly older ESM package preferred over a package in the Ubuntu main archive is a "feature". I understand why this was added, and I am fine with that part of it. The problem is that the fix for that has this unintended and problematic consequences for changing package version priority of repositories other than something in Ubuntu/UbuntuESM.

There are real problems with the workarounds you mentioned:

• Installing a specific version by request with the exact version number is an extra fiddly step that users are not going to do in practice, they are likely just following instructions without that step which work fine for non-ESM Ubuntu users
• Once that version is installed, apt upgrade won't then upgrade them to new versions from that PPA when they are released. Besides not being in line with user expectations, it will also leave them exposed to both bugs and security issues in those packages without constant manual intervention, the exact same thing you're trying to avoid for the packages in the ESM archive.
• The above would be solved by also pinning the ppa/third party repositories to 510, however again, in practice, people are not going to add a pinning step for every PPA, because the instructions everyone publishes don't have such and users without ESM don't need to do that.
• Even if every ppa/third party repository was pinned to 510 the same as ESM, it will then mean that apt will install the version in the PPA, even if that version is older than the one in the main archive. This is also likely not expected, as unlike ESM, those PPAs don't necessarily have the guarantee of closely tracking Ubuntu in the same way you do.
• A few examples of where this might happen in practice, often the Support team will create a PPA with a hotfix or test package of a bit of software that is needed before we SRU it into the main archive. Currently, they are versioned such that it will be replaced by the version the main archive once it's released. With such pinning it place, they'll be stuck on that temporary PPA forever if they don't remove it, which is often the case. Another case I have seen is that some third party PPAs support multiple versions of Ubuntu and include dependency upgrades that won't be used on some versions because the Ubuntu version is newer, but now with pinning, they would still be used incorrectly. Sometimes people also haev a local scratch repository of various packages they have added in, which again without pinning, may have used a newer version from the main archive if it existed, etc.

Hence I think for many different reasons and scenarios, what we need here is a solution that will have older versions in the UbuntuESM origin be preferred over packages in the Ubuntu origin ONLY, without in any way impacting normal version selection for any other origin/repository/ppa combination. Anything else is going to break user expectations and product expectations in unexpected ways. But I am not sure that is possible with apt currently.

Some input from someone much more knowledgable about apt would be helpful, perhaps @julian-klode?

[renanrodrigo]

Thanks for the longer explanations about the problems!

On the APT side, there is nothing we could do. This was discussed with Julian in different contexts - including this one of pinning the ESM stuff - seems there is a planned feature for APT to do this kind of distinction (using Origins and how packages from one Origin can upgrade to some different one), but that's future IIRC. @julian-klode can confirm of course.

What you describe here is not a bug, but a feature (:

What I mean by that is that while the Client is responsible to set this configuration, this wasn't our decision - this was a request from the people who maintain ESM on the security side.

I do understand your concerns and I think they are valid, and I don't think we want users to have bad experiences like the ones described.
I wonder that if the pinning is doing more harm than good, we could reconsider it as a whole.
@dodys let's try to involve the people who can make a decision on the path forward in this discussion?

For instance, @lechsandecki, what would be your take on this?

[dodys]

On a separate thread, when it comes to the Ubuntu archive, it should be fine to have the ESM version pinned because it should never run too far from the versions in -updates for instance. @dodys, do you see how the cargo version is outdated there on ESM in this example here? Is that expected? Is that part of that monitoring of universe updates so ESM does not get too far behind?

the cargo listed here is different from the cargo in esm.

• cargo in esm is the source package name, which generates a binary named cargo
• in the archive, the source package name is rustc , which generates a binary named cargo.
• cargo source package is in universe, rustc is in main.

I don't think there's much we can do here, it needs to be a manual intervention if you want the latest cargo binary package

[dodys]

Thanks for the longer explanations about the problems!

On the APT side, there is nothing we could do. This was discussed with Julian in different contexts - including this one of pinning the ESM stuff - seems there is a planned feature for APT to do this kind of distinction (using Origins and how packages from one Origin can upgrade to some different one), but that's future IIRC. @julian-klode can confirm of course.

What you describe here is not a bug, but a feature (:

What I mean by that is that while the Client is responsible to set this configuration, this wasn't our decision - this was a request from the people who maintain ESM on the security side.

Just to correct here, it wasn't a decision by security. Pinning was a request from cloud customers because of the fact that ESM Apps is subject to SRU and so on, and therefore people are subject to "losing" the fix that went into ESM. It came directly from CPC and was agreed with Product Management. The only thing security did was improve tracking to reduce the time when archive and ESM archive are too different. It was not our call to make on having the pinning or not. As mentioned on the first reply, it is a product decision.