[enhancement]: cloud-init does not provide a schema-compliant way to retain current system_info package_mirrors behaviour

[OddBloke]

TL;DR: If a user specifies only a primary mirror via apt:, the current behaviour with (schema-violating) system_info: in vendor-data cannot be replicated using (schema-compliant) apt: in vendor-data. This means we cannot fix schema validation errors appearing in user systems (due to our non-compliant vendor-data) while maintaining our users' expected behaviour.

---

Currently, we configure our cloud's apt repositories via cloud-init vendor-data using system_info:

#cloud-config
system_info:
   ...
   package_mirrors:
     - arches: [i386, amd64]
       failsafe:
         primary: http://archive.ubuntu.com/ubuntu
         security: http://security.ubuntu.com/ubuntu
       search:
         primary:
           - http://mirrors.digitalocean.com/ubuntu/
         security: []

By default, mirrors.digitalocean.com is configured as the primary mirror, and security.ubuntu.com as the security mirror. However, system_info in vendor-data now produces schema validation warnings in cloud-init logs, which means our users are asking us why it is "broken". In investigating this, I found that the following vendor-data produces the same default behaviour:

#cloud-config
apt:
  primary:
    - arches: [i386, amd64, default]
      uri: http://mirrors.digitalocean.com/ubuntu/
  security:
    - arches: [i386, amd64, default]
      uri: http://security.ubuntu.com/ubuntu

and is schema-compliant.

Unfortunately, the behaviour of these two vendor-data configurations diverge if a user provides their own apt: configuration via user-data. If they specify this configuration:

#cloud-config
apt:
  primary:
    - arches: [default]
      uri: http://us.archive.ubuntu.com/ubuntu

then with the system_info: vendor-data, both primary and security mirrors are set to us.archive.ubuntu.com because https://github.com/canonical/cloud-init/blob/main/cloudinit/config/cc_apt_configure.py#L1053-L1056 sees only primary configuration, and so uses it as the security mirror also. With the apt: vendor-data, however, there is a security: configuration in the merged configuration, and so the primary mirror is set to us.archive.ubuntu.com but the security mirror is set to security.ubuntu.com.

I do think this is generally preferable behaviour, as users have to explicitly opt-out of receiving security updates via the most timely mechanism in their user-data. However, it presents an issue for maintaining the behaviour of already-written user-data: users with such user-data may be relying on the fact that no Ubuntu mirrors are enabled with their current configuration (if, e.g., all software on their infra needs to go through a change management process, so they're using their own managed mirrors), and we don't want to change that "just" to address schema validation warnings.

[TheRealFalcon]

We'll leave this to downstream. The 24.2 label is for Ubuntu downstream patches only

[holmanb]

We'll leave this to downstream. The 24.2 label is for Ubuntu downstream patches only

+1 and actually this can be resolved with a downstream patch based on #5411

[blackboxsw]

Downstream SRU is queued for 24.2 release into Focal, Jammy and Noble.
Since the deprecation of system_info was defined with the deprecation_version:24.2 it will only be an INFO level log on downstreams with a features.DEPRECATION_INFO_BOUNDARY set below "24.2".

For this 24.2 SRU in Ubuntu, each of those downstreams now include the following features.DEPRECATION_INFO_BOUNDARY:

• focal: 20.1
• jammy: 22.1
• noble: 24.1

[holmanb]

This has been resolved in upstream 24.2 release, so I'm closing this issue.