Add option to publish using "No-Support-for-Architecture-all: Packages" repo format

[quba42]

Currently pulp_deb always publishes only separate Architecture = all package indices. This is considered the best practice format, but it is not a format used by any official Debian repos. Instead those use the No-Support-for-Architecture-all: Packages format for more backwards compatibility (for example for Debian bullseye).

I consider it dangerous to exclusively use a format not used by anyone in the wild.

The proposal would be to add a option to publish in the "No-Support-for-Architecture-all: Packages" style.

[akulakhan]

I also need this to be added. Not having a way to combine "all" and "amd64" architecture packages breaks specific bootstrap installer that I rely on, specifically the diskimage-builder ubuntu-minimal stuff here: https://docs.openstack.org/diskimage-builder/latest/elements/ubuntu-minimal/README.html

[quba42]

@akulakhan Can you explain what exact version of Ubuntu software and what exact commands do not work as expected?
This is important to us, since it would be the first confirmed example of something that does not work.

In the meantime, you could probably still work around the problem by using the "verbatim publisher", which publishes the repo that you synced exactly as it was (including all metadata and signatures).

[akulakhan]

@akulakhan Can you explain what exact version of Ubuntu software and what exact commands do not work as expected? This is important to us, since it would be the first confirmed example of something that does not work.

In the meantime, you could probably still work around the problem by using the "verbatim publisher", which publishes the repo that you synced exactly as it was (including all metadata and signatures).

Provided a log for the failure. The issue occurs during the 08-debootstrap step which seems to be solely reliant on the DIB_DISTRIBUTION_MIRROR provided, which can only be a single repo, and doesn't seem to use the normal apt-get workflow (which does work).

2022-04-12 15:07:27.432 | ++ default_ubuntu_mirror=http://archive.ubuntu.com/ubuntu
2022-04-12 15:07:27.432 | ++ export DIB_DISTRIBUTION_MIRROR=https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/
2022-04-12 15:07:27.432 | ++ DIB_DISTRIBUTION_MIRROR=https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/
2022-04-12 15:07:27.432 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/10-ubuntu-minimal.bash
2022-04-12 15:07:27.434 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/10-ubuntu-minimal.bash
2022-04-12 15:07:27.434 | ++ export 'DIB_DEBIAN_COMPONENTS_WS=main universe'
2022-04-12 15:07:27.434 | ++ DIB_DEBIAN_COMPONENTS_WS='main universe'
2022-04-12 15:07:27.434 | ++ DIB_APT_SOURCES_CONF_DEFAULT='default:deb https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic main universe'
2022-04-12 15:07:27.434 | ++ DIB_UBUNTU_MIRROR_DISTS=
2022-04-12 15:07:27.434 | ++ export 'DIB_APT_SOURCES_CONF=default:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-security main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-updates/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-security/2021-05-18-000000/ bionic-security main restricted universe'
2022-04-12 15:07:27.434 | ++ DIB_APT_SOURCES_CONF='default:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000/ bionic-security main restricted universe
2022-04-12 15:07:27.434 | updates:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-updates/2021-05-18-000000/ bionic-updates main restricted universe
2022-04-12 15:07:27.434 | security:deb [trusted=yes arch=amd64] https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic-security/2021-05-18-000000/ bionic-security main restricted universe'
2022-04-12 15:07:27.434 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-init-system.bash
2022-04-12 15:07:27.436 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-init-system.bash
2022-04-12 15:07:27.436 | ++ [[ bionic == \t\r\u\s\t\y ]]
2022-04-12 15:07:27.436 | ++ export DIB_INIT_SYSTEM=systemd
2022-04-12 15:07:27.436 | ++ DIB_INIT_SYSTEM=systemd
2022-04-12 15:07:27.436 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-kernel.bash
2022-04-12 15:07:27.438 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/11-ubuntu-kernel.bash
2022-04-12 15:07:27.438 | ++ export DIB_UBUNTU_KERNEL=linux-image-generic
2022-04-12 15:07:27.438 | ++ DIB_UBUNTU_KERNEL=linux-image-generic
2022-04-12 15:07:27.438 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/12-ubuntu-repo-dists.bash
2022-04-12 15:07:27.440 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/12-ubuntu-repo-dists.bash
2022-04-12 15:07:27.440 | ++ export DIB_UBUNTU_MIRROR_DISTS=updates,security,backports
2022-04-12 15:07:27.440 | ++ DIB_UBUNTU_MIRROR_DISTS=updates,security,backports
2022-04-12 15:07:27.440 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/14-manifests
2022-04-12 15:07:27.442 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/14-manifests
2022-04-12 15:07:27.442 | ++ export DIB_MANIFEST_IMAGE_DIR=/etc/dib-manifests
2022-04-12 15:07:27.442 | ++ DIB_MANIFEST_IMAGE_DIR=/etc/dib-manifests
2022-04-12 15:07:27.442 | ++ export DIB_MANIFEST_SAVE_DIR=/outputdir/tiny.d/
2022-04-12 15:07:27.442 | ++ DIB_MANIFEST_SAVE_DIR=/outputdir/tiny.d/
2022-04-12 15:07:27.442 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/20-network-interface-names
2022-04-12 15:07:27.443 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/20-network-interface-names
2022-04-12 15:07:27.443 | ++ export 'DIB_NETWORK_INTERFACE_NAMES=eth0 eth1'
2022-04-12 15:07:27.443 | ++ DIB_NETWORK_INTERFACE_NAMES='eth0 eth1'
2022-04-12 15:07:27.443 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/50-devuser
2022-04-12 15:07:27.445 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/50-devuser
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_USERNAME=dev
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_USERNAME=dev
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_SHELL=
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_SHELL=
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_PWDLESS_SUDO=yes
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_PWDLESS_SUDO=yes
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_AUTHORIZED_KEYS=
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_AUTHORIZED_KEYS=
2022-04-12 15:07:27.445 | ++ export DIB_DEV_USER_PASSWORD=<a password>
2022-04-12 15:07:27.445 | ++ DIB_DEV_USER_PASSWORD=<a password>
2022-04-12 15:07:27.445 | dib-run-parts Sourcing environment file /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/99-dib-init-system
2022-04-12 15:07:27.447 | + source /tmp/dib_build.AF3pu4pz/hooks/root.d/../environment.d/99-dib-init-system
2022-04-12 15:07:27.447 | ++ '[' -z systemd ']'
2022-04-12 15:07:27.447 | ++ case $DIB_INIT_SYSTEM in
2022-04-12 15:07:27.447 | dib-run-parts Running /tmp/dib_build.AF3pu4pz/hooks/root.d/08-debootstrap
2022-04-12 15:07:27.478 | I: Retrieving InRelease 
2022-04-12 15:07:27.503 | I: Checking Release signature
2022-04-12 15:07:27.506 | I: Valid Release signature (key id 4E0FDBC2E1296EAC6C7EC7E93DA3CBF27F36FB4F)
2022-04-12 15:07:27.536 | I: Retrieving Packages 
2022-04-12 15:07:27.562 | I: Validating Packages 
2022-04-12 15:07:27.612 | I: Retrieving Packages 
2022-04-12 15:07:27.674 | I: Validating Packages 
2022-04-12 15:07:28.650 | I: Resolving dependencies of required packages...
2022-04-12 15:07:34.565 | I: Resolving dependencies of base packages...
2022-04-12 15:07:39.072 | I: Found additional required dependencies: e2fslibs fdisk libaudit1 libblkid1 libbz2-1.0 libc6 libcap-ng0 libcom-err2 libdb5.3 libdbus-1-3 libdebconfclient0 libexpat1 libext2fs2 libfdisk1 libgcrypt20 libgpg-error0 liblz4-1 libmount1 libncursesw5 libpcre3 libsemanage1 libsmartcols1 libss2 libstdc++6 libsystemd0 libtinfo5 libudev1 libuuid1 libzstd1 
2022-04-12 15:07:39.072 | I: Found additional base dependencies: gpgv libapt-pkg5.0 libffi6 libgmp10 libgnutls30 libhogweed4 libidn2-0 libnettle6 libp11-kit0 libseccomp2 libtasn1-6 libunistring2 
2022-04-12 15:07:39.079 | I: Checking component main on https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000...
2022-04-12 15:07:39.178 | I: Checking component universe on https://<pulp 3 server hostname>:8443/pulp/deb/upstream/ubuntu-bionic/2021-05-18-000000...
2022-04-12 15:07:39.796 | E: Couldn't find these debs: ca-certificates gnupg2
2022-04-12 15:07:40.025 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
The build step did not produce a kernel and initramfs. Check builds/dib-output.log
Makefile:104: recipe for target 'builds/tiny.initramfs' failed
make: *** [builds/tiny.initramfs] Error 1

Note that I have tried using the verbatim publisher, however this doesn't work for me since it seems to have various issues. Not the least of which is that it doesn't support translation files. In the workflows I must support, I do not have control over every client to disable fetching of these files, and therefore the builds fail.

In the meantime I am simply pointing these builds to use the pulp 2 instance which I will maintain until pulp 3 supports this.

Also, I have monitored the nginx logs on the pulp3 server to confirm that this process is only grabbing the amd64 package index, which does not contain the missing packages since they are "all" architecture. (gnupg2, for example)

[akulakhan]

@quba42 do you have any idea when this might be fixed or if it's on the roadmap at all? I am presently having to maintain both a pulp 2 and pulp 3 server as a result of this issue, and it would be nice to have an end of life estimation for the former.

[akulakhan]

Anyone have any update on whether this is on the roadmap?

[quba42]

Having discussed this issue internally, I am afraid it has not currently made it onto our roadmap, since none of our use cases are currently hitting this as a problem and we don't have the time to work on it. If any external contributors do want to have a go at it, I can give some pointers (I don't think this would be that hard to implement).

[akulakhan]

I can look into implementing it, any pointers would be helpful.

[quba42]

@akulakhan I guess the place for specifying the option, perhaps named no_support_for_architecture_all, would be after this line: https://github.com/pulp/pulp_deb/blob/main/pulp_deb/app/models/publication.py#L34
Most of the code for actually implementing the feature would have to go in https://github.com/pulp/pulp_deb/blob/main/pulp_deb/app/tasks/publishing.py In particular, I believe something like the following would need to be added here:

if no_support_for_architecture_all and package.architecture=='all':
    for architecture in architectures:
        package_serializer.to822(self.component).dump(
            self.package_index_files[architecture][0]
        )
        self.package_index_files[architecture][0].write(b"\n")

But that is just a rough draft off the top of my head and some of those variables are currently not available in that function. The basic Idea is: "if we have an architecture all package it needs to be added to every package index, and not just the all index".

You do have one other option that does not involve coding a new pulp_deb feature: You could try using the verbatim publisher instead of the APT publisher, to just mirror the upstream repo you are trying to use exactly.

[akulakhan]

Thanks for the info. I have tried the verbatim publisher but it does not work in this case since the systems consuming it will fail if the language packages aren't available. These are static/released images that must be supported without changes, so there is no option to make changes on the client side to work around this, unfortunately.

I will take a crack at implementing this feature at some point soon.

[quba42]

since the systems consuming it will fail if the language packages aren't available

I am afraid I have more bad news here: pulp_deb currently ignores all language files during the sync (if it synced them, then the verbatim publisher would publish them). It looks like pulp_deb won't be usable for your use case without some significant further development. We do have the long term goal of making pulp_deb sync absolutely every type of file that can exist in an upstream repo, and then mirroring those files exactly as they were via the verbatim publisher. However, due to resource constraints, this is more a long term goal than an actual roadmap at this point.

[quba42]

Actually I may have been overly pessimistic with the language files. It is plausible that with verbatim publisher the upstream Release files will advertise language files, that are actually missing, prompting the clients to complain. When using the APT publisher, the language files will also be missing, but they also won't be advertised as present in the Release file, so the clients might be fine with that. In that case all you would need is the "No-Support-for-Architecture-all: Packages" mode described by this issue. The only way to be 100% certain though would be to implement this issue and then test.