adding new component to published repo

[TimoSairiala]

I have a local repo which I have published already. But I have to add deb packages with another component name to this same published repo. Is there any way to publish it again with new component name included without dropping the already published repo?

Detailed Description

What I do is I have one published repo:

aptly repo create -distribution=my_test -component=main test-adding-component
aptly repo add test-adding-component humble/main/amd64
aptly publish repo -architectures="amd64" test-adding-component

=> so far so good
now I have to add more packages with another component name

aptly repo create -distribution=my_test -component=added-component test-adding-component-custom
aptly repo add test-adding-component-custom humble/test_add/amd64/
aptly publish repo -architectures="amd64" -component=, test-adding-component test-adding-component-custom

This last publishing gives an error because this is already published:
ERROR: prefix/distribution already used by another published repo: ./my_test [amd64] publishes {main: [test-adding-component]}

Context

I am running aptly on nas with big amount of deb packages and what ever I do it takes ages. If I run aptly publish drop it also means that repo is not accessible for a while. It is possible I am just doing something wrong or misunderstood instructions, maybe I should be running it another way?

Possible Implementation

maybe some sort of republish or force parameter would do the trick or possibly there already exists something?

[alamaral]

I had exactly the same problem, until a minute ago. To drop the publish you need
to find the distribution and prefix from he aptly publish list command:

root@foobar:/var/aptly# aptly publish list
Published repositories:

• stuff/jammy [amd64] publishes {main: [stuff-jammy-main]: Production 22.04 build stuff software}, {staging: [stuff-jammy-staging]: Staging 22.04 build}, {test: [stuff-jammy-test]: Test 22.04 build stuff software}

The distribution here is jammy and the prefix is stuff, so to drop the publish the command is:

aptly publish drop jammy stuff.

While this works just find for unpublishing the repo so you can add a new component to it,
you LOSE EVERYTHING THAT YOU HAD PUT INTO THE EXISTING REPO...

Yup. You're screwed big time...

I believe that this is a MAJOR flaw in aptly. You should be able to either add a new component
to a published repo, or somehow unpublish the repo without destroying it in the process.

It has been suggested to me to use snapshots, but my repo changes constantly because we use
gitlab CI/CD and every single checkin to gitlab builds a package that ends up in aptly. Plus I
reluctantly inherited what I've got, and don't want to change how this works.

[neolynx]

Hi @TimoSairiala, @alamaral,

and thanks for reporting...

I would also recommend to use snapshots, I think the problems you are describing led to the introduction of snapshots. Apropos 'reluctantly inherited', I feel you ... :) but there is only one way forward...

it should not bee too hard to introduce snapshots (even without breaking the current approach and running both in parallel).

Here is what I would try:

• before the aptly publish repo , take a snapshot of the repo:

  $ aptly snapshot create snapshot1 from repo test-adding-component
  
  Snapshot snapshot1 successfully created. 
  You can run 'aptly publish snapshot snapshot1' to publish snapshot as Debian repository.
  

• after the aptly repo publish, publish the snapshot additionally:

   aptly publish snapshot  -acquire-by-hash -architectures="amd64" -keyring=trustedkeys.gpg -gpg-key=gpg@email snapshot1 publish1
  

- do more releases, and more snapshots
- now, instead of deleting and recreating the repo publish, just switch the snapshot-publish to the newest snpshot:

aptly publish switch -keyring=trustedkeys.gpg -gpg-key=gpg@email my_test publish1 snapshot2

- then delete unused snapshots afterwards, there is also `snapshot rename` in case you wanna do a temporary snapshot, remove the main snapshot, and rename the temporary snapshot to the main name
- this reduces 'downtime', the published files fill not disapear (also thanks to `-acquire-by-hash`)
- there is also `aptly publish update`, in case other parameters of the publish should be modified
- if all of this works, and you see the files you want in the new publish, remove the repo publish code and rename the publish

so publishing a repo directly offers way less functionality, and when you delete such a publish, the files will be gone until you republish. ideally you would do this only once, when switching to snapshots

hope this helps !

[carlosabalde]

Hi @neolynx!

I've been experiencing the same problem (see #1397). After reading this case, I thought snapshots were the solution, but no luck.

Creating the initial local repo, adding packages, then creating the snapshot and finally publishing for the first time works as expected:

$ aptly repo create -distribution=jammy -component=foo ubuntu:project:jammy:foo
$ aptly repo add ubuntu:project:jammy:foo package1.deb
$ aptly snapshot create ubuntu:project:jammy:foo:snapshot from repo ubuntu:project:jammy:foo
$ aptly publish snapshot \
    -component= \
    -gpg-key=... \
    ubuntu:project:jammy:foo:snapshot \
    filesystem:default:ubuntu/project

Adding extra packages (same component and distribution) is not a problem either: the local repo is updated, a new snapshot is created and then published using publish switch.

However, when trying to add a second component (same distribution), publish switch fails because component is not in published repository:

$ aptly repo create -distribution=jammy -component=bar ubuntu:project:jammy:bar
$ aptly repo add ubuntu:project:jammy:bar package2.deb
$ aptly snapshot create ubuntu:project:jammy:bar:snapshot from repo ubuntu:project:jammy:bar
$ aptly publish switch \
    -component=, \
    -gpg-key=... \
    jammy \
    filesystem:default:ubuntu/project \
    ubuntu:project:jammy:foo:snapshot \
    ubuntu:project:jammy:bar:snapshot
ERROR: unable to switch: component  is not in published repository

Perhaps I'm missing something in your explanation, but I'm not sure how to proceed. Any help would be appreciated.

[cfiehe]

Hi @carlosabalde,
actually we should declare the command aptly publish switch as deprecated, because it only allows to switch the snapshot source of an existing component of a published repository. The command cannot be used to add a new component to a published repository, that did not exist beforehand. Removing a component is also not possible.

You have to use the latest code from the master branch that contains #1366. We have introduced a new subcommand group

aptly publish source add|drop|list|remove|replace|update

that gives you full control over the component to snapshot/repo mapping after the published repository has been created. Do not forget to call aptly publish update <distribution> [[<endpoint>:]<prefix>] at the very end to trigger the actual update otherwise the changes stay in the staging area.