Support scheduled package adding and subsequent repo publishing

[acheng-01]

Is your feature request related to a problem? Please describe.
Some of our publishers publish embargoed packages that require them to run pipelines a few hours prior to release. However, this method introduces a few areas of unreliability, including pipeline agent failure, network issues, authentication problems surrounding config changes, etc. that can end up delaying their release.

Describe the solution you'd like
It would be ideal if Pulp is able to support scheduled package adding and repo publishing following that. Publishers would be able to build their packages, upload their packages, and then schedule a time for Pulp to perform the add and publish actions. If anything goes wrong during the upload/scheduling process, publishers can resolve them earlier without delaying their release schedule. But once that is complete, Pulp will guarantee that their packages are distributed publicly on time without additional issues.

Describe alternatives you've considered
Our alternative here is to use a separate container that publishers can use to schedule future releases. The container would have a db and regularly check if it needs to add/publish packages in Pulp. If there is a scheduled task, the container calls Pulp to add the packages to the repo and publish them.

Additional context
Could be related to #6104, but wanted to hear some thoughts on this before proceeding.

[ggainey]

Pulp's repo-version/publication/distribution architecture is designed specifically for the use-case of "don't make my repository visible to the outside world until I have gotten it into the shape I want it". For the "preparation" portion of your use-case, consider the following scenario:

• There is a repository R, whose latest-repository-version, RV1, has a publication P1, and is pointed to by distribution D1.
• You add a lot of packages to R. Each add creates new RVs - RV2-RV8, say.
• When done adding packages, you publish RV8, creating a publication P2 that points to RV8
• At the embargo date - you either point D1 at RV8, or create a new distribution D2 that points at RV8, making the new packages visible.

Note that none of RV2-7 are ever visible to the outside world. RV8 doesn't become visible, until a distribution points to it.

We are striving to not add a general-purpose scheduler into Pulp. There are a (very) few pulp-maintenance-tasks, like orphan-cleanup, that pulp should be responsible for - but your workflow should be scheduled by any of the existing, external, scheduling/workflow tools.

[Note: edited for correctness - a reminder to not answer issues before coffee in the morning...]

[mdellweg]

See here how RepositoryVersion, Publication and Distribution interact:
https://pulpproject.org/pulpcore/docs/user/learn/concepts/#serving-content-with-pulp

[daviddavis]

We have multiple teams that add packages to repos so I'm not sure how we could leverage repo versions. If one team creates a repo version like R8 and then another team uploads packages and creates R9, either the embargoed packages would go out early or if the team somehow creates R9 without the packages, then when R8 goes live, the packages from R9 would be dropped.

I know that Pulp has avoided adding scheduled tasks but I thought maybe if you all were adding periodic tasks for orphan cleanup, there could possibly be some generalized solution we could use for our use case. I totally understand if that's not the case though and would be interested if you have any suggestions for tools that we could use to implement this externally.

[mdellweg]

Just to recap: "Orphan cleanup" is a different story to begin with. It is a (to be) periodicaly executed task. It takes no arguments and no one is really interested in its result. If it was skipped or failed once, it will continue to do its work the next time. It is janitors work and after all the user should not even be told about the concept of orphans.
Adding packages on the other hand is a specific task that should run exactly once with a specific package on a specific repository. And you ultimately want to be notified about it's success / failure. Think about some sort of cron job, calling pulp-cli to copy a set of packages from your "incoming" repository to your "published security updates" repository. And on completion it sends you an email.