Enable auto-generation of Lotman policies #1831
Conversation
This gives caches the ability to inform Lotman of known namespaces, which Lotman can use to automatically create a (currently toothless) purge policy. A lot of the work here involved figuring out how to specify named lot policies through our yaml config. Examples of what I came up with can be found in docs/parameters.yaml. Outside of that, some changes involved figuring out how to merge local config with potentially-discovered config. Until Pelican actually uses the new purge code with the Lotman purge plugin, none of this really does much other than set up a database. For now, this lets us set up most of the lot config in a cache through Pelican, and then the last bits for testing purge stuff need to be configured manually. Until then, the actual purge ordering portion of a policy isn't plumbed into anything -- eventually it will modify XRootD config directly. Finally, I still haven't worked out completely how to manage expiration/deletion of lots. I'll save that for another PR, potentially in the next release cycle.
jhiemstrawisc
added
enhancement
|
For testing this PR, I'd recommend configuring a cache with some lotman policy and a few defined lots, of which at least one should should be an existing namespace. The cache should start up and create the lotman sqlite database, and the tester should be able to verify the values are correctly merged, that a lot is created for each top-level namespace (e.g. for |
|
@CannonLock, this one also requires a ping for you -- there's a new |
|
Also note that tests will continue to fail until the Harbor outage is fixed. |
| ```yaml | ||
| Lotman: | ||
| EnabledPolicy: "my-policy" | ||
| PolicyDefinitions: |
There was a problem hiding this comment.
Looks like this line is causing validate-parameters to fail. Seems like it might cause web-ui issues. If it doesn't currently break anything, can you create an issue for Cannon to fix so that we don't have it consistently failing?
There was a problem hiding this comment.
Cannon is tagged as a reviewer and I pinged cannon for the new PolicyDefinitions object explicitly in a comment. Is that enough?
There was a problem hiding this comment.
@jhiemstrawisc Yup, I will post the fix within the hour.
|
@jhiemstrawisc I've looked through your changes and they seem correct, but I need to do another run and look at the sql database |
|
Cannon is going to open a followup PR with the new parameter validation code, so I'm merging even though this currently fails that action. |
This gives caches the ability to inform Lotman of known namespaces, which Lotman can use to automatically create a (currently toothless) purge policy.
A lot of the work here involved figuring out how to specify named lot policies through our yaml config. Examples of what I came up with can be found in docs/parameters.yaml. Outside of that, some changes involved figuring out how to merge local config with potentially-discovered config.
Until Pelican actually uses the new purge code with the Lotman purge plugin, none of this really does much other than set up a database. For now, this lets us set up most of the lot config in a cache through Pelican, and then the last bits for testing purge stuff need to be configured manually. Until then, the actual purge ordering portion of a policy isn't plumbed into anything -- eventually it will modify XRootD config directly.
Finally, I still haven't worked out completely how to manage expiration/deletion of lots. I'll save that for another PR, potentially in the next release cycle.