Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LotMan configuration to the cache #889

Merged
merged 16 commits into from
Apr 2, 2024

Conversation

jhiemstrawisc
Copy link
Member

@jhiemstrawisc jhiemstrawisc commented Mar 5, 2024

This PR adds a variety of ways to interface with Lotman in the cache.

Note that as of my opening this PR, the CRUD actions exposed via gin don't have tests. While the token verification is very nuanced and undoubtedly contains mistakes, I propose we wait until the next release cycle to work on these tests because there's a cyclic import issue I need to figure out (launchers imports lotman to register everything, and lotman tests would need to import launchers to spin up the entire federation). For now, the API endpoints can be turned off, and the lots are configurable via the pelican config.

Closes #817

@jhiemstrawisc
Copy link
Member Author

@cannon, let me know if you need anything else to define the Lots list in your object UI.

Copy link
Collaborator

@turetske turetske left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First major issue. Because you branched off of add-cache-launcher at the beginning before it was finished, there are multiple duplicated commits at the beginning of this pull request. This will also cause conflicts with the finished version that is currently in the code. I suggest creating a new branch off of main and applying your specific commits (the last 9 as opposed to all 12) to the branch and recreating the PR otherwise merging this in will be a mess.

@haoming29
Copy link
Contributor

First major issue. Because you branched off of add-cache-launcher at the beginning before it was finished, there are multiple duplicated commits at the beginning of this pull request. This will also cause conflicts with the finished version that is currently in the code. I suggest creating a new branch off of main and applying your specific commits (the last 9 as opposed to all 12) to the branch and recreating the PR otherwise merging this in will be a mess.

Since the cache launcher has been merged, it would be easier to rebase against main to forward commits to your new changes, but you might need to fix the merge conflicts in rebasing, if that's too much hassle, then Emma's idea should help

@turetske
Copy link
Collaborator

turetske commented Mar 5, 2024

First major issue. Because you branched off of add-cache-launcher at the beginning before it was finished, there are multiple duplicated commits at the beginning of this pull request. This will also cause conflicts with the finished version that is currently in the code. I suggest creating a new branch off of main and applying your specific commits (the last 9 as opposed to all 12) to the branch and recreating the PR otherwise merging this in will be a mess.

Since the cache launcher has been merged, it would be easier to rebase against main to forward commits to your new changes, but you might need to fix the merge conflicts in rebasing, if that's too much hassle, then Emma's idea should help

If he had just branched off of my branch, I'd agree with you, but since I think he applied specific commits from my PR rather than branching, I don't this it's possible. This is one of the downsides of working with forked repos rather than on the main repo, it's not easy to branch off of other peoples branches.

I just rebased and those 3 commits are still there. Without any of the potential later fixes I added. This will also add duplicate commits that say the same thing. So I think the answer is to drop those first three commits and then rebase.

@turetske turetske force-pushed the issue-817 branch 2 times, most recently from 5e1dab7 to ed1ef01 Compare March 5, 2024 23:13
launchers/cache_serve_linux.go Outdated Show resolved Hide resolved
@turetske turetske added this to the v7.6.0 milestone Mar 6, 2024
Copy link
Contributor

@CannonLock CannonLock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I will tack on a UI in a separate PR.

Copy link
Collaborator

@turetske turetske left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few clarification comments requests.

lotman/lotman.go Show resolved Hide resolved
lotman/lotman.go Outdated Show resolved Hide resolved
lotman/lotman.go Outdated Show resolved Hide resolved
lotman/lotman.go Outdated Show resolved Hide resolved
lotman/lotman.go Outdated Show resolved Hide resolved
lotman/lotman.go Outdated Show resolved Hide resolved
docs/parameters.yaml Outdated Show resolved Hide resolved
Copy link
Collaborator

@turetske turetske left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review focused on the token logic

lotman/lotman.go Outdated Show resolved Hide resolved
lotman/lotman.go Outdated Show resolved Hide resolved
lotman/lotman_ui.go Outdated Show resolved Hide resolved
lotman/lotman_ui.go Outdated Show resolved Hide resolved
lotman/lotman_ui.go Outdated Show resolved Hide resolved
lotman/lotman_ui.go Outdated Show resolved Hide resolved
lotman/lotman_ui.go Outdated Show resolved Hide resolved
lotman/lotman_ui.go Show resolved Hide resolved
lotman/lotman_ui.go Outdated Show resolved Hide resolved
lotman/lotman_ui.go Show resolved Hide resolved
@turetske turetske modified the milestones: v7.6.0, v7.7.0 Mar 11, 2024
Copy link
Collaborator

@turetske turetske left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will approve once the parameter validation is fixed.

@jhiemstrawisc
Copy link
Member Author

@CannonLock Is the parameter addition is something you'll need to handle, or can you give me a quick overview of how I can do it?

@CannonLock
Copy link
Contributor

@jhiemstrawisc Will push a commit on top of this in ~1 hours to add UI.

- Add LotsField
- Add DateTimeField
- Add PathField
- Add Lots to verified object structures
- Pre-commit fixup
@CannonLock
Copy link
Contributor

@jhiemstrawisc You will have to merge with main for the parameters check to pass.

Copy link
Collaborator

@turetske turetske left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving since Cannon's change is in

@jhiemstrawisc
Copy link
Member Author

Okay, merging then! Thanks everyone 😄

@jhiemstrawisc jhiemstrawisc merged commit 8b4229b into PelicanPlatform:main Apr 2, 2024
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create basic set of CRUD actions for LotMan in the cache
4 participants