[v8r0] Improve getTransformationFiles performance #7812
Merged
+86
−95
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DIRACGridBot
added
the
alsoTargeting:integration
chrisburr
force-pushed
the
perf-getTransformationFiles
branch
2 times, most recently
from
3f1c5f0 to
ddf159b
Compare
chrisburr
changed the title
chrisburr
force-pushed
the
perf-getTransformationFiles
branch
from
ddf159b to
b24a791
Compare
chrisburr
force-pushed
the
perf-getTransformationFiles
branch
2 times, most recently
from
9078b1c to
bc47ed7
Compare
andresailer
reviewed
Oct 3, 2024
fstagni
previously approved these changes
Oct 7, 2024
chrisburr
commented
Oct 7, 2024
chaen
dismissed
fstagni’s stale review
Apparently, Chris found something wrong
chrisburr
force-pushed
the
perf-getTransformationFiles
branch
from
0e47bdb to
836e34c
Compare
|
I've rolled back incorrect commit and added the comment as suggested in #7812 (comment) so this should be good to go |
chaen
approved these changes
Oct 14, 2024
DIRACGridBot
added
the
sweep:done
DIRACGridBot
pushed a commit
to DIRACGridBot/DIRAC
that referenced
this pull request
Oct 14, 2024
|
Sweep summary Sweep ran in https://github.com/DIRACGrid/DIRAC/actions/runs/11324634755 Successful:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This significantly improves the performance of
getTransformationFilesby:JOINinstead of manually looking up the LFNs from file IDs__getFileIDsForLfnsOFFSET N LIMIT 10000made the function O(N^2) due to the database having to rebuild and scan the results for every batch.For example in LHCb: this optimises a ~1200 second call to be ~20 seconds.
BEGINRELEASENOTES
*TransformationSystem
NEW:
CHANGE: Improve getTransformationFiles performance
ENDRELEASENOTES