Pax process sthml #23
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -72,7 +72,7 @@ | |||
| "username": "bobau", | |||
| "download_options": ["midway-login1"], | |||
| "pax_processing_versions": [ | |||
| "v4.10.0" | |||
| "v4.10.2" | |||
There was a problem hiding this comment.
won't this trigger a reprocessing?
|
Anyways, added a few comments. Some big things (check main.py). One key thing: the data type is processed and status is transferred. That means good. If you manually change verifying to transferred, then you have no idea if the data is any good... that's why we have checksumming. |
…rmissions every 60min
…rmissions every 60min
…rmissions every 60min
| def pax_deploy_directories(host, pax_version): | ||
| PAX_DEPLOY_DIRS = { | ||
| 'midway-login1' : '/project/lgrandi/deployHQ/pax', | ||
| 'tegner-login-1': '/afs/pdc.kth.se/projects/xenon/software/pax/pax_'+pax_version |
There was a problem hiding this comment.
Why do you need to specify pax version here? You should have one anaconda setup that has different environments for different pax versions. That's why this wasn't a function.
There was a problem hiding this comment.
I think my aim was to specify to the actual pax version in the path. But this is not important. Changed it back
|
Looks good, added some comments. Not far from merging. Need to resolve some merge conflcits. |
…om SetPermissions
|
Can you merge with master? |
|
Also, @XeBoris when you get a chance, can you please update the README with all the latest features/options? |
tunnell
added a commit
that referenced
this pull request
Jul 1, 2016
tunnell
added a commit
that referenced
this pull request
Jul 1, 2016
tunnell
added a commit
that referenced
this pull request
Jul 12, 2016
* Parallel support * Processing now doesn't create new task * Update classifiers * Redo qsub script * Redo qsub script * Redo qsub script * Redo qsub script * Massive cax * Skip if exists * Redo qsub script * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * test_env * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Yo * Only email on failure (was clogging email servers lol) -PatrickDP * Remove email * Update email * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Update tegner username * Set permission (See Pull #23) * Set permission (See Pull #23) * Catch timeout * Update config.py * Update config.py * Update config.py * Update config.py * Update qsub.py * Update config.py * Update qsub.py * Allow massive cax to submit jobs for new runs * Allow massive cax to submit jobs for new runs * Better waiting * Have massive cax search just for actionable runs, except once a day. Fix pax version issue until figure out how to deal with multiple pax versions. * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * ncpus = 1 * Use central pax env * Four cores * Fix typo in env * Make run over everything at start * Proper version * Don't delete bad checksums if no master checksum * Unused line * - Implement lcg-cp and recursiveness - Add timer in Copy class * Tweak submission options
pdeperio
pushed a commit
that referenced
this pull request
Oct 11, 2016
* Parallel support * Processing now doesn't create new task * Update classifiers * Redo qsub script * Redo qsub script * Redo qsub script * Redo qsub script * Massive cax * Skip if exists * Redo qsub script * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * test_env * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Yo * Only email on failure (was clogging email servers lol) -PatrickDP * Remove email * Update email * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/config.py * Update tegner username * Set permission (See Pull #23) * Set permission (See Pull #23) * Catch timeout * Update config.py * Update config.py * Update config.py * Update config.py * Update qsub.py * Update config.py * Update qsub.py * Allow massive cax to submit jobs for new runs * Allow massive cax to submit jobs for new runs * Better waiting * Have massive cax search just for actionable runs, except once a day. Fix pax version issue until figure out how to deal with multiple pax versions. * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * Merge branch 'parallel' of github.com:XENON1T/cax into parallel # Conflicts: # cax/main.py * ncpus = 1 * Use central pax env * Four cores * Fix typo in env * Make run over everything at start * Proper version * Don't delete bad checksums if no master checksum * Unused line * - Implement lcg-cp and recursiveness - Add timer in Copy class * Tweak submission options
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I added some features to cax-status which helps to identify the status of files/folder of our data in the data base. I started to use this tool as replacement for actual using the web interface to the run data base
I tested cax-process manually. Furthermore I added parser options to make life more easier in case someone wants to use cax-process. Batch processing should work too.