-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Processing on OSG #67
Open
ershockley
wants to merge
113
commits into
master
Choose a base branch
from
OSG_dev2
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
Show all changes
113 commits
Select commit
Hold shift + click to select a range
9da727a
Incorporate OSG Connect as a processing site, first steps:
b08ae8f
Merge with master v3.0.5
ershockley 59e0457
prepare process.py for installation on OASIS
ershockley c546e55
Prepare for testing api/osg_dev on ci-connect machine. Python not wor…
ershockley 529a72e
Revert back to old (semi-working) version of OSG_dev and add api.py
ershockley 8aa8fca
Revert back to old (semi-working) version of OSG_dev and add api.py
ershockley b3d4600
Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2
ershockley 732517a
This version launches processing but throws a pickling error
fbe8a9c
Working version on midway
e49b71f
Update for login.ci-connect.uchicago.edu
ershockley 17ff365
first attempt at resolving conflicts from merge with master
ershockley 85984da
Still debugging. Pushing to be able to test on midway (ci running slo…
ershockley b3479b9
API calls work on OSG. Need to implement workaround for pax pymongo call
ershockley d14a5d0
modify process.py so that config dictionary for processing called usi…
ershockley c7942c8
Processing works on OSG, though bookmarking still needs polishing. La…
ershockley 536de88
Adding first pass of OSG scripts
briedel 2bea618
Improved version
briedel 755d882
debugging, minor changes
briedel 5babaea
Adding Friends of MWT2 special sauce
briedel f7f86d9
Modify cax to submit condor dagman jobs rather than regular jobs. Min…
ershockley 47dfda8
Adding preliminary OSG guide
briedel 71ee993
Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2
briedel e261147
Adding more OSG docs
briedel 0fe90a6
Small changes
briedel c59c515
Adding first draft of rucio docs
briedel 518f8cd
Breaking out admin guide
briedel 4724bde
Moving more things around
briedel 9eb2b03
Add post script to dag and first implementation of database bookkeeping
ershockley cbaa0da
Adding tutorial
briedel 2ea9df2
Update run_xenon.sh
tunnell 62017c1
Adding more detail to tutorial
briedel d413397
Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2
briedel 759a6ae
Minor formatting
briedel 90b4d63
Make several changes to optimize dag submission, post scripts, pre sc…
ershockley 45ae1bf
Merge with Benedikts changes in osg_scripts
ershockley b285824
Make several changes including dag submission, post scripts, pre scri…
ershockley 89eee80
Add flexible API queries. Some other small changes
ershockley fb7db43
More small changes
ershockley 798ba77
Merge branches 'OSG_dev2' and 'master' of https://github.com/XENON1T/…
pdeperio 16c6971
Missed additional merge from HEAD
pdeperio d247b3c
Exception to not purge Stash processing errors
pdeperio 56be9f4
Fix conditional bug
pdeperio 77d8081
Remove .idea stuff
pdeperio 6d64324
Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2
pdeperio bad0664
Make thishost and pax_version common member variables
pdeperio e921024
Fix qsub for massive-cax
pdeperio ea50970
Last commit before merging with master-merged OSG_dev2
ershockley 1aa0af8
Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2
ershockley ab8c572
Small fix in cax.json
ershockley 9cd2325
Fix bug allowing for duplicate DB entries on midway. Also change oute…
ershockley 9a8ba4b
Add some optimizations to OSG processing. Make cax-like processing wo…
ershockley beb30ad
Update run_xenon.sh for rucio downloads. Small change in process.py t…
ershockley 524917c
add get_gains.py. Originally written to only try processing runs that…
ershockley 48bc695
Add one-line fix for to corrections.py to add acquisition monitor cha…
ershockley 8045b1a
Merge with master to implement rsync and ruciax
ershockley 2f80f25
Update dag_writer.py
ershockley 13c0886
Add midway gfal functionality to dag_writer.py
ershockley b3da3df
Optimizations on DAG side of things. Make sure rsync implemented
ershockley ac8000c
have dag_writer.py skips runs if raw data not in chicago (stash, midw…
ershockley ef82706
Add drift velocity check to dag_prescript.py. Few other small changes.
ershockley fb3f681
Implement Automatic OSG processing
ershockley 29cb5ee
Implement MV processing on OSG. Other small changes including queries…
ershockley 7673e68
Optimizations in run_xenon.sh for rucio transfers, few other small ch…
ershockley 755ff45
Fix things that broke for some reason in DB handling
ershockley 50f87a7
Fix DB handling which broke somehow
ershockley 8f84294
Make daily processing work again. Not sure what happened...
ershockley 99724dc
Make daily processing work again. Not sure what happened...
ershockley 8206766
Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2
ershockley d955f5a
Process NG data only
ershockley b28b706
Starting to make changes to api, but not finished
ershockley b697d8b
Add --api option so that we can circumvent pymongo calls
ershockley 1b34fd9
Add --api option so that we can circumvent pymongo calls
ershockley 7d057d5
remove hardcoded 'transferred' in data_mover.py. Other small changes
ershockley 8b3ccb9
Remove NG specification in process.py
ershockley 68154d5
fix conflicts in api.py and data_mover.py
ershockley 2db8120
Some cleaning up. Add .gitignore and stage_data.py
ershockley f1a7cc0
Implement MV changes. Some other small changes.
ershockley f8707bd
Implement MV changes. Some other small changes.
ershockley 2c3de99
Fix small bug in process.py
ershockley e7f218d
Small changes. Commit before dag_writer.py overhaul
ershockley bc08822
Add determine_rse.py so that jobs download from European rucio endpoi…
ershockley e5a670b
Remove ershockley dependence
ershockley 2dbd1d0
More changes - hopefully to remove any ershockley dependence
ershockley d041356
Alfio's info
ershockley 6b18a4a
Set GLIDEIN_Country to US if not set
ershockley bdde49a
Add dag_config
ershockley a8785f2
Merge branch 'any_user' of https://github.com/XENON1T/cax into any_user
ershockley c1744d1
More fixes to allow any user to run code
ershockley 91b1b8b
dunno
ershockley c166b4b
Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2
ershockley 0030597
multiple retries for copy back to stash
ershockley 948d14c
Change DB query to only process newest runs with v6.6.6
ershockley ea972a3
Add curl commands for job monitoring
ershockley 90046dd
Fix bug in run_xenon.sh
ershockley f3878cc
Updates to make_runlist.py, some other small improvements
ershockley d5fab9d
Remove annoying print statement
ershockley 4dfd0c3
Merge
ershockley 68b6a60
Clean up finished
ershockley 8cddb56
Commit before merging with benedikt cleanup
ershockley d092e2b
Merge with benedikt_cleanup
ershockley 44d673e
Revert back to old commit
ershockley 221317b
dumb commit
ershockley 0e697eb
Merge branch 'revert' into OSG_dev2
ershockley 889e932
Improvements with permissions setting, etc.
ershockley 9caed6d
Hopefully fix for any user to use again
ershockley 305e4ec
Add some hardcoded pieces but hopefully will result in less errors
ershockley b6000e3
Change settings for SR1 reprocessing
ershockley de24493
Update hadd_and_upload.sh
ershockley 58c948f
Modify caxdir lines to use home directory
ershockley 7ee5800
Try to remove gfal-debug stuff?
ershockley 4e781c8
Shouldn't be any major changes - SR1 daily processing running normal
ershockley 8d2a1e7
Small changes to dag_writer, adding SURFSARA
ershockley 55fabbf
Remove gfal debug, add CNAF to some of the rucio scripts, other small…
ershockley File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Optimizations on DAG side of things. Make sure rsync implemented
- Loading branch information
commit b3da3dff9f9b88c63bda0a9f354a366d07f4d747
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,27 +1,29 @@ | ||
[ | ||
{ | ||
"name": "login", | ||
"method": "rsync", | ||
"hostname": "login.ci-connect.uchicago.edu", | ||
"dir_raw": "/xenon/xenon1t_test", | ||
"dir_processed": "/xenon/ershockley/processed", | ||
"upload_options": ["midway-login1"], | ||
"username": "pdeperio", | ||
"task_list": ["CopyPush"], | ||
"data_types" : ["processed"] | ||
}, | ||
|
||
{ | ||
"name": "midway-login1", | ||
"method": "rsync", | ||
"hostname": "midway-login1.rcc.uchicago.edu", | ||
"dir_raw": "/project/lgrandi/xenon1t/raw", | ||
"dir_raw": "/project2/lgrandi/xenon1t/raw", | ||
"dir_processed": "/project2/lgrandi/xenon1t/processed", | ||
"dir_minitrees": "/project/lgrandi/xenon1t/minitrees", | ||
"dir_minitrees": "/project2/lgrandi/xenon1t/minitrees", | ||
"upload_options": [], | ||
"username": "tunnell", | ||
"download_options": ["xe1t-datamanager"], | ||
"purge" : 25, | ||
"grid_cert": "/project/lgrandi/deployHQ/cax/user_cert" | ||
} | ||
] | ||
}, | ||
{ | ||
"name": "login", | ||
"method": "rsync", | ||
"hostname": "login", | ||
"dir_raw": "/xenon/xenon1t/raw", | ||
"dir_processed": "/xenon/xenon1t_processed", | ||
"upload_options": ["midway-login1"], | ||
"username": "pdeperio", | ||
"download_options": [], | ||
"task_list": ["CopyPush"], | ||
"grid_cert": "/stash2/project/@xenon1t/deployHQ/cax/user_cert", | ||
"data_types" : ["processed"] | ||
} | ||
|
||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -26,17 +26,25 @@ cd /home/ershockley/cax/ | |
export PYTHONPATH=/home/ershockley/cax/lib/python3.4/site-packages/:$PYTHONPATH | ||
python setup.py install --prefix ${PWD} | ||
|
||
if [[ ! -e /xenon/xenon1t_processed/pax_$2/$1.root ]]; then | ||
python /home/ershockley/cax/osg_scripts/upload.py $1 $5 $2 >> $post_log 2>&1 | ||
# save exit status of pipe | ||
set -o pipefail | ||
|
||
if [[ $? -ne 0 ]]; then | ||
exit 1 | ||
fi | ||
fi | ||
#if [[ ! -e /xenon/xenon1t_processed/pax_$2/$1.root ]]; then | ||
python /home/ershockley/cax/osg_scripts/upload.py $1 $5 $2 >> $post_log 2>&1 | ||
|
||
if [[ $? -ne 0 ]]; then | ||
exit 1 | ||
fi | ||
#fi | ||
# transfer to midway | ||
echo "Beginning cax transfer to midway" >> $post_log | ||
echo "Paused for now because gfal issues" | ||
#/home/ershockley/cax/bin/cax --once --run $3 --config /home/ershockley/cax/cax/cax_transfer.json >> $post_log 2>&1 | ||
/home/ershockley/cax/bin/cax --once --run $3 --config /home/ershockley/cax/cax/cax_transfer.json >> $post_log 2>&1 | ||
|
||
# submit massive-cax job to verify transfer | ||
|
||
echo "Submitting massive cax job on midway for run $3" >> $post_log | ||
|
||
ssh [email protected] "/home/tunnell/verify_stash_transfers.sh $3" >> $post_log 2>&1 | ||
|
||
ex=$? | ||
|
||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
merge with main cax.json
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't. Need separate config for a separate transfer-only process that happens in post-script of DAG.
(cax_stash replication is also running a separate .json, but just not under version control.)