Code

Processing on OSG#67

Code

Open

ershockley wants to merge 113 commits intomasterXENON1T/cax:masterfrom OSG_dev2XENON1T/cax:OSG_dev2Copy head branch name to clipboard

+3,886-433

Commits

Commits on Jun 24, 2016

• Incorporate OSG Connect as a processing site, first steps:

  Show description for 9da727a
  

  Evan Shockley

  committed
  9da727a

  Copy full SHA for 9da727a

Commits on Jun 29, 2016

• Merge with master v3.0.5

  ershockley

  committed
  b08ae8f

  Copy full SHA for b08ae8f

Commits on Jul 5, 2016

• prepare process.py for installation on OASIS

  ershockley

  committed
  59e0457

  Copy full SHA for 59e0457

Commits on Aug 30, 2016

• Prepare for testing api/osg_dev on ci-connect machine. Python not working on ci, so will test on midway

  ershockley

  committed
  c546e55

  Copy full SHA for c546e55

• Revert back to old (semi-working) version of OSG_dev and add api.py

  ershockley

  committed
  529a72e

  Copy full SHA for 529a72e

• Revert back to old (semi-working) version of OSG_dev and add api.py

  ershockley

  committed
  8aa8fca

  Copy full SHA for 8aa8fca

• Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2

  ershockley

  committed
  b3d4600

  Copy full SHA for b3d4600

Commits on Aug 31, 2016

• This version launches processing but throws a pickling error

  

  Evan Ray Shockley

  committed
  732517a

  Copy full SHA for 732517a

Commits on Sep 1, 2016

• Working version on midway

  

  Evan Ray Shockley

  committed
  fbe8a9c

  Copy full SHA for fbe8a9c

• Update for login.ci-connect.uchicago.edu

  ershockley

  committed
  e49b71f

  Copy full SHA for e49b71f

Commits on Sep 7, 2016

• first attempt at resolving conflicts from merge with master

  ershockley

  committed
  17ff365

  Copy full SHA for 17ff365

Commits on Sep 9, 2016

• Still debugging. Pushing to be able to test on midway (ci running slowly)

  ershockley

  committed
  85984da

  Copy full SHA for 85984da

Commits on Sep 15, 2016

• API calls work on OSG. Need to implement workaround for pax pymongo call

  ershockley

  committed
  b3479b9

  Copy full SHA for b3479b9

Commits on Sep 20, 2016

• modify process.py so that config dictionary for processing called using API and then passed into core.Processor()

  ershockley

  committed
  d14a5d0

  Copy full SHA for d14a5d0

Commits on Nov 9, 2016

• Processing works on OSG, though bookmarking still needs polishing. Last commit before implementing single zip processing

  ershockley

  committed
  c7942c8

  Copy full SHA for c7942c8

• Adding first pass of OSG scripts

  briedel

  committed
  536de88

  Copy full SHA for 536de88

Commits on Nov 10, 2016

• Improved version

  briedel

  committed
  2bea618

  Copy full SHA for 2bea618

Commits on Nov 13, 2016

• debugging, minor changes

  briedel

  committed
  755d882

  Copy full SHA for 755d882

• Adding Friends of MWT2 special sauce

  briedel

  committed
  5babaea

  Copy full SHA for 5babaea

Commits on Nov 14, 2016

• Modify cax to submit condor dagman jobs rather than regular jobs. Minor changes in some files in osg_scripts

  ershockley

  committed
  f7f86d9

  Copy full SHA for f7f86d9

Commits on Nov 15, 2016

• Adding preliminary OSG guide

  briedel

  committed
  47dfda8

  Copy full SHA for 47dfda8

• Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2

  briedel

  committed
  71ee993

  Copy full SHA for 71ee993

• Adding more OSG docs

  briedel

  committed
  e261147

  Copy full SHA for e261147

Commits on Nov 16, 2016

• Small changes

  briedel

  committed
  0fe90a6

  Copy full SHA for 0fe90a6

Commits on Nov 22, 2016

• Adding first draft of rucio docs

  briedel

  committed
  c59c515

  Copy full SHA for c59c515

Commits on Nov 23, 2016

• Breaking out admin guide

  briedel

  committed
  518f8cd

  Copy full SHA for 518f8cd

• Moving more things around

  briedel

  committed
  4724bde

  Copy full SHA for 4724bde

Commits on Nov 24, 2016

• Add post script to dag and first implementation of database bookkeeping

  ershockley

  committed
  9eb2b03

  Copy full SHA for 9eb2b03

Commits on Nov 30, 2016

• Adding tutorial

  briedel

  committed
  cbaa0da

  Copy full SHA for cbaa0da

• Update run_xenon.sh

  tunnell

  authored
  2ea9df2

  Copy full SHA for 2ea9df2

Commits on Dec 1, 2016

• Adding more detail to tutorial

  briedel

  committed
  62017c1

  Copy full SHA for 62017c1

• Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2

  briedel

  committed
  d413397

  Copy full SHA for d413397

• Minor formatting

  briedel

  committed
  759a6ae

  Copy full SHA for 759a6ae

Commits on Dec 5, 2016

• Make several changes to optimize dag submission, post scripts, pre scripts, etc. Includes dag_writer.py class that handles writing of dags and subdags.

  ershockley

  committed
  90b4d63

  Copy full SHA for 90b4d63

• Merge with Benedikts changes in osg_scripts

  ershockley

  committed
  45ae1bf

  Copy full SHA for 45ae1bf

• Make several changes including dag submission, post scripts, pre scripts, subdags, and more. Adds dag_writer.py class which handles all the file writing needed for this workflow

  ershockley

  committed
  b285824

  Copy full SHA for b285824

Commits on Dec 6, 2016

• Add flexible API queries. Some other small changes

  ershockley

  committed
  89eee80

  Copy full SHA for 89eee80

• More small changes

  ershockley

  committed
  fb7db43

  Copy full SHA for fb7db43

Commits on Dec 7, 2016

• Merge branches 'OSG_dev2' and 'master' of https://github.com/XENON1T/cax into OSG_dev2

  Show description for 798ba77
  pdeperio

  committed
  798ba77

  Copy full SHA for 798ba77

• Missed additional merge from HEAD

  pdeperio

  committed
  16c6971

  Copy full SHA for 16c6971

• Exception to not purge Stash processing errors

  Show description for d247b3c
  pdeperio

  committed
  d247b3c

  Copy full SHA for d247b3c

• Fix conditional bug

  pdeperio

  authored
  56be9f4

  Copy full SHA for 56be9f4

• Remove .idea stuff

  pdeperio

  committed
  77d8081

  Copy full SHA for 77d8081

• Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2

  pdeperio

  committed
  6d64324

  Copy full SHA for 6d64324

• Make thishost and pax_version common member variables

  pdeperio

  committed
  bad0664

  Copy full SHA for bad0664

• Fix qsub for massive-cax

  Show description for e921024
  pdeperio

  committed
  e921024

  Copy full SHA for e921024

Commits on Dec 12, 2016

• Last commit before merging with master-merged OSG_dev2

  ershockley

  committed
  ea50970

  Copy full SHA for ea50970

• Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2

  ershockley

  committed
  1aa0af8

  Copy full SHA for 1aa0af8

• Small fix in cax.json

  ershockley

  committed
  ab8c572

  Copy full SHA for ab8c572

Commits on Dec 16, 2016

• Fix bug allowing for duplicate DB entries on midway. Also change outer dag structure from using subdags to splices to lessen load on submit node

  ershockley

  committed
  9cd2325

  Copy full SHA for 9cd2325

Commits on Jan 3, 2017

• Add some optimizations to OSG processing. Make cax-like processing work again.

  ershockley

  committed
  9a8ba4b

  Copy full SHA for 9a8ba4b

Commits on Feb 18, 2017

• Update run_xenon.sh for rucio downloads. Small change in process.py to prevent problems with different pax versions being processed at same time. TEMPORARY CHANGE IN dag_writer.py

  ershockley

  committed
  beb30ad

  Copy full SHA for beb30ad

• add get_gains.py. Originally written to only try processing runs that have gains calculated, it also now queries for _sciencerun0_candidate tag

  ershockley

  committed
  524917c

  Copy full SHA for 524917c

• Add one-line fix for to corrections.py to add acquisition monitor channel gains

  ershockley

  committed
  48bc695

  Copy full SHA for 48bc695

Commits on Feb 19, 2017

• Merge with master to implement rsync and ruciax

  ershockley

  committed
  8045b1a

  Copy full SHA for 8045b1a

Commits on Feb 27, 2017

• Update dag_writer.py

  ershockley

  authored
  2f80f25

  Copy full SHA for 2f80f25

Commits on Feb 28, 2017

• Add midway gfal functionality to dag_writer.py

  ershockley

  committed
  13c0886

  Copy full SHA for 13c0886

Commits on Mar 9, 2017

• Optimizations on DAG side of things. Make sure rsync implemented

  ershockley

  committed
  b3da3df

  Copy full SHA for b3da3df

Commits on Mar 14, 2017

• have dag_writer.py skips runs if raw data not in chicago (stash, midway, or rucio UC_OSG_USERDISK)

  ershockley

  committed
  ac8000c

  Copy full SHA for ac8000c

Commits on Mar 20, 2017

• Add drift velocity check to dag_prescript.py. Few other small changes.

  ershockley

  committed
  ef82706

  Copy full SHA for ef82706

Commits on Mar 28, 2017

• Implement Automatic OSG processing

  ershockley

  committed
  fb3f681

  Copy full SHA for fb3f681

Commits on May 10, 2017

• Implement MV processing on OSG. Other small changes including queries for new correction format

  ershockley

  committed
  29cb5ee

  Copy full SHA for 29cb5ee

Commits on May 17, 2017

• Optimizations in run_xenon.sh for rucio transfers, few other small changes

  ershockley

  committed
  7673e68

  Copy full SHA for 7673e68

Commits on May 23, 2017

• Fix things that broke for some reason in DB handling

  ershockley

  committed
  755ff45

  Copy full SHA for 755ff45

• Fix DB handling which broke somehow

  ershockley

  committed
  50f87a7

  Copy full SHA for 50f87a7

• Make daily processing work again. Not sure what happened...

  ershockley

  committed
  8f84294

  Copy full SHA for 8f84294

• Make daily processing work again. Not sure what happened...

  ershockley

  committed
  99724dc

  Copy full SHA for 99724dc

• Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2

  ershockley

  committed
  8206766

  Copy full SHA for 8206766

Commits on May 26, 2017

• Process NG data only

  ershockley

  committed
  d955f5a

  Copy full SHA for d955f5a

Commits on May 27, 2017

• Starting to make changes to api, but not finished

  ershockley

  committed
  b28b706

  Copy full SHA for b28b706

• Add --api option so that we can circumvent pymongo calls

  ershockley

  committed
  b697d8b

  Copy full SHA for b697d8b

• Add --api option so that we can circumvent pymongo calls

  ershockley

  committed
  1b34fd9

  Copy full SHA for 1b34fd9

Commits on Jun 1, 2017

• remove hardcoded 'transferred' in data_mover.py. Other small changes

  ershockley

  committed
  7d057d5

  Copy full SHA for 7d057d5

• Remove NG specification in process.py

  ershockley

  committed
  8b3ccb9

  Copy full SHA for 8b3ccb9

• fix conflicts in api.py and data_mover.py

  ershockley

  committed
  68154d5

  Copy full SHA for 68154d5

• Some cleaning up. Add .gitignore and stage_data.py

  ershockley

  committed
  2db8120

  Copy full SHA for 2db8120

Commits on Jun 6, 2017

• Implement MV changes. Some other small changes.

  ershockley

  committed
  f1a7cc0

  Copy full SHA for f1a7cc0

• Implement MV changes. Some other small changes.

  ershockley

  committed
  f8707bd

  Copy full SHA for f8707bd

• Fix small bug in process.py

  ershockley

  committed
  2c3de99

  Copy full SHA for 2c3de99

Commits on Jun 9, 2017

• Small changes. Commit before dag_writer.py overhaul

  ershockley

  committed
  e7f218d

  Copy full SHA for e7f218d

Commits on Jun 11, 2017

• Add determine_rse.py so that jobs download from European rucio endpoints if at a European job site. Major overhaul of dag_writer.py accompanies this, as well as changes to pretty much everything

  ershockley

  committed
  bc08822

  Copy full SHA for bc08822

• Remove ershockley dependence

  ershockley

  committed
  e5a670b

  Copy full SHA for e5a670b

Commits on Jun 16, 2017

• More changes - hopefully to remove any ershockley dependence

  ershockley

  committed
  2dbd1d0

  Copy full SHA for 2dbd1d0

Commits on Jun 21, 2017

• Alfio's info

  ershockley

  committed
  d041356

  Copy full SHA for d041356

• Set GLIDEIN_Country to US if not set

  ershockley

  committed
  6b18a4a

  Copy full SHA for 6b18a4a

• Add dag_config

  ershockley

  committed
  bdde49a

  Copy full SHA for bdde49a

• Merge branch 'any_user' of https://github.com/XENON1T/cax into any_user

  ershockley

  committed
  a8785f2

  Copy full SHA for a8785f2

Commits on Jul 5, 2017

• More fixes to allow any user to run code

  ershockley

  committed
  c1744d1

  Copy full SHA for c1744d1

Commits on Jul 9, 2017

• dunno

  ershockley

  committed
  91b1b8b

  Copy full SHA for 91b1b8b

• Merge branch 'OSG_dev2' of https://github.com/XENON1T/cax into OSG_dev2

  ershockley

  committed
  c166b4b

  Copy full SHA for c166b4b

Commits on Jul 10, 2017

• multiple retries for copy back to stash

  ershockley

  committed
  0030597

  Copy full SHA for 0030597

Commits on Jul 14, 2017

• Change DB query to only process newest runs with v6.6.6

  ershockley

  committed
  948d14c

  Copy full SHA for 948d14c

• Add curl commands for job monitoring

  ershockley

  committed
  ea972a3

  Copy full SHA for ea972a3

• Fix bug in run_xenon.sh

  ershockley

  committed
  90046dd

  Copy full SHA for 90046dd

Commits on Jul 16, 2017

• Updates to make_runlist.py, some other small improvements

  ershockley

  committed
  f3878cc

  Copy full SHA for f3878cc

Commits on Jul 17, 2017

• Remove annoying print statement

  ershockley

  committed
  d5fab9d

  Copy full SHA for d5fab9d

• Merge

  ershockley

  committed
  4dfd0c3

  Copy full SHA for 4dfd0c3

Commits on Aug 11, 2017

• Clean up finished

  ershockley

  committed
  68b6a60

  Copy full SHA for 68b6a60

Commits on Aug 16, 2017

• Commit before merging with benedikt cleanup

  ershockley

  committed
  8cddb56

  Copy full SHA for 8cddb56

• Merge with benedikt_cleanup

  ershockley

  committed
  d092e2b

  Copy full SHA for d092e2b

Commits on Aug 17, 2017

• Revert back to old commit

  ershockley

  committed
  44d673e

  Copy full SHA for 44d673e

• dumb commit

  ershockley

  committed
  221317b

  Copy full SHA for 221317b

• Merge branch 'revert' into OSG_dev2

  ershockley

  committed
  0e697eb

  Copy full SHA for 0e697eb

Commits on Aug 22, 2017

• Improvements with permissions setting, etc.

  ershockley

  committed
  889e932

  Copy full SHA for 889e932

Commits on Aug 25, 2017

• Hopefully fix for any user to use again

  ershockley

  committed
  9caed6d

  Copy full SHA for 9caed6d

Commits on Aug 29, 2017

• Add some hardcoded pieces but hopefully will result in less errors

  ershockley

  committed
  305e4ec

  Copy full SHA for 305e4ec

Commits on Sep 14, 2017

• Change settings for SR1 reprocessing

  ershockley

  committed
  b6000e3

  Copy full SHA for b6000e3

• Update hadd_and_upload.sh

  ershockley

  authored
  de24493

  Copy full SHA for de24493

Commits on Sep 18, 2017

• Modify caxdir lines to use home directory

  ershockley

  committed
  58c948f

  Copy full SHA for 58c948f

Commits on Sep 19, 2017

• Try to remove gfal-debug stuff?

  ershockley

  committed
  7ee5800

  Copy full SHA for 7ee5800

Commits on Oct 26, 2017

• Shouldn't be any major changes - SR1 daily processing running normal

  ershockley

  committed
  4e781c8

  Copy full SHA for 4e781c8

Commits on Oct 31, 2017

• Small changes to dag_writer, adding SURFSARA

  ershockley

  committed
  8d2a1e7

  Copy full SHA for 8d2a1e7

Commits on Nov 6, 2017

• Remove gfal debug, add CNAF to some of the rucio scripts, other small changes

  ershockley

  committed
  55fabbf

  Copy full SHA for 55fabbf