Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tickets/dm 29952 - Uberjobs proof-of-concept #617

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

Tickets/dm 29952 - Uberjobs proof-of-concept #617

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
7ef3f19
Added loop for making UberJobs.
jgates108 May 3, 2021
6589959
Added JobBase class.
jgates108 May 4, 2021
52245e7
Filled in most of the methods needed for UberJob.
jgates108 May 6, 2021
001af50
Fixed issue, unit test required all requests to have the same chunkId.
jgates108 May 7, 2021
0083ad3
Fixed a race condition.
jgates108 May 10, 2021
358f9f8
Added code to build the UberJobMsg.
jgates108 May 11, 2021
a683124
Added code to turn and UberJob into Tasks and run them.
jgates108 May 12, 2021
275a7ee
Made changes to ResourceMonitor related code to work with UberJobs.
jgates108 May 13, 2021
9cff46c
Added some code for mono-node integration tests.
jgates108 May 17, 2021
6d41224
Changed some maps to use shared pointers.
jgates108 May 18, 2021
269068d
Changed UberJob building algorithm.
jgates108 May 18, 2021
4140dc9
Minor changes for integration tests.
jgates108 Jun 7, 2021
e4aa4a1
Integration tests working.
jgates108 Jun 8, 2021
ecc1b8e
Removed dead code.
jgates108 Jun 8, 2021
a827fcc
Added code to load worker chunk resource info from file.
jgates108 Jun 11, 2021
0356e4f
Changed to use wise_01 database.
jgates108 Jun 11, 2021
26d928f
Added code for testing.
jgates108 Jun 18, 2021
a366fbc
Logging changes for xrootd testing.
jgates108 Jul 6, 2021
d64dc05
Added workerchunkdata.txt.
jgates108 Jul 19, 2021
54c5e3d
Moved a wait to a separate function so it's easier to find in gdb.
jgates108 Jul 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions admin/templates/configuration/etc/log4cxx.czar.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
# used by Qserv-czar
#

log4j.rootLogger=INFO, CONSOLE
#log4j.rootLogger=INFO, CONSOLE //&&&
log4j.rootLogger=ERROR, CONSOLE

# Use this appender to interlace xrootd debug messages with Qserv-czar standard
# logs. stdout/stderr are redirected to {{QSERV_LOG_DIR}}/qserv-czar-console.log.
Expand All @@ -26,4 +27,6 @@ log4j.appender.FILE.layout.conversionPattern=%d{yyyy-MM-ddTHH:mm:ss.SSSZ} %X %-5
#log4j.logger.lsst.qserv.qproc=DEBUG
#log4j.logger.lsst.qserv.util=DEBUG
#log4j.logger.lsst.qserv.qana=DEBUG
log4j.logger.lsst.qserv.xrdssi.msgs=WARN
#log4j.logger.lsst.qserv.xrdssi.msgs=WARN //&&&
log4j.logger.lsst.qserv.xrdssi.msgs=ERROR

6 changes: 4 additions & 2 deletions admin/templates/configuration/etc/log4cxx.worker.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,13 @@
# Configuration file for log4cxx
#

log4j.rootLogger=DEBUG, CONSOLE
#log4j.rootLogger=DEBUG, CONSOLE //&&&
log4j.rootLogger=ERROR, CONSOLE

# Appender for xrootd log file
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=[%d{yyyy-MM-ddTHH:mm:ss.SSSZ}] %X %-5p %c{2} (%F:%L) - %m%n

log4j.logger.lsst.qserv.xrdssi.msgs=WARN
#log4j.logger.lsst.qserv.xrdssi.msgs=WARN &&&
log4j.logger.lsst.qserv.xrdssi.msgs=ERROR
3 changes: 3 additions & 0 deletions admin/templates/configuration/etc/lsp.cf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,9 @@ xrd.port {{XROOTD_PORT}}

# Uncomment the following line for detailed xrootd debugging
# xrootd.trace all debug
xrootd.trace fs request response

ssi.trace all debug

# Set limits for the number of threads open by XRootD services
# See more details on these parameters at:
Expand Down
6 changes: 4 additions & 2 deletions admin/templates/configuration/etc/qserv-czar.cnf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,14 +106,16 @@ qMetaSecsBetweenChunkCompletionUpdates = 59
# Please see qdisp/QdispPool.h QdispPool::QdispPool for more information
[qdisppool]
#size of the pool
poolSize = 50
#poolSize = 50
poolSize = 1500
# Low numbers are higher priority. Largest priority 3 creates 4 priority queues 0, 1, 2, 3
# Must be greater than 0.
largestPriority = 3
# Maximum number of threads running for each queue. No spaces. Values separated by ':'
# Using largestPriority = 2 and vectRunsizes = 3:5:8
# queue 0 would have runSize 3, queue 1 would have runSize 5, and queue 2 would have runSize 8.
vectRunSizes = 50:50:50:50
#vectRunSizes = 50:50:50:50
vectRunSizes = 1500:1500:1500:1500
# Minimum number of threads running for each queue. No spaces. Values separated by ':'
vectMinRunningSizes = 0:1:3:3

Expand Down
4 changes: 2 additions & 2 deletions admin/templates/configuration/etc/sysconfig/qserv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ ulimit -c unlimited
export PYTHONUNBUFFERED=1

export XRD_LOGLEVEL=Debug
unset XRDSSIDEBUG
#export XRDSSIDEBUG=0
#unset XRDSSIDEBUG
export XRDSSIDEBUG=1

# Increase timeouts, without that, long-running queries fail.
# The proper fix is coming, see DM-3433
Expand Down
Loading