Project to migrate WMAgent to a work unit level handling through late materialization of jobs

Overview

The idea is for the Workflow Management and Submission system to be able to dynamically adjust the size of jobs as needed. For example, if we match on a resource with high number of CPUs per node but limited number of jobs in the batch system (typical HPC case), we should be able to increase the length of many core jobs to use the CPUs as efficiently as possible. This requires being able to materialize the jobs only after the workflow has been matched/assigned to a set of resources, with the job length depending on some inputs defined by in the resource (e.g.: the job length in traditional resources vs HPC resources would be different).

We would like to have the WMAgents think as much as they can at a work unit/package level without thinking in the job level layer. Migrating WMAgent to manage these work units, such that each job in the end can be composed of one or more work units. Also, partial job success needs to handled in the system, returning the non-processed work units back to the system to create other jobs(s). Additionally, we need to think about how taskChain (multiple cmsRuns per job) impacts this process.

Regarding late materialization

HTCondor has a new feature that allows the late materialization of jobs in the Scheduler to enable submission of very large sets of jobs. More jobs are materialized once the number of idle jobs drops below a threshold, in a similar fashion to DAGMan throttling.

HTCondor versions supporting this

• HTCondor 8.6.x (current versions on CERN schedds) does not support late materialization.
• HTCondor 8.7.1 supports late materialization, except for condor_dagman.
• HTCondor >8.7.4 / has condor_dagman support.

This is becoming part of the new 8.8 stable release branch

Documentation

At present (18 February, 2019), there is no documentation regarding late materialization, but there have been live demos of this functionality during HTCondor Weeks (8.8.0 was released in January, 2019, including this feature, but there is still no documentation of this).

However, the HTCondor github repo has some perl and python-based examples of it. So, it looks like not only condor_submit but also the python bindings support this feature.

• Using condor_submit -factory
• Using late materialization with condor_dagman
• Python bindings example for late materialization

Pool Configuration

Schedulers need to include the following in the condor configuration:

SCHEDD_ALLOW_LATE_MATERIALIZATION = True
SCHEDD_MATERIALIZE_LOG = /path/to/file
SUBMIT_FACTORY_JOBS_BY_DEFAULT = True

Job submission examples

Using condor submit

Using this feature requires a single additional classad in the submit file:

# submit.jdl
universe = vanilla
executable = x_sleep.pl
arguments = 3
log = $submitfile.log
notification = never
max_materialize = 4 # Maximum number of materialized jobs in the queue
queue 10

and the submission is done via $ condor_submit <submit.jdl> -factory

Using python bindings

Python bindings can be used in the regular way, including the new classad option. E.g.:

import classad
import htcondor
sub = htcondor.Submit("""
	universe = vanilla
	executable = x_sleep.pl
	arguments = 3
	notification = never
	max_materialize = 4
	queue 10
	""")

logfile = 'logfile.log'
sub['log'] = logfile

schedd = htcondor.Schedd()
with schedd.transaction() as txn:
	cluster = sub.queue(txn)

Note: There also seems to be a max_idle option that can replace max_materialize.