Task distribution for GA4GH TES instances.
Proof of concept implementation of a task distribution logic for a federated network of GA4GH Task Execution Service (TES) instances.
You can use TEStribute in three ways:
- Call the HTTP API service, e.g. with
curl:
curl -X POST SERVICE_URI -H "Content-Type: application/json" -d PAYLOAD- Call the console script:
testribute [-h] --tes-uri URI --cpu-cores INT --ram-gb FLOAT --disk-gb FLOAT
--execution-time-sec INT [--jwt TOKEN] [--object-id ID] [--drs-uri
URI] [-m MODE] [-v]- Call the main function directly, from within your Python code:
from TEStribute import rank_services
rank_services(...)Given a set of available GA4GH Task Execution Service (TES) instances, a task's compute resource requirements, the Data Repository Service (DRS) object identifiers of all task inputs (if any), and a list of DRS instances where these objects might be obtained from, TEStribute returns a list of combinations of TES instances and input object locations, rank-ordered according to either increasing estimated total costs, increasing estimated total processing times, or a weighting factor that balances both of these properties.
The application currently relies on [modifications] to the TES specifications
and assumes that DRS object identifiers are globally unique (i.e., a given
identifier will point to the same exact file on any DRS instance), which is
not warranted by current DRS specs. More detailed information on these
requirements is available at mock-TES and mock-DRS, mockup services which
implement these modifications/assumptions. The corresponding clients TES-cli
and DRS-cli are used within TEStribute to interact with these services.
Ensure you have the following software installed:
- Docker (18.06.1-ce, build e68fc7a)
- docker-compose (1.23.1, build b02f1306)
- Git (tested with version 2.17.1)
Clone repository and start Docker service
git clone https://github.com/elixir-europe/TEStribute.git app
cd app
docker-compose up --build --detachYou can explore the HTTP API via the Swagger UI:
firefox http://localhost:7979/ui/Ensure you have the following software installed:
Install package and testribute console script:
pip install TEStributeThe following properties/options are available when running TEStribute, regardless of whether the software is run as an HTTP API service, as a console script or directly from within your Python code. The CLI option is indicated in parentheses in those cases where it differs from API / import usage:
object_ids(object-id): DRS IDs of objects required by the task. When using the console script, indicate the option multiple times to pass multiple arguments.drs_uris(drs-id): URIs of DRS instances that objects may be read from or written to. When using the console script, indicate the option multiple times to pass multiple arguments.mode: Defines how service combinations are ranked, either by 'cost', 'time' or both. For the latter, specify a number between 0 and 1, with the boundaries representing weights at which services are ranked entirely by cost and time, respectively. It is also possible to randomize rankings (specify 'random' or -1).resource_requirements(not available as CLI option, use properties directly): Map of resources required for the task:cpu_cores(cpu-cores): Requested number of CPUs.disk_gb(disk-gb): Requested disk size in gigabytes (GB).execution_time_sec(execution-time-sec): Requested execution time in seconds (s).preemptible(not available as CLI option): Is the task allowed to run on preemptible compute instances (e.g. AWS Spot)? Currently not used.ram_gb: Requested RAM required in gigabytes (GB).zones(not available as CLI option): Request that the task be run in these compute zones. Currently not used.
tes_uris(tes-uri): URIs of known TES instances that the task may be computed on. When using the console script, indicate the option multiple times to pass multiple arguments.jwt: JSON Web Token (JWT) bearer token that is attached as anAuthorizationrequest header, following the keywordBearer, to any outgoing service call, if provided, in order to ascertain whether the user has permissions to access resources and/or whether user-specific policies or contstraints apply (e.g., custom prices, discounts, quotas). Note that when using TEStribute through the HTTP API, this property is not available. Instead, the token value itself needs to be passed as anAuthorizationrequest header, also following theBearerkeyword.
For more details, including typing information, explore the API definition, which also forms the basis for validating CLI arguments and the inputs to the
rank_services()function.
The following are equivalent calls for either of the TEStribute entry points defined above. Note that the provided TES and DRS URIs point to test instances of the services which may or may not be up and running at any given time. Therefore, the success of the calls cannot be guaranteed.
{
"object_ids": [
"a001",
"a002"
],
"drs_uris": [
"http://131.152.229.71/ga4gh/drs/v1/",
"http://193.166.24.114/ga4gh/drs/v1/"
],
"mode": 0.5,
"resource_requirements": {
"cpu_cores": 1,
"disk_gb": 1,
"execution_time_sec": 1800,
"ram_gb": 1
},
"tes_uris": [
"http://131.152.229.70/ga4gh/tes/v1/",
"http://193.166.24.111/ga4gh/tes/v1/"
]
}testribute \
--tes-uri="http://131.152.229.70/ga4gh/tes/v1/" \
--tes-uri="http://193.166.24.111/ga4gh/tes/v1/" \
--cpu-cores=1 \
--ram-gb=1 \
--disk-gb=1 \
--execution-time-sec=1800 \
--object-id="a001" \
--object-id="a002" \
--drs-id="http://131.152.229.71/ga4gh/drs/v1/" \
--drs_id="http://193.166.24.114/ga4gh/drs/v1/" \
--mode=0.5from TEStribute import rank_services
rank_services(
object_ids=[
"a001",
"a002"
],
resource_requirements={
"cpu_cores": 1,
"ram_gb": 1,
"disk_gb": 1,
"execution_time_sec": 1800
},
tes_uris=[
"http://131.152.229.70/ga4gh/tes/v1/",
"http://193.166.24.111/ga4gh/tes/v1/"
],
drs_uris=[
"http://131.152.229.71/ga4gh/drs/v1/",
"http://193.166.24.114/ga4gh/drs/v1/"
],
mode=0.5,
jwt=None
)Upon success, the API service returns a JSON object such as this:
{
"service_combinations": [
{
"access_uris": {
"a001": "ftp://ftp.ensembl.org/pub/release-96/fasta/homo_sapiens/dna//Homo_sapiens.GRCh38.dna.chromosome.19.fa.gz",
"a002": "ftp://ftp.ensembl.org/pub/release-81/bed/ensembl-compara/11_teleost_fish.gerp_constrained_elements/gerp_constrained_elements.tetraodon_nigroviridis.bed.gz",
"tes_uri": "http://193.166.24.111/ga4gh/tes/v1/"
},
"cost_estimate": {
"amount": 294727.1443451331,
"currency": "EUR"
},
"rank": 1,
"time_estimate": 2514
},
{
"access_uris": {
"a001": "ftp://ftp.ensembl.org/pub/release-96/fasta/homo_sapiens/dna//Homo_sapiens.GRCh38.dna.chromosome.19.fa.gz",
"a002": "ftp://ftp.ensembl.org/pub/release-81/bed/ensembl-compara/11_teleost_fish.gerp_constrained_elements/gerp_constrained_elements.tetraodon_nigroviridis.bed.gz",
"tes_uri": "http://131.152.229.70/ga4gh/tes/v1/"
},
"cost_estimate": {
"amount": 294697.1938522269,
"currency": "EUR"
},
"rank": 2,
"time_estimate": 3298
}
],
"warnings": []
}You can check out the Response model in the API definition for more details.
For the other entry points, the general response upon success is the same, but
provided in different ways. When calling rank_services() directly from within
Python code, the response is an instance of Python class Response, which is
based on the corresponding model in the API definition and defined in module
TEStribute.models.response. It can be converted to dictionary form with:
response = rank_service(...)
response.to_dict()It can be further converted to JSON with:
import json
json.dumps(response.to_dict())When using the testribute console script, the JSONified response is printed to
STDOUT.
In case of failure, the API service returns a JSON object of the following form:
{
"code": 400,
"errors": [
{
"reason": "werkzeug.exceptions.BadRequest",
"message": [
"Services cannot be ranked. None of the specified TES instances provided any task info."
]
}
],
"message": "The request caused an error."
}When using the console script testribute, an error will lead to the script
exiting with a non-zero return code. In addition, warnings and errors are
written to the log which is streamed to STDERR, e.g.:
[WARNING] TES unavailable: the provided URI 'http://i.do.not.exist/' could not be resolved.
[ERROR] ResourceUnavailableError: Services cannot be ranked. None of the specified TES instances provided any task info.When calling rank_services() directly from within Python code, traceback
information for any error is provided, too. For
example:
[WARNING] TES unavailable: the provided URI 'http://i.do.not.exist/' could not be resolved.
Traceback (most recent call last):
File "<stdin>", line 21, in <module>
File "/home/uniqueg/Dropbox/repos/TEStribute/TEStribute/__init__.py", line 129, in rank_services
target_currency=models.Currency[config["target_currency"]],
File "/home/uniqueg/Dropbox/repos/TEStribute/TEStribute/models/response.py", line 55, in __init__
timeout=self.timeout,
File "/home/uniqueg/Dropbox/repos/TEStribute/TEStribute/utils/service_calls.py", line 311, in fetch_tes_task_info
"Services cannot be ranked. None of the specified TES instances " \
TEStribute.errors.ResourceUnavailableError: Services cannot be ranked. None of the specified TES instances provided any task info.It is possible to configure some settings of the app, e.g., how JWTs are parsed, processed and forwarded or in which prices costs are reported, by modifying the the config file before starting the service / running TEStribute.
Unit and integration tests can be run with the following command:
pytestNote that test coverage is currently sparse and tests are unstable.
This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, or fixes and other code changes. Please read the contributing guidelines if you want to contribute. And please mind the code of conduct for all interactions with the community.
Development of the app is currently still in alpha stage, and current versioning is for internal use only. In the future, we are aiming to adopt semantic versioning that is synchronized to the versioning of mock-TES, TES-cli, mock-DRS, and DRS-cli in order to ensure that these apps will be compatible as long as both their major and minor versions match.
This project is covered by the Apache License 2.0 also available shipped with this repository.
Please contact the project leader for inquiries, proposals, questions etc. that are not covered by the Contributing section.
The project is a collaborative effort under the umbrella of the ELIXIR Cloud and AAI group. It was started during the 2019 Google Summer of Code as part of the Global Alliance for Genomics and Health organization.
