Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more documentation #80

Merged
merged 10 commits into from
Mar 1, 2024
Merged
3 changes: 3 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,12 @@ ADD requirements.txt /src/requirements.txt

WORKDIR "/src"


RUN pip install -r requirements.txt
ADD . /src

EXPOSE 9999

ENV IN_CONTAINER=1

CMD [ "python3", "-m", "JobRunner.Callback"]
2 changes: 1 addition & 1 deletion JobRunner/Callback.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
class Callback():
def __init__(self):
workdir = os.environ.get("JOB_DIR", '/tmp/')
self.conf = Config(workdir=workdir, use_ee2=False)
self.conf = Config(job_id="callback", workdir=workdir, use_ee2=False)
self.ip = os.environ.get('CALLBACK_IP', get_ip())
self.port = os.environ.get('CALLBACK_PORT')
self.cbs = None
Expand Down
80 changes: 80 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,86 @@

The job runner is started by the batch/resource manager system and is the component that actually executes the SDK module. It also provides the callback handler and pushes logs to the execution engine logging system.

## Callback Server Mode

It is possible to run the callback server in a standalone mode. This can be used to speed up
testing of SDK apps (avoiding a full make test cycle) or to allow some automated operations.
The callback server can be launched in a few ways.

### Native Launch

To launch the callback server natively you can do the following.

```
pip install .
# Set the DOCKER_HOST if this doesn't work out of the box
export DOCKER_HOST=unix://$HOME/.docker/run/docker.sock

# Set job dir where work output will go
export JOB_DIR=/full/path/to/work/area

# Set tokens and URL
export KB_AUTH_TOKEN="xxxxxxxxx"
export KB_ADMIN_AUTH_TOKEN="xxxxxxxxxxxx"
export KB_BASE_URL=https://ci.kbase.us/services

# Optional
export KB_REF_DATA=/path/to/local/refdata

# Launch the callback server
python -m JobRunner.Callback
```

### Container Launch

The callback server can also be launched via a container. This may be useful
where the local python and what works with the JobRunner are incompatible.
Note that the JOB_DIR needs to be accessible by docker. Also you must pass through
the docker socket to the container. The path to the socket can vary depending on
the container runtime and how the container runtime is configured.

```
export JOB_DIR=/full/path/to/work/area
export KB_AUTH_TOKEN="xxxxxxxxx"
export KB_ADMIN_AUTH_TOKEN="xxxxxxxxxxxx"

docker run --name cb -d \
-e KB_AUTH_TOKEN \
-e KB_ADMIN_AUTH_TOKEN \
-e KB_BASE_URL=https://ci.kbase.us/services \
-e CALLBACK_PORT=9999 \
-e JOB_DIR \
-v $JOB_DIR:$JOB_DIR \
-v /var/run/docker.sock:/run/docker.sock \
-p 9999:9999 \
ghcr.io/kbase/jobrunner:latest-rc

export SDK_CALLBACK_URL=http://localhost:9999
```

## Development and Testing the Job Runner

Here is a quick start guide for running test for the Job Runner code.
See the SDK guide for information about running test of SDK apps.

```
pip install -r requirements.txt -r requirements-dev.txt

# Set the DOCKER_HOST if this doesn't work out of the box
export DOCKER_HOST=unix://$HOME/.docker/run/docker.sock

# Set tokens and URL
export KB_AUTH_TOKEN="xxxxxxxxx"
export KB_ADMIN_AUTH_TOKEN="xxxxxxxxxxxx"
export KB_BASE_URL=https://ci.kbase.us/services

# Set ref data to an area acceessible by Docker
export KB_REF_DATA=/path/to/local/refdata

make mock
make testimage
make test
```

## Debug Mode

Expand Down
Loading