Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spawn reactive runner #8

Merged
merged 21 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 25 additions & 7 deletions src-docs/reactive.consumer.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,12 @@ Module responsible for consuming jobs from the message queue.

---

<a href="../src/github_runner_manager/reactive/consumer.py#L36"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/github_runner_manager/reactive/consumer.py#L50"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>function</kbd> `consume`

```python
consume(mongodb_uri: str, queue_name: str) → None
consume(queue_config: QueueConfig, runner_manager: RunnerManager) → None
```

Consume a job from the message queue.
Expand All @@ -24,8 +24,8 @@ Log the job details and acknowledge the message. If the job details are invalid,

**Args:**

- <b>`mongodb_uri`</b>: The URI of the MongoDB database.
- <b>`queue_name`</b>: The name of the queue.
- <b>`queue_config`</b>: The configuration for the message queue.
- <b>`runner_manager`</b>: The runner manager used to create the runner.



Expand All @@ -36,7 +36,7 @@ Log the job details and acknowledge the message. If the job details are invalid,

---

<a href="../reactive/consumer/signal_handler#L66"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../reactive/consumer/signal_handler#L80"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>function</kbd> `signal_handler`

Expand All @@ -57,7 +57,7 @@ The signal handler exits the process.

---

<a href="../src/github_runner_manager/reactive/consumer.py#L20"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/github_runner_manager/reactive/consumer.py#L23"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>class</kbd> `JobDetails`
A class to translate the payload.
Expand All @@ -75,7 +75,25 @@ A class to translate the payload.

---

<a href="../src/github_runner_manager/reactive/consumer.py#L32"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/github_runner_manager/reactive/consumer.py#L34"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>class</kbd> `QueueConfig`
The configuration for the message queue.



**Attributes:**

- <b>`mongodb_uri`</b>: The URI of the MongoDB database.
- <b>`queue_name`</b>: The name of the queue.





---

<a href="../src/github_runner_manager/reactive/consumer.py#L46"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>class</kbd> `JobError`
Raised when a job error occurs.
Expand Down
25 changes: 23 additions & 2 deletions src-docs/reactive.runner.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Script to spawn a reactive runner process.

---

<a href="../src/github_runner_manager/reactive/runner.py#L14"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/github_runner_manager/reactive/runner.py#L20"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>function</kbd> `setup_root_logging`

Expand All @@ -25,7 +25,7 @@ Set up logging for the reactive runner process.

---

<a href="../src/github_runner_manager/reactive/runner.py#L24"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
<a href="../src/github_runner_manager/reactive/runner.py#L48"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>function</kbd> `main`

Expand All @@ -42,3 +42,24 @@ Spawn a process that consumes a message from the queue to create a runner.
- <b>`ValueError`</b>: If the required environment variables are not set


---

<a href="../src/github_runner_manager/reactive/runner.py#L30"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>

## <kbd>class</kbd> `ReactiveRunnerConfig`
The configuration for the reactive runner to spawn.



**Attributes:**

- <b>`queue`</b>: The queue configuration.
- <b>`runner_manager`</b>: The runner manager configuration.
- <b>`runner`</b>: The GitHub runner configuration.
- <b>`openstack_cloud`</b>: The OpenStack cloud configuration.
- <b>`openstack_server`</b>: The OpenStack server configuration.





20 changes: 17 additions & 3 deletions src/github_runner_manager/reactive/consumer.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@
from kombu import Connection
from kombu.simple import SimpleQueue
from pydantic import BaseModel, HttpUrl, ValidationError
from pydantic.networks import MongoDsn

from github_runner_manager.manager.runner_manager import RunnerManager

logger = logging.getLogger(__name__)

Expand All @@ -28,20 +31,31 @@ class JobDetails(BaseModel):
labels: list[str]
run_url: HttpUrl

class QueueConfig(BaseModel):
"""The configuration for the message queue.

Attributes:
mongodb_uri: The URI of the MongoDB database.
queue_name: The name of the queue.
"""

mongodb_uri: MongoDsn
queue_name: str


class JobError(Exception):
"""Raised when a job error occurs."""


def consume(mongodb_uri: str, queue_name: str) -> None:
def consume(queue_config: QueueConfig, runner_manager: RunnerManager) -> None:
"""Consume a job from the message queue.

Log the job details and acknowledge the message.
If the job details are invalid, reject the message and raise an error.

Args:
mongodb_uri: The URI of the MongoDB database.
queue_name: The name of the queue.
queue_config: The configuration for the message queue.
runner_manager: The runner manager used to create the runner.

Raises:
JobError: If the job details are invalid.
Expand Down
26 changes: 25 additions & 1 deletion src/github_runner_manager/reactive/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,13 @@
import os
import sys

from github_runner_manager.reactive.consumer import consume
from pydantic import BaseModel

from github_runner_manager.manager.cloud_runner_manager import GitHubRunnerConfig
from github_runner_manager.manager.runner_manager import RunnerManagerConfig
from github_runner_manager.openstack_cloud.openstack_runner_manager import OpenStackCloudConfig, \
OpenStackServerConfig
from github_runner_manager.reactive.consumer import consume, QueueConfig
from github_runner_manager.reactive.runner_manager import MQ_URI_ENV_VAR, QUEUE_NAME_ENV_VAR


Expand All @@ -21,6 +27,24 @@ def setup_root_logging() -> None:
)


class ReactiveRunnerConfig(BaseModel):
"""The configuration for the reactive runner to spawn.

Attributes:
queue: The queue configuration.
runner_manager: The runner manager configuration.
runner: The GitHub runner configuration.
openstack_cloud: The OpenStack cloud configuration.
openstack_server: The OpenStack server configuration.
"""

queue: QueueConfig
runner_manager: RunnerManagerConfig
runner: GitHubRunnerConfig
openstack_cloud: OpenStackCloudConfig
openstack_server: OpenStackServerConfig | None = None


def main() -> None:
"""Spawn a process that consumes a message from the queue to create a runner.

Expand Down
Loading