Externalise Runner Manager (#358)

* Spawn a manual test env

* Disable spawning on manual test env

* Remove useless class

* Fix runner deletion

* Fix import error

* Add more docs

* Fix get no-existing openstack server

* Add debug statement

* Fix variable name and function name mixup

* Fix id variable name, function name mixup

* Add debug statement

* Move debug

* Add busy runner test

* Add debug statement.

* Disable some test

* Disable some test

* Fix runner label in workflow

* Fix lambda

* Debug

* Debug

* Add debug

* Start new manual test env

* Add none check

* Fix missing prefix

* Add more logging

* Refactor runner manager one runner fixture

* Fix error string formatting

* Adding the docstring for github_runner_manager

* Fix test fixture scope

* Add docstring on cloud_runner_manager

* Add debug

* Fix docstring for cloud runner manager

* Add more docstrings

* Add metrics for deleted and cleanup runners

* Enable tests again

* Add debug

* Get runner info not on GitHub

* Fix dict access

* Add debug of userdata

* Fix metric path

* Debug metric

* Fix variable naming

* Test

* Fix iterator

* Debug

* Debug

* Fix for iterator return value

* Add more log path patching

* Fix path naming

* Fix monkey patch

* Start a arm64 manual test env

* Not spawning manual test env

* Update fmt

* Fix metric storage implementation for openstack

* Fix metric storage provider usage in openstack runner manager

* Debug

* Fix iterator

* Add debug

* Fix None in iterator

* Add debug

* Trying fix for get runner filter

* Add test

* Patch the path for logs

* Add cleanup test

* Debug

* Fix github state determining busy runner

* Fix wrong naming for method in ABC

* Remove debugging

* Add more docstrings

* Fix runner deletion

* Add more docs

* Fix typing

* Debug

* Update SSH health check

* Tmp disable a passing test

* Add deubg

* Remove a debug

* Fix Cloud runner state init

* Change clean up to cleanup

* Fix attr naming issue in openstack cloud

* Fix reference to non-existing instance_name in openstack cloud

* Add metric log processing to test

* Enable all tests

* Fix health check return value

* Fix all flake8 lints

* Fix test

* Fix all lints

* Fix unit test issue due to method sig change

* Ignore openstack cloud from coverage due to the test requires private endpoint

* Enable all tests

* Remove a repeated test

* Re-enable test.yaml

* Fix integration tests workflwo

* Add docs on cleanup method of cloud runner manager

* Add parallel spawning of runners.

* Enable dev testing

* Fix parallel spawn

* Allow openstack server to take a bit of time on deletion

* Refactor test detection of no runners

* Re-enable the tests

* Fix lints

* Disable tests again

* Disable some test

* Add wait until runner is running

* Enable openstack runner manager tests

* Add debug

* Wait for github state

* Refactor wait until runner spawn

* Add keyfile erorr

* Remove debug statement

* Re-enable all tests

* Update src/manager/github_runner_manager.py

Co-authored-by: Yanks Yoon <[email protected]>

* Update src/openstack_cloud/openstack_cloud.py

Co-authored-by: Yanks Yoon <[email protected]>

* Suggestions

* Refactor remove openstack server

* Test spawning two runners.

* Fix test

* Fix naming

* Fix according comment

* Fix clouds yaml write issue.

* Fix format

* Add delete runner by amount

* Add getting runner health state for metrics

* Fix security group ID issues

* Fix according to review

* Refactor health state for runner

* Fix lint issues

* Add missing docs

* Update the github state enum to use auto

* Rename class to fit convension

* Fix according to review

* Fix name_prefix property cloud runner manager

* Add class for scaling runners

* Fix lints

* Fix unit test

* Fix according to review comment

* Fix test according to comments

* Fix unit test

* Fix typo of attr

* Add debug

* Add debug statement

* Debug

* Fix return code of the kill command

* Remove debug

* Add comments on the flush kill command

* Add debug

* Fix debug

* Debug

* Debug

* Remove debug

* Add cleanup during idle and busy runner test

* Debug

* Disable tests during debug

* Debug missing keyfiles

* Fix keyfile path matching issue

* testing

* debug

* Add debug

* Use OR

* debug

* Debug

* Debug

* Debug

* Debug

* Debug

* Fix flush mode

* Remove debug

* Re-enable all tests

* Initial unit test for runner scaler

* Add more unit tests for runner scaler

* Add more tests

* Fix merge issues

* Fix states in get_runners methods

* Add docstring for unit test mocks

* Fix construction of repo-policy-compliance from config

* Fix get_runners action output

* Fix the lints

* Fix a naming issue

* Fix naming prefix of runner

* Improve unit test

* Remove the old OpenstackRunnerManager

* Fix test contstruction of runner manager.

* Fix flavor naming

* Fix flush action result output.

* Fix flavor of metric

* Testing out a integration test fikx

* change flush runner to flush idle.

* Add debug in integration test

* Manual test mode

* Start new manual test env

* Spawn x64 manual test env.

* Improve logging during reconcile

* Fix crashed metric collection

* Remove debug workflow

* Format

* Test

* externalise

* fix tests

* Add reactive back in

* Fix flushing of runners

* Debug workflow

* Add debug

* Fix logging of health state

* Remove debug

* Debug

* Fix set contruction

* Fix SSH key path in integration test setup

* Add more checks to repo-policy-compliance setup in tests

* Fix key path check

* Fix format string issue

* Fix format string typo

* fix integration test import

* outcomment externalised workflow

* Add some logging of test setup

* Fix missing await

* Revert config-change flushing

* Add maintance status for image relation change

* Fix HTTP format

* Revert "outcomment externalised workflow"

This reverts commit e0a78af.

* re-checkin integration test

* use github types from externalised app

* use github types from externalised app

* Update coverage ignore of github_runner_manager

* Minor fix in test comments

* lint and fix unit tests

* fix merge

* remove reactive script

* fix merge

* remove unused OpenstackUnauthorizedError

* final new line

* remove code duplication

* remove duplicate src-docs

* remove openstack-userdata.sh.j2

* pin commit in github-runner-manager

---------

Co-authored-by: yhaliaw <[email protected]>
Co-authored-by: Yanks Yoon <[email protected]>

pyproject.toml

@@ -25,7 +25,7 @@ omit = [
 ]
 
 [tool.coverage.report]
-fail_under = 85
+fail_under = 83
 show_missing = true
 
 [tool.pytest.ini_options]

requirements.txt

@@ -14,3 +14,4 @@ PyYAML ==6.0.*
 pyOpenSSL==24.2.1
 kombu==5.4.0
 pymongo==4.8.0
+github_runner_manager @ git+https://github.com/canonical/github-runner-manager.git@1f310b22b99a94bd5429184191558426b014ee82

src-docs/charm.md

@@ -20,7 +20,7 @@ Charm for creating and managing GitHub self-hosted runner instances.
 
 ---
 
-<a href="../src/charm.py#L109"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../src/charm.py#L113"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>function</kbd> `catch_charm_errors`
 
@@ -46,7 +46,7 @@ Catch common errors in charm.
 
 ---
 
-<a href="../src/charm.py#L155"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../src/charm.py#L154"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>function</kbd> `catch_action_errors`
 
@@ -72,7 +72,7 @@ Catch common errors in actions.
 
 ---
 
-<a href="../src/charm.py#L102"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../src/charm.py#L106"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>class</kbd> `ReconcileRunnersEvent`
 Event representing a periodic check to ensure runners are ok. 
@@ -83,7 +83,7 @@ Event representing a periodic check to ensure runners are ok.
 
 ---
 
-<a href="../src/charm.py#L193"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../src/charm.py#L192"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ## <kbd>class</kbd> `GithubRunnerCharm`
 Charm for managing GitHub self-hosted runners. 
@@ -100,7 +100,7 @@ Charm for managing GitHub self-hosted runners.
  - <b>`ram_pool_path`</b>:  The path to memdisk storage. 
  - <b>`kernel_module_path`</b>:  The path to kernel modules. 
 
-<a href="../src/charm.py#L216"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
+<a href="../src/charm.py#L215"><img align="right" style="float:right;" src="https://img.shields.io/badge/-source-cccccc?style=flat-square"></a>
 
 ### <kbd>method</kbd> `__init__`