Integrating Telemetry into GA4GH API-powered federated microservice networks

The Global Alliance for Genomics and Health (GA4GH) defines various web APIs for federated computing solutions that are well suited for the microservice architecture pattern. The ELIXIR Cloud & AAI Driver Project of the GA4GH is actively developing a GA4GH API-powered cloud environment based on this principle. The GA4GH Cloud Telemetry Project aims to provide a starting point for instrumentation and tracing of GA4GH Cloud Components / Microservices.

In this project, you would design and implement a solution that adds rich telemetry support to the ELIXIR Cloud federated service. The primary goal would be deployment artifacts (e.g., Terraform/OpenTofu) that deploy both our core services, as well as telemetry services at our fragmented compute centers at multiple locations (Kubernetes, OpenShift, OpenStack/VMs), as well as on one or more commercial cloud providers. An ideal solution would be easily extensible to apply to GA4GH-powered clouds in general, i.e., proposals that use abstractions offered by GA4GH APIs, where applicable/reasonable, are preferred.

A simple telemetry sidecar deployment might be a reasonable aim for a short project, whereas a generic solution with various connections to specific GA4GH APIs (e.g., integration with the Service Info / Service Registry API) might be suitable for a medium or even long project. Note that the exclusive use of free and open source tooling for any used dependencies (e.g., OpenTelemetry) is a requirement.

Table of Contents

• Basic Usage
• Installation
• Development
• Contributing
• Code of Conduct
• Versioning
• License
• Contact

Basic Usage

Installation

Development

For ease of use, certain scripts have been abbreviated in Makefile, make sure that you have installed the dependencies before running the commands.

Note: make commands are only available for Unix-based systems.

To view the commands available, run:

make

Here are certain commands that you might find useful:

• Make a virtual environment

make v

• Install all dependencies including optional dependencies

make i

Note: This project uses optional dependency groups such as types, code_quality, docs, vulnerability, test, and misc. To install stubs or types for the dependencies, you must use the following command:

poetry add types-foo --group types

Replace types-foo with the name of the package for the types. All runtime dependencies should be added to the default group. For example, to install requests and its type stubs, run:

poetry add requests
poetry add types-requests --group types

This ensures that the type checker functions correctly.

Note: Since the dependencies are segregated into groups, if you add a new group make sure to add it in make install command in Makefile.

• Run tests

make t

• Run linter, formatter and spell checker

make fl

• Build the documentation

make d

Note: If you make changes to the code, make sure to generate and push the documentation using above command, else the documentation check CI will fail. Do NOT edit auto-generated documentation manually.

• Run type checker

make tc

• Run all pre-commit checks

make pc

• Update the cookiecutter template

make u

Note: This is not the complete list of commands, run make to find out if more have been added.

Contributing

This project is a community effort and lives off your contributions, be it in the form of bug reports, feature requests, discussions, fixes or any other form of contribution!

Please refer to the guidelines available at CONTRIBUTING.md if you are interested in contributing.

Code of Conduct

We kindly request all contributors to abide by our organization's Code of Conduct. Please also refer to this document if you want to report an incident with respect to conduct in our community. Thank you for your cooperation.

Versioning

The project adopts the semantic versioning scheme for versioning. Currently the software is in a pre-release stage, so changes to the API, including breaking changes, may occur at any time without further notice.

License

This project is distributed under the Apache License 2.0, a copy of which is also available in LICENSE.

Contact

The project is maintained by ELIXIR Cloud & AAI, a Driver Project of the Global Alliance for Genomics and Health (GA4GH), under the umbrella of the ELIXIR Compute Platform.

To get in touch with us, please use one of the following routes:

• For filing bug reports, feature requests or other code-related issues, please make use of the project's issue tracker.
• For private/personal issues, more involved communication, or if you would like to join our team as a regular contributor, you can either join our chat board or email the community leaders.
• Project Board
• Communication: ELIXIR Cloud Slack Channel, Topic #telemetry