From a13b0513957881e0ff28e3134d0a92a35368e070 Mon Sep 17 00:00:00 2001 From: Surax98 Date: Tue, 18 Jun 2024 07:04:56 +0000 Subject: [PATCH] updated Readme Signed-off-by: Surax98 --- README.md | 152 +++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 121 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index 16b85f3..2ad23fd 100644 --- a/README.md +++ b/README.md @@ -1,43 +1,133 @@ -# Template for interTwin repositories +# :information_source: Overview -This repository is to be used as a repository template for creating a new interTwin -repository, and is aiming at being a clean basis promoting currently accepted -good practices. +![Interlink logo](./docs/static/img/interlink_logo.png) -It includes: +## Introduction -- License information -- Copyright and author information -- Code of conduct and contribution guidelines -- Templates for PR and issues -- Code owners file for automatic assignment of PR reviewers -- [GitHub actions](https://github.com/features/actions) workflows for linting - and checking links +InterLink aims to provide an abstraction for the execution of a Kubernetes pod on any remote resource capable of managing +a Container execution lifecycle. We target to facilitate the development of provider specific plugins, so the resource +providers can leverage the power of virtual kubelet without a black belt in kubernetes internals. -Content is based on: +The project consists of two main components: -- [Contributor Covenant](http://contributor-covenant.org) -- [Semantic Versioning](https://semver.org/) -- [Chef Cookbook Contributing Guide](https://github.com/chef-cookbooks/community_cookbook_documentation/blob/master/CONTRIBUTING.MD) +- __A Kubernetes Virtual Node:__ based on the [VirtualKubelet](https://virtual-kubelet.io/) technology. +Translating request for a kubernetes pod execution into a remote call to the interLink API server. +- __The interLink API server:__ a modular and pluggable REST server where you can create your own Container manager plugin +(called sidecars), or use the existing ones: remote docker execution on a remote host, singularity Container on +a remote SLURM batch system. This repository aims to maintain the SLURM sidecar as a standalone plugin. -## GitHub repository management rules +The project got inspired by the [KNoC](https://github.com/CARV-ICS-FORTH/knoc) and +[Liqo](https://github.com/liqotech/liqo/tree/master) projects, enhancing that with the implemention a generic API +layer b/w the virtual kubelet component and the provider logic for the container lifecycle management. -All changes should go through Pull Requests. +## :electron: Usage -### Merge management +### :bangbang: Requirements -- Only squash should be enforced in the repository settings. -- Update commit message for the squashed commits as needed. +- __[Our Kubernetes Virtual Node and the interLink API server](https://github.com/interTwin-eu/interLink)__ -### Protection on main branch +- __[The Go programming language](https://go.dev/doc/install)__ (to build binaries) -To be configured on the repository settings. +- __[Docker Engine](https://docs.docker.com/engine/)__ (optional) -- Require pull request reviews before merging - - Dismiss stale pull request approvals when new commits are pushed - - Require review from Code Owners -- Require status checks to pass before merging - - GitHub actions if available - - Other checks as available and relevant - - Require branches to be up to date before merging -- Include administrators +Note: if you want a quick start setup (using a Docker container), Go is not necessary + +### $${:warning:\ It\ is\ very\ important\ for\ you\ to\ remember\ to\ \color{red}set\ CPU\ and\ Memory\ Limits\ \color{white} in\ your\ Pod/Deployment\ YAML,\ otherwise\ default\ resources\ will\ be\ applied;\ specifically,\ if\ you\ don't\ set\ a\ CPU\ limit,\ 1\ CPU\ will\ be\ used\ for\ each\ task,\ while\ if\ you\ don't\ set\ any\ Memory\ limit,\ only\ 1MB\ will\ be\ used\ for\ each\ task.\ :warning:}$$ + +### :fast_forward: Quick Start + +Just run: + +```bash +cd docker && docker compose up -d +``` + +This way, a docker container with a full SLURM environment will be started. You can find the used SLURM configuration +in /docker/SlurmConfig.yaml. If you want to update the config after you already started it once, run: + +```bash +docker compose up -d --build --force-recreate +``` + +So, the old container will be deleted, the image rebuilt and a new container with the updated config will be deployed. + +### :hammer: Building binaries + +It is of course possible to use binaries as a standalone application. Just run + +```bash +make all +``` + +and you will be able to find the built slurm-sd binary inside the bin directory. Before executing it, remember to check +if the configuration file is correctly set according to your needs. You can find an example one under examples/config/SlurmConfig.yaml. +Do not forget to set the SLURMCONFIGPATH environment variable to point to your config. + +### :gear: A SLURM config example + +```yaml +SidecarPort: "4000" +SbatchPath: "/usr/bin/sbatch" +ScancelPath: "/usr/bin/scancel" +SqueuePath: "/usr/bin/squeue" +CommandPrefix: "" +SingularityPrefix: "" +ExportPodData: true +DataRootFolder: ".local/interlink/jobs/" +Namespace: "vk" +Tsocks: false +TsocksPath: "$WORK/tsocks-1.8beta5+ds1/libtsocks.so" +TsocksLoginNode: "login01" +BashPath: /bin/bash +VerboseLogging: true +ErrorsOnlyLogging: false +``` + +### :pencil2: Annotations + +It is possible to specify Annotations when submitting Pods to the K8S cluster. A list of all Annotations follows: +| Annotation | Description| +|--------------|------------| +| slurm-job.vk.io/singularity-commands | Used to add specific Commands to be executed before the actual SLURM Job starts. It adds Commands on the Singularity exection line, in the SLURM bastch file | +| slurm-job.vk.io/pre-exec | Used to add commands to be executed before the Job starts. It adds a command in the SLURM batch file after the #SBATCH directives | +| slurm-job.vk.io/singularity-mounts | Used to add mountpoints to the Singularity Containers | +| slurm-job.vk.io/singularity-options | Used to specify Singularity arguments | +| slurm-job.vk.io/image-root | Used to specify the root path of the Singularity Image | +| slurm-job.vk.io/flags | Used to specify SLURM flags. These flags will be added to the SLURM script in the form of #SBATCH flag1, #SBATCH flag2, etc | +| slurm-job.vk.io/mpi-flags | Used to prepend "mpiexec -np $SLURM_NTASKS \*flags\*" to the Singularity Execution | + +### :gear: Explanation of the SLURM Config file + +Detailed explanation of the SLURM config file key values. Edit the config file before running the binary or before +building the docker image (`docker compose up -d --build --force-recreate` will recreate and re-run the updated image) +| Key | Value | +|--------------|-----------| +| SidecarPort | the sidecar listening port. Sidecar and Interlink will communicate on this port. Set $SIDECARPORT environment variable to specify a custom one | +| SbatchPath | path to your Slurm's sbatch binary | +| ScancelPath | path to your Slurm's scancel binary | +| CommandPrefix | here you can specify a prefix for the programmatically generated script (for the slurm plugin). Basically, if you want to run anything before the script itself, put it here. | +| ExportPodData | Set it to true if you want to export Pod's ConfigMaps and Secrets as mountpoints in your Singularity Container | +| DataRootFolder | Specify where to store the exported ConfigMaps/Secrets locally | +| Namespace | Namespace where Pods in your K8S will be registered | +| Tsocks | true or false values only. Enables or Disables the use of tsocks library to allow proxy networking. Only implemented for the Slurm sidecar at the moment. | +| TsocksPath | path to your tsocks library. | +| TsocksLoginNode | specify an existing node to ssh to. It will be your "window to the external world" | +| BashPath | Path to your Bash shell | +| VerboseLogging | Enable or disable Debug messages on logs. True or False values only | +| ErrorsOnlyLogging | Specify if you want to get errors only on logs. True or false values only | + +### :wrench: Environment Variables list + +Here's the complete list of every customizable environment variable. When specified, it overwrites the listed key +within the SLURM config file. + +| Env | Value | +|--------------|-----------| +| SLURMCONFIGPATH | your SLURM config file path. Default is `/etc/interlink/SlurmConfig.yaml` | +| SIDECARPORT | the Sidecar listening port. Docker default is 4000, Slurm default is 4001. | +| SBATCHPATH | path to your Slurm's sbatch binary. Overwrites SbatchPath. | +| SCANCELPATH | path to your Slurm's scancel binary. Overwrites ScancelPath. | +| SHARED_FS | set this env to "true" to save configmaps values inside files directly mounted to Singularity containers instead of using ENVS to create them later | +| CUSTOMKUBECONF | path to a service account kubeconfig | +| TSOCKS | true or false, to use tsocks library allowing proxy networking. Working on Slurm sidecar at the moment. Overwrites Tsocks. | +| TSOCKSPATH | path to your tsocks library. Overwrites TsocksPath. | \ No newline at end of file