Skip to content

Commit

Permalink
Merge pull request #5105 from nuwang/update_kubernetes_2024
Browse files Browse the repository at this point in the history 
Update the intro to k8s tutorial
  • Loading branch information
@hexylena
hexylena authored Jun 30, 2024
2 parents 0c1bb89 + 27880f7 commit 6b78e76
Show file tree
Hide file tree
Showing 2 changed files with 102 additions and 151 deletions.
2 changes: 1 addition & 1 deletion topics/admin/tutorials/backup-cleanup/slides.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@

???

- One of the important admimn tasks is to keep an eye on the storage consumption.
- One of the important admin tasks is to keep an eye on the storage consumption.
- Every instance should have a data rention policy defined and shared with users.
- Codebase contains scripts that can assist with cleaning up and reclaiming space.
- The gxadmin tool can assist with invoking these scripts.
Expand Down
251 changes: 101 additions & 150 deletions topics/admin/tutorials/k8s-deploying-galaxy/tutorial.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,37 +34,27 @@ priority: 2

## Overview


Galaxy has a minimal number of required dependencies, which makes its basic
installation quick for both users and developers. However, configuring a
multi-user production instance is a complex undertaking due to Galaxy’s many
interacting and dependent systems, components, and configurations. Software
containerization has become the preferred method of addressing deployment
challenges across operating environments. Containerization also requires
orchestration, so that multiple containers can work together to deliver a
complex application. [Kubernetes](https://kubernetes.io/) has emerged as the
primary container orchestration technology, as it is both container agnostic and
widely adopted. Kubernetes allows managing, scaling, and deploying different
pieces of an application–in a standardized way–while providing excellent tooling
for doing so.

In this tutorial, we'll take a look at Kubernetes and [Helm](https://helm.sh/)
as tools for deploying containerized Galaxy. The goals for this model of
deploying Galaxy is to use best-practices from the Galaxy community on how to
deploy the Galaxy application in a well-defined package. This model can simplify
deployment and management burden of running Galaxy. While it is possible to
follow this tutorial by simply copying and pasting supplied commands, and a
production-grade Galaxy will be installed, it is desirable to have a basic
understanding of the container concepts and Kubernetes and Helm technologies.

Some of the goals for deploying and running Galaxy in this mode include:
- Design a mostly stateless model for running Galaxy where processes can be
horizontally scaled as needed
- Integrate components from the Galaxy project ecosystem to leverage existing
resources
- Provide a unified handling of Galaxy configurations
- Minimize customized dependencies
- Minimize the need to build custom components
This tutorial describes how to use the Galaxy Helm Chart to deploy a production
grade instance of Galaxy on a Kubernetes cluster. The Helm Chart has been designed
to follow best practices adopted by the community, including the usegalaxy.* federation,
and will install a Galaxy with the following features by default:
- Zero-downtime configuration changes and upgrades
- Scalable web and job handlers
- Automatic failure recovery based on liveness and readiness probes
- A built-in nginx for efficiently serving large files
- TUSD for resumable uploads
- Celery for background jobs
- Access to CVMFS reference data
- A toolset matching the usegalaxy.* federation (also served off CVMFS)
- Interactive tools (wildcard DNS mapping required)
- Minimal privileges, with jobs running as non-root and only having access to datasets they need
- Automatic maintenance scripts to cleanup the galaxy database and tmp directories

Optionally, the chart can be configured with
- High-availability components - this includes trivial scaling of clustered Postgres, Rabbit MQ etc.
- Replacement components - You can replace the built-in operators with a managed or existing Postgres database (e.g. Amazon RDS), RabbitMQ cluster etc.
- Use S3 as an alternative to CVMFS
- Automatic scraping of metrics which can be sent to Influxdb

> <agenda-title></agenda-title>
>
Expand All @@ -74,11 +64,13 @@ Some of the goals for deploying and running Galaxy in this mode include:
{: .agenda}

## Prerequisites
We'll be using the [Galaxy Helm chart] to install and manage a Galaxy
deployment. To be able to use this chart, we'll need access to a Kubernetes
cluster, with Helm installed. For development and testing purposes this can be
easily achieved by installing
[Docker Desktop locally and enabling Kubernetes][DfD]. Afterwards, also install
Some familarity with Kubernetes is assumed. This includes general administrative
familarity and how to install and configure Helm Charts.

A running Kubernetes cluster is also required (1.27 or higher), with Helm
(3.5 or higher) configured to access it.
For development and testing purposes this can be easily achieved by installing
[Docker Desktop locally and enabling Kubernetes][DfD]. Afterwards, install
[Helm](https://helm.sh).

For production deployments, we'll also need some storage resources for data
Expand All @@ -87,41 +79,6 @@ creating a [Persistent Volume and a corresponding Persistent Volume Claim][PV].
Once created, just keep a note of the resources Persistent Volume Claim ID and
to use later.

For the CVMFS-enabled version of the chart (more on this below), it is also
necessary to run Kubernetes version 1.13 or newer because we'll be using the
[Container Storage Interface (CSI)][CSI].

# Downloading the Galaxy Helm Chart
The Galaxy Helm Chart is currently under active development with enhancements
continuously trickling in. As a result, there are no regular releases yet and
instead we recommend just cloning the GitHub repository with the chart
implementation. This will be the easiest method to keep up with chart changes
for the time being.

> <hands-on-title>Download the chart</hands-on-title>
>
> Clone the chart repository from the machine where you would like to deploy
> Galaxy and change into the chart directory.
>
> {% raw %}
> ```bash
> git clone https://github.com/galaxyproject/galaxy-helm
> cd galaxy-helm/galaxy
> ```
> {% endraw %}
{: .hands_on}
# Deploying Galaxy
The Galaxy Helm chart packages best-practice solutions for deploying Galaxy
into a single package that can be readily deployed as a unit. Behind the
scenes, all the supporting services are started and configured into an
interoperable system. Specifically, this involves starting a database service
based on Postgres, using Nginx as a web proxy, and running an independently
scalable set of web and job handler processes for Galaxy. This follows
the production-quality deployment recommendation setup for Galaxy and leverages
some of the Kubernetes features to help with running long-term services (e.g.,
liveness probes that automatically restart stuck processes).
## Deploying the Default Configuration
The default set of values for the Galaxy chart configures only a minimal set
of Galaxy options necessary. The configured options are required for suitable
Expand All @@ -132,129 +89,123 @@ chart later in this tutorial.

> <hands-on-title>Deploying the Galaxy Helm Chart</hands-on-title>
>
> 1. First, we need to fetch any dependencies for the chart. One of the
> advantages of using Helm is that we can reuse best-practice deployment
> methods for other software right out of the box by relying on published
> charts and integrating them into the Galaxy chart.
> 1. First, we need to add the helm repository for the chart. The chart is
> automatically packaged, versioned and uploaded to a helm repository on github
> with each accepted PR. Therefore, the latest version of the chart can be directly
> installed from that repository.
>
> {% raw %}
> ```bash
> helm dependency update
> helm repo add galaxyproject https://raw.githubusercontent.com/galaxyproject/helm-charts/master/
> helm repo update
> ```
> {% endraw %}
>
> 2. We can now deploy Galaxy via the Chart. Before running this command make
> sure you are in the chart source code directory (where `values.yaml` file
> resides) and note the trailing dot. Running this command will create a new
> Helm release (i.e., chart installation) called `galaxy`.
> 2. We can now deploy Galaxy via the Chart. Running this command will create a new
> Helm release (i.e., chart installation) called `mygalaxy`.
>
> {% raw %}
> ```bash
> helm install --name galaxy .
> helm install mygalaxy galaxyproject/galaxy
> ```
> {% endraw %}
>
> 3. It will take about a minute or two for the database to be initialized,
> necessary containers downloaded, and Galaxy processes started. Ultimately,
> 3. It will take about a minute or two for the necessary containers to download,
> the database to initialize, and Galaxy processes to start. Ultimately,
> while this may depend on the Kubernetes cluster setup you are using,
> Galaxy should be available at the root URI for the given machine. We can
> Galaxy should be available at https://<root URI>/galaxy for the given machine. We can
> always check the status of our release by typing `helm status galaxy`.
>
{: .hands_on}
## Deploying a CVMFS-enabled Configuration
The Galaxy Helm chart also comes with a more comprehensive set of configuration
options that leverage more of the Galaxy project ecosystem. In practice this
means deploying Galaxy with the same toolset as that of
_[usegalaxy.org](https://usegalaxy.org/)_ right out of the box. It's important to note
that this deployment configuration leverages all the same chart components but
just defines more configuration options. Namely, we attach to the
[Galaxy CVMFS][CVMFS] ready-only file system for retrieving the tool
configurations while leveraging [BioContainers] for resolving tool dependencies.
## Setting the admin user and changing the brand
The chart is designed to follow standard Kubernetes and Helm idioms, and therefore,
it should be intuitively similar to the steps required to change configuration in
any other Helm chart. For example, ingress paths, resource allocations, container
images etc. can be changed following standard helm conventions. The list of
available configuration options are also documented in the Galaxy Helm
_[Chart repository](https://github.com/galaxyproject/galaxy-helm/tree/master?tab=readme-ov-file#configuration)_
> <hands-on-title>Deploying the CVMFS-enabled Configuration</hands-on-title>
To change Galaxy specific configuration, such as setting the admin user or change the brand in `galaxy.yml`,
we can follow the following steps. Once done, we will also rollback our change to demonstrate how Helm manages
configuration.
> <hands-on-title>Setting admin user and changing the brand</hands-on-title>
>
> 1. Modify the following entries in your `mygalaxy.yml`. Make sure to add these
> keys under the `configs:` section of the file.
>
> 1. If you are following this tutorial sequentially and have a release of
> Galaxy already running, let's delete it (assuming that's fine and you have
> no data to keep). More details about the deletion process are available in
> the [Deleting a Deployment section](#deleting-a-deployed-helm-release). If
> you're just playing around, run `helm delete --purge galaxy`.
> {% raw %}
> ```
> configs:
> galaxy.yml:
> galaxy:
> brand: "Hello World"
> admin_users: "[email protected]"
> ```
> {% endraw %}
>
> 2. The CVMFS variant of the Galaxy chart has an additional dependency on the
> [Galaxy CVMFS chart](https://github.com/CloudVE/galaxy-cvmfs-csi-chart).
> We'll deploy this chart into its own [Namespace] to keep its resources
> nicely grouped. We'll also fetch the chart from a packaged chart
> repository instead of its GitHub repo.
> 2. Now, let’s upgrade the chart to apply the new configuration.
>
> {% raw %}
> ```bash
> kubectl create namespace cvmfs
> helm repo add galaxy https://raw.githubusercontent.com/CloudVE/helm-charts/master/
> helm repo update
> helm install --name cvmfs --namespace cvmfs galaxy/galaxy-cvmfs-csi
> helm upgrade --reuse-values -f mygalaxy.yml mygalaxy galaxyproject/galaxy
> ```
> {% endraw %}
>
> 3. We can now install the CVMFS-enabled set of values.
> 3. Inspect the currently set Helm values by:
>
> {% raw %}
> ```bash
> helm install --name galaxy galaxy/galaxy
> helm get values mygalaxy
> ```
> {% endraw %}
>
> 4. Again, it will take a few minutes for Galaxy to start up. This time most of
> the waiting is due to the tool definition files to be cached on CVMFS and
> loaded into the tool panel. We can check the status of the deployment by
> running `helm status galaxy`. We can also watch the boot process by tailing
> the logs of the relevant container with a command similar to
> `kubectl logs -f galaxy-web-7568c58b94-hjl9w` where the last argument is
> the name of the desired pod, as printed following the `helm install`
> command. Once the boot process has completed, we can access Galaxy at
> `/galaxy/` URI (note the trailing `/`; it's significant).
> 4. List the installed Helm charts again and note that the revision of the chart has changed as expected.
>
{: .hands_on}
## Deleting a Deployed Helm Release
After we're done experimenting with an installation of the chart, we can just
as easily delete all the resources as we've created them. However, that may
not be desirable so make sure you understand the system you're working on to
avoid undesired surprises. Namely, deleting and recreating a Helm release is
generally not a problem where the processes will just respawn and everything will go back to operational; however, underlying storage configuration may
interfere here with all the application data being potentially lost. This
predominantly depends on how the relevant storage class was configured.
> <hands-on-title>Deleting a Deployed Helm Release</hands-on-title>
> {% raw %}
> ```bash
> helm list
> NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
> mygalaxy 2 Wed Jun 26 14:51:17 2023 DEPLOYED galaxy-5.14.2 v24.0.2 default
> ```
> {% endraw %}
>
> 5. Revisit the Galaxy Application in your browser to check whether the settings have changed. This will
> take a short while (< 1 minute) for the new container to come up. You should experience no downtime.
>
> 1. Before we delete a deployment, let's ensure we understand what will happen
> with the underlying storage used by Galaxy.
> 6. Let’s now roll back to the previous revision.
>
> {% raw %}
> ```bash
> $ kubectl get pv
> NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
> cvmfs-cache-pv 1000Mi RWX Retain Bound cvmfs/cvmfs-cache-pvc manual 31m
> pvc-55806281-96c6-11e9-8e96-0251cc6c62f4 1Gi ROX Delete Bound default/galaxy-cvmfs-gxy-data-pvc cvmfs-gxy-data 28m
> pvc-5580c830-96c6-11e9-8e96-0251cc6c62f4 1Gi ROX Delete Bound default/galaxy-cvmfs-gxy-main-pvc cvmfs-gxy-main 28m
> pvc-55814757-96c6-11e9-8e96-0251cc6c62f4 10Gi RWX Delete Bound default/galaxy-galaxy-pvc nfs-provisioner 28m
> pvc-70d4cc48-96be-11e9-8e96-0251cc6c62f4 8Gi RWO Delete Bound default/data-galaxy-galaxy-postgres-0 nfs-provisioner 84m
> pvc-8cb27bc9-9679-11e9-8e96-0251cc6c62f4 100Gi RWO Delete Bound cloudman/data-nfs-provisioner-0 ebs-provisioner 9h
> helm rollback mygalaxy 1
> ```
> {% endraw %}
>
> As we can see in the command output, the storage resources associated with
> the current deployment have the reclaim policy set to `Delete`, which will
> happen once no resources are using the given resource. If what you see is
> the not the intended behavior, you can change the [reclaim policy].
>
> 2. Once we're ok with the state of the resources and are ready to delete a
> a deployment, we can do so with the following commands:
> Use `helm get values` again to observe that the values have reverted to
> the previous revision. After a short while, once the new container is up
> and running, Kubernetes will automatically switch over to it and you can
> see that the previous configuration has been restored.
>
{: .hands_on}
## Deleting a Deployed Helm Release
By default, the Helm chart is designed to install all required dependencies, so that it's easy
to get an instance up and running quickly for experimentation. However, in production, we
recommend installing the dependency charts separately, once per cluster, by installing
Galaxy with helm options
`--set postgresql.deploy=false --set s3csi.deploy=false --set cvmfs.deploy=false --set rabbitmq.deploy=false`.
This is particularly important during uninstallation, where orderly destruction of dependencies is often required
For example, if the rabbitmq operator is uninstalled before the rest of the Galaxy helm chart is deleted, there will be
no operator left to cleanup rabbitmq resources. Installing the aforementioned operators separately sidesteps this problem.
> <hands-on-title>Deleting a Deployed Helm Release</hands-on-title>
>
> {% raw %}
> ```bash
> helm delete --purge galaxy
> helm delete --purge cvmfs
> helm delete mygalaxy
> helm delete mycvmfs # and any other operators
> ```
> {% endraw %}
>
Expand Down

0 comments on commit 6b78e76

Please sign in to comment.