Skip to content

RSP Deployment instructions on Openstack with Magnum

Jump to bottom
gpfrancis edited this page Aug 5, 2024 · 26 revisions

Deploy the RSP on Openstack

Assumptions:

  • We are running on an (admin) node with Docker installed
  • The floating IP that our configuration uses as the LoadBalancer is available (192.41.122.16 for our production service)

Phalanx configuration for latest version used for the RSP:UK can be found here:

https://github.com/lsst-uk/phalanx

Setup your clouds configuration file

user@admin-node

Placed under ${HOME:?}/clouds-iris.yaml or elsewhere (but note that the ansibler container path below should then change)

Fetch your openrc.sh file from Openstack

user@admin-node

Placed under ${HOME:?}/RSP-openrc.sh or elsewhere (but note that the ansibler container path below should then change)

Deploy a K8s cluster (Magnum/Heat method)

Run the ansibler container (Includes openstack client)

user@admin-node

clientname=ansibler-iris

sudo docker run \
    --rm \
    --tty \
    --interactive \
    --name     "${clientname:?}" \
    --hostname "${clientname:?}" \
    --volume "${HOME:?}/clouds-iris.yaml:/etc/openstack/clouds.yaml:ro,z" \
    --volume "${HOME:?}/RSP-openrc.sh:/etc/openstack/RSP-openrc.sh:ro,z" \
    ghcr.io/wfau/atolmis/ansible-client:2022.07.25 \
    bash

Run the OpenRC to load Openstack credentials

root@ansibler-iris

source /etc/openstack/RSP-openrc.sh

Clone Phalanx installer

root@ansibler-iris

git clone https://github.com/stvoutsin/phlx-installer

Optional / One-time step: Create Template

root@ansibler-iris

If you haven't already, optionally create template ..

pushd phlx-installer/scripts/openstack/

    ./create-magnum-template.sh stv-template-large

popd

The template we've setup and are using has the following attributes:

> Master Flavor ID: qserv-utility
> Volume Driver:  cinder
> Image ID: fedora-coreos-35.20211203.3.0
> Network Driver: calico
> boot_volume_size: 50

Create Cluster with Openstack Client / Magnum

Note: You may need to modify the parameters in create_cluster.sh (i.e. keypair, template name, number of worker nodes)

root@ansibler-iris

pushd phlx-installer/scripts/openstack/

    ./create-magnum-cluster.sh stv-rsp-prod-blue stv-amphora-nova

popd

Wait until the cluster has been created

Open ports 80, 443, 8443, 6379

root@ansibler-iris

pushd phlx-installer/scripts/openstack/

    ./open-ports.sh stv-rsp-prod-blue

popd

Note: We need to ensure that the IP address that we will point the Load Balancer to is open

i.e. see: https://github.com/lsst-uk/phalanx/blob/upgrade/roe-env-202304/applications/ingress-nginx/values-roe.yaml

Copy the kube config for the cluster

   openstack coe cluster config ${cluster-name}

This will create a copy under this directory, named config. Grab a copy of that

Exit the Ansibler container

exit

Deploy a K8s cluster (Magnum/CAPI method)

Install clients

user@admin-node

sudo snap install openstackclients
sudo snap install kubectl --classic

Create K8s cluster

user@admin-node

openstack coe cluster template show -f yaml kubernetes-1-29-jammy
openstack coe cluster create --cluster-template kubernetes-1-29-jammy --master-count 1 --node-count 2 --keypair ${keypair} ${cluster-name}

Copy the kube config for the cluster

user@admin-node

openstack coe cluster config ${cluster-name}

Open ports 80, 443, 8443, 6379

Not sure whether this step is required. If so the commands are:

openstack security group rule create --ingress --protocol tcp --dst-port 80:80 --remote-ip 0.0.0.0/0 ${secgroup-controlplane}
openstack security group rule create --ingress --protocol tcp --dst-port 443:443 --remote-ip 0.0.0.0/0 ${secgroup-controlplane}
openstack security group rule create --ingress --protocol tcp --dst-port 8443:8443 --remote-ip 0.0.0.0/0 ${secgroup-controlplane}
openstack security group rule create --ingress --protocol tcp --dst-port 6379:6379 --remote-ip 0.0.0.0/0 ${secgroup-controlplane}
openstack security group rule create --ingress --protocol tcp --dst-port 80:80 --remote-ip 0.0.0.0/0 ${secgroup-worker}
openstack security group rule create --ingress --protocol tcp --dst-port 443:443 --remote-ip 0.0.0.0/0 ${secgroup-worker}
openstack security group rule create --ingress --protocol tcp --dst-port 8443:8443 --remote-ip 0.0.0.0/0 ${secgroup-worker}
openstack security group rule create --ingress --protocol tcp --dst-port 6379:6479 --remote-ip 0.0.0.0/0 ${secgroup-worker}

Setup & run the phlx installer.

First make sure we have exited the ansibler-container. After clone the phlx-installer on the admin node

user@admin-node

git clone https://github.com/lsst-uk/phlx-installer

Copy the kube config from the previous step

user@admin-node

> Copy into: phlx-installer/kube/config

Build installer

user@admin-node

sudo docker build phlx-installer/ --tag installer

Create ENVIRONMENT vars for the required deployment params:

user@admin-node

export VAULT_ROLE_ID=
export VAULT_SECRET_ID=
export VAULT_ADDR=
export REPO=
export BRANCH=
export ENVIRONMENT=
export CUR_DIRECTORY=/home/ubuntu # Or whichever directory you have cloned to

Run Docker install for the RSP:

user@admin-node

 sudo docker run   \
   -it  \
   --hostname installer  \
   --env REPO=${REPO:?}  \
   --env VAULT_ADDR=${VAULT_ADDR:?}  \
   --env VAULT_SECRET_ID=${VAULT_SECRET_ID:?}  \
   --env VAULT_ROLE_ID=${VAULT_ROLE_ID:?}  \
   --env BRANCH=${BRANCH:?}  \
   --env ENVIRONMENT=${ENVIRONMENT:?}     \
   --volume ${CUR_DIRECTORY:?}"/phlx-installer/certs:/etc/kubernetes/certs"  \
   --volume ${CUR_DIRECTORY:?}"/phlx-installer/kube/config:/root/.kube/config" \
   --volume ${CUR_DIRECTORY:?}"/phlx-installer/scripts/install.sh:/root/install.sh"  \
   --volume ${CUR_DIRECTORY:?}"/phlx-installer/scripts/helper.sh:/root/helper.sh" \
   installer

Setup Amphora LoadBalancer Healthchecker

The load balancer pool can be found in the horizon dashboard, or using the openstack client

Navigate to the dynamically created load balancer (named something like: [kube_service_065a6815-cea1-42c9-b825-bee91c6b3591_ingress-nginx_ingress-nginx-controller).

Change pool TCP_443_pool, create Health Monitor:

  • Delay: 5
  • Timeout: 5
  • Max Retries: 3
  • Max Retries Down: 3
  • HTTP Method: GET
  • URL Path: /
  • Expected Codes: 400

Change pool TCP_80_pool, create Health Monitor:

  • Delay: 5
  • Timeout: 5
  • Max Retries: 3
  • Max Retries Down: 3
  • HTTP Method: GET
  • URL Path: /
  • Expected Codes: 404

Installation Completed

Clone this wiki locally