Multi-cell adoption

Change the initially planned renaming scheme for cells,
where the OSP's 'default' cell becomes the last cell:

* Default becomes cell1
* Former cell1 becomes cell2, and so on.

That is because cells naming do not matter, and is internal
Nova detail. However, this changed renaming scheme simplifies
iterating cells in loops and define misc mapping for the source
and destination names for cells, databases instances and RHOSO
cells indexes in Nova/OSCP CR definitions.

Make ansible and shell variables compute cells aware.
Split edpm nodes into compute cells by 1:1 mapping it as
dataplane nodesets.

Separate vars and secrets YAML values for the source and edpm
nodes to not confuse its different naming schemes for cells
in OSP/TripleO and RHOSO. This also allows adoption of the source
tripleo nodes into different EDPM nodes with another IPs but the
same FQDNs (those must remain unchanged as a strict requirement).

Remove edpm_computes vars no longer used after moving stopping
control-plane tripleo services into edpm-ansible

Remove cached fact for pulled OSP configuration as it can no longer
be generated in a multi-cell setup, where related shell variables
become bash arrays.

Signed-off-by: Bohdan Dobrelia <[email protected]>

docs_dev/assemblies/development_environment.adoc

@@ -187,6 +187,87 @@ https://openstack-k8s-operators.github.io/data-plane-adoption/dev/#_reset_the_en
 
 '''
 
+== Deploying TripleO With Multiple Cells
+
+A TripleO Standalone setup creates only a single Nova v2 cell, with a combined controller and compute services on it.
+In order to deploy multiple compute cells for adoption testing (without Ceph), create a 5 VMs, with the following requirements met:
+
+* Named `edpm-compute-0` .. `edpm-compute-4`.
+* Running RHEL 9.2, with RHOSP 17.1 repositiries configured.
+* Can login via SSH without a password as the root user, from the hypervisor host.
+* User `zuul` is created, and can sudo without a password, and login via SSH without a password, from the hypervisor host.
+* User `zuul` can login to `edpm-compute-1`, `edpm-compute-2`, `edpm-compute-3`, `edpm-compute-4` nodes via SSH without a password, from the `edpm-compute-0` node,
+by using the generated `/home/zuul/.ssh/id_rsa` private key.
+* RedHat registry credentials are exported on the hypervisor host.
+
+Adjust the following commands for a repositories configuration tool of your choice:
+
+[,bash]
+----
+ export RH_REGISTRY_USER="<insert your registry.redhat.io user>"
+ export RH_REGISTRY_PWD="<insert your registry.redhat.io password>"
+
+cd ~/install_yamls/devsetup
+cat <<EOF > /tmp/osp17_repos
+# Use a tool of your choice:
+# 1. Rhos-release example steps are only available from the internal RedHat network
+# ... skipping download and install steps ...
+# sudo rhos-release -x
+# sudo rhos-release 17.1
+
+# 2. Subscription-manager example steps require an active registration
+# sudo subscription-manager repos \
+#   --enable=rhel-9-for-x86_64-baseos-eus-rpms \
+#   --enable=rhel-9-for-x86_64-appstream-eus-rpms \
+#   --enable=rhel-9-for-x86_64-highavailability-eus-rpms \
+#   --enable=openstack-17.1-for-rhel-9-x86_64-rpms \
+#   --enable=rhceph-6-tools-for-rhel-9-x86_64-rpms \
+#   --enable=fast-datapath-for-rhel-9-x86_64-rpms
+
+sudo dnf install -y git curl wget util-linux lvm2 podman python3-tripleoclient
+EOF
+
+export CENTOS_9_STREAM_URL=<insert url to rhel-guest-image-9.2.x86_64.qcow2>
+export NTP_SERVER=<insert ntp server of your choice>
+
+export MANILA_ENABLED=false
+export EDPM_COMPUTE_CEPH_ENABLED=false
+export EDPM_COMPUTE_CEPH_NOVA=false
+export EDPM_COMPUTE_CELLS=3
+
+export STANDALONE_EXTRA_CMD="bash -c 'echo \"$RH_REGISTRY_PWD\" > ~/authfile; chmod 0600 ~/authfile; sudo /bin/podman login registry.redhat.io -u \"$RH_REGISTRY_USER\" --password-stdin < ~/authfile'"
+export REPO_SETUP_CMDS=/tmp/osp17_repos
+export EDPM_TOTAL_NODES=1
+export SKIP_TRIPLEO_REPOS=false
+export IP_ADRESS_SUFFIX=100
+export EDPM_COMPUTE_NETWORK_IP=192.168.122.1
+export HOST_PRIMARY_RESOLV_CONF_ENTRY=192.168.122.1
+export BASE_DISK_FILENAME="rhel-9-base.qcow2"
+
+make edpm_compute EDPM_COMPUTE_SUFFIX=0 EDPM_COMPUTE_DISK_SIZE=10 EDPM_COMPUTE_RAM=9 EDPM_COMPUTE_VCPUS=2
+make edpm_compute EDPM_COMPUTE_SUFFIX=1 EDPM_COMPUTE_DISK_SIZE=17 EDPM_COMPUTE_RAM=12 EDPM_COMPUTE_VCPUS=4
+make edpm_compute EDPM_COMPUTE_SUFFIX=2 EDPM_COMPUTE_DISK_SIZE=14 EDPM_COMPUTE_RAM=12 EDPM_COMPUTE_VCPUS=4
+make edpm_compute EDPM_COMPUTE_SUFFIX=3 EDPM_COMPUTE_DISK_SIZE=10 EDPM_COMPUTE_RAM=4 EDPM_COMPUTE_VCPUS=2
+make edpm_compute EDPM_COMPUTE_SUFFIX=4 EDPM_COMPUTE_DISK_SIZE=16 EDPM_COMPUTE_RAM=12 EDPM_COMPUTE_VCPUS=4
+
+for n in 0 1 2 3 4; do
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} useradd --create-home --shell /bin/bash --groups root zuul
+    scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]${n}:/home/zuul/.ssh/id_rsa
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} cp /root/.ssh/authorized_keys /home/zuul/.ssh/authorized_keys
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} chown zuul: /home/zuul/.ssh/*
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} echo "zuul ALL=NOPASSWD:ALL" '>' /etc/sudoers.d/zuul; done
+done
+
+make tripleo_deploy
+
+for n in 0 1 2 3 4; do make standalone_snapshot EDPM_COMPUTE_SUFFIX=$n; done
+----
+
 == Network routing
 
 Route VLAN20 to have access to the MariaDB cluster:
@@ -340,14 +421,26 @@ make openstack
 
 == Performing the adoption procedure
 
-To simplify the adoption procedure, copy the deployment passwords that
+To simplify the adoption procedure with additional cells, copy and rename the deployment passwords that
 you use in copy the deployment passwords that you use in the
 https://openstack-k8s-operators.github.io/data-plane-adoption/user/#deploying-backend-services_migrating-databases[backend
 services deployment phase of the data plane adoption].
 
+For a single-cell standalone TripleO deployment:
+[,bash]
+----
+mkdir -p ~/overcloud-deploy/overcloud
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:/root/tripleo-standalone-passwords.yaml ~/overcloud-deploy/overcloud/overcloud-passwords.yaml
+----
+
+Further on, this password is going to be referenced as `TRIPLEO_PASSWORDS[default]` for a `default` cell name in terms of TripleO (which becomes a `cell1` after adoption).
+
+For a multi-cell deployment, change the above command to these:
 [,bash]
 ----
-scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:/root/tripleo-standalone-passwords.yaml ~/
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:overcloud-deploy/overcloud/overcloud-passwords.yaml ~/
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:overcloud-deploy/cell1/cell1-passwords.yaml ~/
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:overcloud-deploy/cell2/cell2-passwords.yaml ~/
 ----
 
 The development environment is now set up, you can go to the https://openstack-k8s-operators.github.io/data-plane-adoption/[Adoption
@@ -386,14 +479,21 @@ oc delete --wait=false pod mariadb-copy-data || true
 oc delete secret osp-secret || true
 ----
 
-Revert the standalone vm to the snapshotted state
+Revert the standalone vm(s) to the snapshotted state
 
 [,bash]
 ----
 cd ~/install_yamls/devsetup
 make standalone_revert
 ----
 
+For a multi-cell deployment, change the above command to these:
+[,bash]
+----
+cd ~/install_yamls/devsetup
+for n in 0 1 2 3 4; do make standalone_revert EDPM_COMPUTE_SUFFIX=$n; done
+----
+
 Clean up and initialize the storage PVs in CRC vm
 
 [,bash]

docs_dev/assemblies/tests.adoc

@@ -25,9 +25,15 @@ work out of the box. The comments in the YAML files will guide you
 regarding the expected values. You may want to double check that
 these variables suit your environment:
  ** `install_yamls_path`
- ** `tripleo_passwords`
- ** `controller*_ssh`
+ ** `controller*_ssh`         (for each controller in each Heat stack on the source cloud)
+ ** `tripleo_cells_passwords` (for each Heat stack on the source cloud)
+ ** `source_mariadb_ip`       (for each cell controller on the source cloud)
+ ** `source_galera_members``  (for each cell controller on the source cloud)
+ ** `source_node_hostname``   (for each cell controller on the source cloud)
+ ** `edpm_node_hostname`      (for each cell compute node on the destination)
+ ** `edpm_node_ip`            (for each cell compute node on the destination)
  ** `edpm_privatekey_path`
+ ** `source_ovndb_ip``
  ** `timesync_ntp_servers`
 
 == Running the tests

docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc

@@ -229,18 +229,25 @@ EOF
 ----
 endif::[]
 
-. Deploy the `OpenStackDataPlaneNodeSet` CR:
+. Deploy the `OpenStackDataPlaneNodeSet` CRs (for each Nova compute cell):
 +
 . If TLS Everywhere is enabled, change spec:tlsEnabled to true
 . If using a custom DNS Domain, modify the spec:nodes:[NODE NAME]:hostName to use fqdn for the node
+. Use node set names, like `openstack-cell1`, `openstack-cell2`
+. Assign all compute nodes from the source cloud `default` cell into `openstack-cell1`
+. Assign all compute nodes from the source cloud `cell1` into `openstack-cell2`, and so on.
 +
+Note that after control plane adoption, the cell controllers should be decomissioned.
+They can become cell computes by following the scale-out, or scale-up, procedures
+for dataplane node sets and its members. However, that is not covered in this guide.
+
 [source,yaml]
 ----
 oc apply -f - <<EOF
 apiVersion: dataplane.openstack.org/v1beta1
 kind: OpenStackDataPlaneNodeSet
 metadata:
-  name: openstack
+  name: openstack-cell1
 spec:
   tlsEnabled: false
   networkAttachments:

docs_user/modules/proc_adopting-the-compute-service.adoc

@@ -144,18 +144,27 @@ $ openstack server list
 
 Compare the following outputs with the topology specific configuration in xref:proc_retrieving-topology-specific-service-configuration_migrating-databases[Retrieving topology-specific service configuration].
 
-* Query the superconductor for cell1 existance and compare it to pre-adoption values:
+* Query the superconductor for renamed cells existance, and compare it to pre-adoption values:
 +
 ----
 . ~/.source_cloud_exported_variables
-echo $PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS
-oc rsh nova-cell0-conductor-0 nova-manage cell_v2 list_cells | grep -F '| cell1 |'
+for CELL in $CELLS; do
+  if [ "$CELL" = "default" ] ; then
+    RCELL=cell1
+  else
+    RCELL="cell$(( ${CELL##*cell} + 1 ))"
+  fi
+
+  echo $PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS[$CELL]
+  oc rsh nova-${RCELL}-conductor-0 nova-manage cell_v2 list_cells | grep -F '| $RCELL |'
+done
 ----
 +
 The expected changes to happen:
 
- ** cell1's `nova` DB and user name become `nova_cell1`.
- ** Default cell is renamed to `cell1` (in a multi-cell setup, it should become indexed as the last cell instead).
+ ** The `cell1` cell's `nova` DB and user name should change to `nova_cell1`.
+ ** The `default` cell should be renamed to `cell1`.
+ ** The `cell1` cell should become `cell2`, and so on.
  ** RabbitMQ transport URL no longer uses `guest`.
 
 [NOTE]

docs_user/modules/proc_deploying-backend-services.adoc

@@ -44,7 +44,9 @@ ADMIN_PASSWORD=SomePassword
 To use the existing {OpenStackShort} deployment password:
 +
 ----
-ADMIN_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' AdminPassword:' | awk -F ': ' '{ print $2; }')
+declare -A TRIPLEO_PASSWORDS
+TRIPLEO_PASSWORDS[default]='~/overcloud-deploy/overcloud/overcloud-passwords.yaml'
+ADMIN_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' AdminPassword:' | awk -F ': ' '{ print $2; }')
 ----
 * Set service password variables to match the original deployment.
 Database passwords can differ in the control plane environment, but
@@ -54,21 +56,21 @@ For example, in developer environments with {OpenStackPreviousInstaller} Standal
 passwords can be extracted like this:
 +
 ----
-AODH_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' AodhPassword:' | awk -F ': ' '{ print $2; }')
-BARBICAN_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' BarbicanPassword:' | awk -F ': ' '{ print $2; }')
-CEILOMETER_METERING_SECRET=$(cat ~/tripleo-standalone-passwords.yaml | grep ' CeilometerMeteringSecret:' | awk -F ': ' '{ print $2; }')
-CEILOMETER_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' CeilometerPassword:' | awk -F ': ' '{ print $2; }')
-CINDER_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' CinderPassword:' | awk -F ': ' '{ print $2; }')
-GLANCE_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' GlancePassword:' | awk -F ': ' '{ print $2; }')
-HEAT_AUTH_ENCRYPTION_KEY=$(cat ~/tripleo-standalone-passwords.yaml | grep ' HeatAuthEncryptionKey:' | awk -F ': ' '{ print $2; }')
-HEAT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' HeatPassword:' | awk -F ': ' '{ print $2; }')
-IRONIC_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' IronicPassword:' | awk -F ': ' '{ print $2; }')
-MANILA_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' ManilaPassword:' | awk -F ': ' '{ print $2; }')
-NEUTRON_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' NeutronPassword:' | awk -F ': ' '{ print $2; }')
-NOVA_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' NovaPassword:' | awk -F ': ' '{ print $2; }')
-OCTAVIA_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' OctaviaPassword:' | awk -F ': ' '{ print $2; }')
-PLACEMENT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' PlacementPassword:' | awk -F ': ' '{ print $2; }')
-SWIFT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' SwiftPassword:' | awk -F ': ' '{ print $2; }')
+AODH_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' AodhPassword:' | awk -F ': ' '{ print $2; }')
+BARBICAN_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' BarbicanPassword:' | awk -F ': ' '{ print $2; }')
+CEILOMETER_METERING_SECRET=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' CeilometerMeteringSecret:' | awk -F ': ' '{ print $2; }')
+CEILOMETER_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' CeilometerPassword:' | awk -F ': ' '{ print $2; }')
+CINDER_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' CinderPassword:' | awk -F ': ' '{ print $2; }')
+GLANCE_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' GlancePassword:' | awk -F ': ' '{ print $2; }')
+HEAT_AUTH_ENCRYPTION_KEY=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' HeatAuthEncryptionKey:' | awk -F ': ' '{ print $2; }')
+HEAT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' HeatPassword:' | awk -F ': ' '{ print $2; }')
+IRONIC_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' IronicPassword:' | awk -F ': ' '{ print $2; }')
+MANILA_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' ManilaPassword:' | awk -F ': ' '{ print $2; }')
+NEUTRON_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' NeutronPassword:' | awk -F ': ' '{ print $2; }')
+NOVA_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' NovaPassword:' | awk -F ': ' '{ print $2; }')
+OCTAVIA_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' OctaviaPassword:' | awk -F ': ' '{ print $2; }')
+PLACEMENT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' PlacementPassword:' | awk -F ': ' '{ print $2; }')
+SWIFT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' SwiftPassword:' | awk -F ': ' '{ print $2; }')
 ----
 
 .Procedure