Multi-cell adoption

Keep renaming 'default' cell consistent for single and multi cells:

* Default becomes cellX (or it can be imported as is, for a multi-cell
  case only)
* cell1 becomes mapped to openstack-cell1 osdp node set
* cell2 becomes mapped to openstack-cell2 osdp node set, etc.
* cellX (X=3 here) becomes mapped to openstack-cell3. Alternatively,
  default cell retains its name for the openstack-default osdpns
  mapping

Make ansible and shell variables compute cells aware.
Split edpm nodes into compute cells by 1:1 mapping it as
dataplane nodesets.

Separate vars and secrets YAML values for the source and edpm
nodes to not confuse its different naming schemes for cells
in OSP/TripleO and RHOSO. This also allows adoption of the source
tripleo nodes into different EDPM nodes with another IPs but the
same FQDNs (those must remain unchanged as a strict requirement).

Remove edpm_computes vars no longer used after moving stopping
control-plane tripleo services into edpm-ansible

Remove cached fact for pulled OSP configuration as it can no longer
be generated in a multi-cell setup, where related shell variables
become bash arrays.

Signed-off-by: Bohdan Dobrelia <[email protected]>

docs_dev/assemblies/development_environment.adoc

@@ -187,6 +187,89 @@ https://openstack-k8s-operators.github.io/data-plane-adoption/dev/#_reset_the_en
 
 '''
 
+== Deploying TripleO With Multiple Cells
+
+A TripleO Standalone setup creates only a single Nova v2 cell, with a combined controller and compute services on it.
+In order to deploy multiple compute cells for adoption testing (without Ceph), create a 5 VMs, with the following requirements met:
+
+* Named `edpm-compute-0` .. `edpm-compute-4`.
+* Running RHEL 9.2, with RHOSP 17.1 repositiries configured.
+* Can login via SSH without a password as the root user, from the hypervisor host.
+* User `zuul` is created, and can sudo without a password, and login via SSH without a password, from the hypervisor host.
+* User `zuul` can login to `edpm-compute-1`, `edpm-compute-2`, `edpm-compute-3`, `edpm-compute-4` nodes via SSH without a password, from the `edpm-compute-0` node,
+by using the generated `/home/zuul/.ssh/id_rsa` private key.
+* RedHat registry credentials are exported on the hypervisor host.
+
+Adjust the following commands for a repositories configuration tool of your choice:
+
+[,bash]
+----
+ export RH_REGISTRY_USER="<insert your registry.redhat.io user>"
+ export RH_REGISTRY_PWD="<insert your registry.redhat.io password>"
+
+cd ~/install_yamls/devsetup
+cat <<EOF > /tmp/osp17_repos
+# Use a tool of your choice:
+# 1. Rhos-release example steps are only available from the internal RedHat network
+# ... skipping download and install steps ...
+# sudo rhos-release -x
+# sudo rhos-release 17.1
+
+# 2. Subscription-manager example steps require an active registration
+# subscription-manager release --set=9.2
+# subscription-manager repos --disable=*
+# sudo subscription-manager repos \
+#   --enable=rhel-9-for-x86_64-baseos-eus-rpms \
+#   --enable=rhel-9-for-x86_64-appstream-eus-rpms \
+#   --enable=rhel-9-for-x86_64-highavailability-eus-rpms \
+#   --enable=openstack-17.1-for-rhel-9-x86_64-rpms \
+#   --enable=rhceph-6-tools-for-rhel-9-x86_64-rpms \
+#   --enable=fast-datapath-for-rhel-9-x86_64-rpms
+
+sudo dnf install -y git curl wget util-linux lvm2 podman python3-tripleoclient
+EOF
+
+export CENTOS_9_STREAM_URL=<insert url to rhel-guest-image-9.2.x86_64.qcow2>
+export NTP_SERVER=<insert ntp server of your choice>
+
+export MANILA_ENABLED=false
+export EDPM_COMPUTE_CEPH_ENABLED=false
+export EDPM_COMPUTE_CEPH_NOVA=false
+export EDPM_COMPUTE_CELLS=3
+
+export STANDALONE_EXTRA_CMD="bash -c 'echo \"$RH_REGISTRY_PWD\" > ~/authfile; chmod 0600 ~/authfile; sudo /bin/podman login registry.redhat.io -u \"$RH_REGISTRY_USER\" --password-stdin < ~/authfile'"
+export REPO_SETUP_CMDS=/tmp/osp17_repos
+export EDPM_TOTAL_NODES=1
+export SKIP_TRIPLEO_REPOS=false
+export IP_ADRESS_SUFFIX=100
+export EDPM_COMPUTE_NETWORK_IP=192.168.122.1
+export HOST_PRIMARY_RESOLV_CONF_ENTRY=192.168.122.1
+export BASE_DISK_FILENAME="rhel-9-base.qcow2"
+
+make edpm_compute EDPM_COMPUTE_SUFFIX=0 EDPM_COMPUTE_DISK_SIZE=10 EDPM_COMPUTE_RAM=9 EDPM_COMPUTE_VCPUS=2
+make edpm_compute EDPM_COMPUTE_SUFFIX=1 EDPM_COMPUTE_DISK_SIZE=17 EDPM_COMPUTE_RAM=12 EDPM_COMPUTE_VCPUS=4
+make edpm_compute EDPM_COMPUTE_SUFFIX=2 EDPM_COMPUTE_DISK_SIZE=14 EDPM_COMPUTE_RAM=12 EDPM_COMPUTE_VCPUS=4
+make edpm_compute EDPM_COMPUTE_SUFFIX=3 EDPM_COMPUTE_DISK_SIZE=10 EDPM_COMPUTE_RAM=4 EDPM_COMPUTE_VCPUS=2
+make edpm_compute EDPM_COMPUTE_SUFFIX=4 EDPM_COMPUTE_DISK_SIZE=16 EDPM_COMPUTE_RAM=12 EDPM_COMPUTE_VCPUS=4
+
+for n in 0 1 2 3 4; do
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} useradd --create-home --shell /bin/bash --groups root zuul
+    scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]${n}:/home/zuul/.ssh/id_rsa
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} cp /root/.ssh/authorized_keys /home/zuul/.ssh/authorized_keys
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} chown zuul: /home/zuul/.ssh/*
+    ssh -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa \
+        [email protected]${n} echo "zuul ALL=NOPASSWD:ALL" '>' /etc/sudoers.d/zuul; done
+done
+
+make tripleo_deploy
+
+for n in 0 1 2 3 4; do make standalone_snapshot EDPM_COMPUTE_SUFFIX=$n; done
+----
+
 == Network routing
 
 Route VLAN20 to have access to the MariaDB cluster:
@@ -340,14 +423,26 @@ make openstack
 
 == Performing the adoption procedure
 
-To simplify the adoption procedure, copy the deployment passwords that
+To simplify the adoption procedure with additional cells, copy and rename the deployment passwords that
 you use in copy the deployment passwords that you use in the
 https://openstack-k8s-operators.github.io/data-plane-adoption/user/#deploying-backend-services_migrating-databases[backend
 services deployment phase of the data plane adoption].
 
+For a single-cell standalone TripleO deployment:
+[,bash]
+----
+mkdir -p ~/overcloud-deploy/overcloud
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:/root/tripleo-standalone-passwords.yaml ~/overcloud-deploy/overcloud/overcloud-passwords.yaml
+----
+
+Further on, this password is going to be referenced as `TRIPLEO_PASSWORDS[default]` for a `default` cell name, in terms of TripleO.
+
+For a multi-cell deployment, change the above command to these:
 [,bash]
 ----
-scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:/root/tripleo-standalone-passwords.yaml ~/
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:overcloud-deploy/overcloud/overcloud-passwords.yaml ~/
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:overcloud-deploy/cell1/cell1-passwords.yaml ~/
+scp -i ~/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected]:overcloud-deploy/cell2/cell2-passwords.yaml ~/
 ----
 
 The development environment is now set up, you can go to the https://openstack-k8s-operators.github.io/data-plane-adoption/[Adoption
@@ -386,14 +481,21 @@ oc delete --wait=false pod mariadb-copy-data || true
 oc delete secret osp-secret || true
 ----
 
-Revert the standalone vm to the snapshotted state
+Revert the standalone vm(s) to the snapshotted state
 
 [,bash]
 ----
 cd ~/install_yamls/devsetup
 make standalone_revert
 ----
 
+For a multi-cell deployment, change the above command to these:
+[,bash]
+----
+cd ~/install_yamls/devsetup
+for n in 0 1 2 3 4; do make standalone_revert EDPM_COMPUTE_SUFFIX=$n; done
+----
+
 Clean up and initialize the storage PVs in CRC vm
 
 [,bash]

docs_dev/assemblies/tests.adoc

@@ -25,9 +25,15 @@ work out of the box. The comments in the YAML files will guide you
 regarding the expected values. You may want to double check that
 these variables suit your environment:
  ** `install_yamls_path`
- ** `tripleo_passwords`
- ** `controller*_ssh`
+ ** `controller*_ssh`         (for each controller in each Heat stack on the source cloud)
+ ** `tripleo_cells_passwords` (for each Heat stack on the source cloud)
+ ** `source_mariadb_ip`       (for each cell controller on the source cloud)
+ ** `source_galera_members``  (for each cell controller on the source cloud)
+ ** `source_node_hostname``   (for each cell controller on the source cloud)
+ ** `edpm_node_hostname`      (for each cell compute node on the destination)
+ ** `edpm_node_ip`            (for each cell compute node on the destination)
  ** `edpm_privatekey_path`
+ ** `source_ovndb_ip``
  ** `timesync_ntp_servers`
 
 == Running the tests

docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc

@@ -229,18 +229,26 @@ EOF
 ----
 endif::[]
 
-. Deploy the `OpenStackDataPlaneNodeSet` CR:
+. Deploy the `OpenStackDataPlaneNodeSet` CRs (for each Nova compute cell):
 +
 . If TLS Everywhere is enabled, change spec:tlsEnabled to true
 . If using a custom DNS Domain, modify the spec:nodes:[NODE NAME]:hostName to use fqdn for the node
+. Use node set names, like `openstack-cell1`, `openstack-cell2`
+. Assign all compute nodes from the source cloud `default` cell into `openstack-X`,
+where `X` should become a real value defined in the `DEFAULT_CELL_NAME` environment variable
+. Assign all compute nodes from the source cloud `cell1` into `openstack-cell1`, and so on.
 +
+Note that after control plane adoption, the cell controllers should be decomissioned.
+They can become cell computes by following the scale-out, or scale-up, procedures
+for dataplane node sets and its members. However, that is not covered in this guide.
+
 [source,yaml]
 ----
 oc apply -f - <<EOF
 apiVersion: dataplane.openstack.org/v1beta1
 kind: OpenStackDataPlaneNodeSet
 metadata:
-  name: openstack
+  name: openstack-cell1
 spec:
   tlsEnabled: false
   networkAttachments:

docs_user/modules/proc_adopting-the-compute-service.adoc

@@ -144,18 +144,23 @@ $ openstack server list
 
 Compare the following outputs with the topology specific configuration in xref:proc_retrieving-topology-specific-service-configuration_migrating-databases[Retrieving topology-specific service configuration].
 
-* Query the superconductor for cell1 existance and compare it to pre-adoption values:
+* Query the superconductor for renamed cells existance, and compare it to pre-adoption values:
 +
 ----
 . ~/.source_cloud_exported_variables
-echo $PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS
-oc rsh nova-cell0-conductor-0 nova-manage cell_v2 list_cells | grep -F '| cell1 |'
+for CELL in $CELLS; do
+  RCELL=$CELL
+  [ "$CELL" = "default" ] && RCELL=$DEFAULT_CELL_NAME
+
+  echo $PULL_OPENSTACK_CONFIGURATION_NOVAMANAGE_CELL_MAPPINGS[$CELL]
+  oc rsh nova-${RCELL}-conductor-0 nova-manage cell_v2 list_cells | grep -F '| $RCELL |'
+done
 ----
 +
 The expected changes to happen:
 
- ** cell1's `nova` DB and user name become `nova_cell1`.
- ** Default cell is renamed to `cell1` (in a multi-cell setup, it should become indexed as the last cell instead).
+ ** The `cellX` cell's `nova` DB and user name should change to `nova_cellX`.
+ ** The `default` cell should become renamed to `DEFAULT_CELL_NAME` (it may contain the original name, if there are multiple cells).
  ** RabbitMQ transport URL no longer uses `guest`.
 
 [NOTE]

docs_user/modules/proc_deploying-backend-services.adoc

@@ -44,7 +44,9 @@ ADMIN_PASSWORD=SomePassword
 To use the existing {OpenStackShort} deployment password:
 +
 ----
-ADMIN_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' AdminPassword:' | awk -F ': ' '{ print $2; }')
+declare -A TRIPLEO_PASSWORDS
+TRIPLEO_PASSWORDS[default]='~/overcloud-deploy/overcloud/overcloud-passwords.yaml'
+ADMIN_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' AdminPassword:' | awk -F ': ' '{ print $2; }')
 ----
 * Set service password variables to match the original deployment.
 Database passwords can differ in the control plane environment, but
@@ -54,21 +56,21 @@ For example, in developer environments with {OpenStackPreviousInstaller} Standal
 passwords can be extracted like this:
 +
 ----
-AODH_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' AodhPassword:' | awk -F ': ' '{ print $2; }')
-BARBICAN_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' BarbicanPassword:' | awk -F ': ' '{ print $2; }')
-CEILOMETER_METERING_SECRET=$(cat ~/tripleo-standalone-passwords.yaml | grep ' CeilometerMeteringSecret:' | awk -F ': ' '{ print $2; }')
-CEILOMETER_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' CeilometerPassword:' | awk -F ': ' '{ print $2; }')
-CINDER_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' CinderPassword:' | awk -F ': ' '{ print $2; }')
-GLANCE_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' GlancePassword:' | awk -F ': ' '{ print $2; }')
-HEAT_AUTH_ENCRYPTION_KEY=$(cat ~/tripleo-standalone-passwords.yaml | grep ' HeatAuthEncryptionKey:' | awk -F ': ' '{ print $2; }')
-HEAT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' HeatPassword:' | awk -F ': ' '{ print $2; }')
-IRONIC_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' IronicPassword:' | awk -F ': ' '{ print $2; }')
-MANILA_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' ManilaPassword:' | awk -F ': ' '{ print $2; }')
-NEUTRON_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' NeutronPassword:' | awk -F ': ' '{ print $2; }')
-NOVA_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' NovaPassword:' | awk -F ': ' '{ print $2; }')
-OCTAVIA_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' OctaviaPassword:' | awk -F ': ' '{ print $2; }')
-PLACEMENT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' PlacementPassword:' | awk -F ': ' '{ print $2; }')
-SWIFT_PASSWORD=$(cat ~/tripleo-standalone-passwords.yaml | grep ' SwiftPassword:' | awk -F ': ' '{ print $2; }')
+AODH_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' AodhPassword:' | awk -F ': ' '{ print $2; }')
+BARBICAN_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' BarbicanPassword:' | awk -F ': ' '{ print $2; }')
+CEILOMETER_METERING_SECRET=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' CeilometerMeteringSecret:' | awk -F ': ' '{ print $2; }')
+CEILOMETER_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' CeilometerPassword:' | awk -F ': ' '{ print $2; }')
+CINDER_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' CinderPassword:' | awk -F ': ' '{ print $2; }')
+GLANCE_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' GlancePassword:' | awk -F ': ' '{ print $2; }')
+HEAT_AUTH_ENCRYPTION_KEY=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' HeatAuthEncryptionKey:' | awk -F ': ' '{ print $2; }')
+HEAT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' HeatPassword:' | awk -F ': ' '{ print $2; }')
+IRONIC_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' IronicPassword:' | awk -F ': ' '{ print $2; }')
+MANILA_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' ManilaPassword:' | awk -F ': ' '{ print $2; }')
+NEUTRON_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' NeutronPassword:' | awk -F ': ' '{ print $2; }')
+NOVA_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' NovaPassword:' | awk -F ': ' '{ print $2; }')
+OCTAVIA_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' OctaviaPassword:' | awk -F ': ' '{ print $2; }')
+PLACEMENT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' PlacementPassword:' | awk -F ': ' '{ print $2; }')
+SWIFT_PASSWORD=$(cat ${TRIPLEO_PASSWORDS[default]} | grep ' SwiftPassword:' | awk -F ': ' '{ print $2; }')
 ----
 
 .Procedure