doc: Add documentation on how to set up a dedicated Ceph cluster network

Signed-off-by: Gabriel Mougard <[email protected]>

doc/explanation/microcloud.rst

@@ -75,6 +75,9 @@ Disk sizes can vary.
 
 For distributed storage, MicroCloud uses MicroCeph, which is a lightweight way of deploying a Ceph cluster.
 To use distributed storage, you must have at least three disks (attached to at least three different machines).
+It is also possible to specify a Ceph network through the initialization wizard or through the preseed file.
+If you choose to have a dedicated Ceph network, each of your cluster member must have at least one network interface with
+an IP address in the same subnet as the Ceph network you specified.
 
 Troubleshooting
 ---------------

doc/how-to/initialise.rst

@@ -73,6 +73,9 @@ Complete the following steps to initialise MicroCloud:
       Wiping a disk will destroy all data on it.
 
    #. You can choose to optionally set up a CephFS distributed file system.
+   #. Select whether you want to set up a dedicated Ceph network.
+
+         If you choose ``yes``, you must enter the network subnet in CIDR notation. It could either be an IPv4 or an IPv6 subnet.
 #. Select whether you want to set up distributed networking (using MicroOVN).
 
    If you choose ``yes``, configure the distributed networking:

doc/how-to/preseed.yaml

@@ -24,6 +24,10 @@ systems:
 - name: micro04
   ovn_uplink_interface: eth1
 
+# `ceph` is optional and represents the Ceph global configuration
+ceph:
+  network: 10.0.1.0/24
+
 # `ovn` is optional and represents the OVN & uplink network configuration for LXD.
 ovn:
   ipv4_gateway: 192.0.2.1/24

doc/tutorial/get_started.rst

@@ -128,7 +128,7 @@ Complete the following steps to create the required disks in a LXD storage pool:
       | custom | remote3 |             | block        | 0       |
       +--------+---------+-------------+--------------+---------+
 
-3. Create a network
+3. Create networks
 -------------------
 
 MicroCloud requires an uplink network that the cluster members can use for external connectivity.
@@ -145,6 +145,17 @@ Complete the following steps to set up this network:
      lxc network get microbr0 ipv4.address
      lxc network get microbr0 ipv6.address
 
+You also have the option to create a dedicated network for the Ceph traffic:
+
+1. Create a bridge network without any parameters::
+
+     lxc network create cephbr0
+
+#. Enter the following commands to find out the assigned IPv4 and IPv6 addresses for the network and note them down::
+
+     lxc network get cephbr0 ipv4.address
+     lxc network get cephbr0 ipv6.address
+
 4. Create and configure your VMs
 --------------------------------
 
@@ -183,6 +194,13 @@ Complete the following steps:
      lxc config device add micro3 eth1 nic network=microbr0 name=eth1
      lxc config device add micro4 eth1 nic network=microbr0 name=eth1
 
+#. If you created a `cephbr0` network, create and add network interfaces that use the dedicated Ceph network to each VM::
+
+     lxc config device add micro1 eth2 nic network=cephbr0 name=eth2
+     lxc config device add micro2 eth2 nic network=cephbr0 name=eth2
+     lxc config device add micro3 eth2 nic network=cephbr0 name=eth2
+     lxc config device add micro4 eth2 nic network=cephbr0 name=eth2
+
 #. Start the VMs::
 
      lxc start micro1
@@ -218,6 +236,14 @@ Complete the following steps on each VM (``micro1``, ``micro2``, ``micro3``, and
 
      ip link set enp6s0 up
 
+
+#. If you set a `cephbr0` network, bring the other network interface up and give it an IP address within the network subnet::
+
+     # If the `cephbr0` gateway address is `192.168.0.1/24` (subnet should be `192.168.0.0/24`)
+     ip link set enp7s0 up
+     # `X` should be a number between 2 and 254, different for each VM
+     ip addr add 192.168.0.X/24 dev enp7s0
+
 #. Install the required snaps::
 
      snap install microceph --channel=quincy/stable --cohort="+"
@@ -272,6 +298,8 @@ Complete the following steps:
    #. Select all listed disks (these should be ``remote1``, ``remote2``, and ``remote3``).
    #. You don't need to wipe any disks (because we just created them).
    #. Select ``yes`` to optionally configure the CephFS distributed file system.
+   #. Select ``yes`` to configure a dedicated Ceph network.
+   #. Enter the IPv4 or IPv6 subnet address that you noted down for your ``cephbr0`` network (like ``192.168.0.0/24`` for example).
    #. Select ``yes`` to configure distributed networking.
    #. Select all listed network interfaces (these should be ``enp6s0`` on the four different VMs).
    #. Specify the IPv4 address that you noted down for your ``microbr0`` network as the IPv4 gateway.
@@ -385,6 +413,11 @@ See the full initialisation process here:
     Using 1 disk(s) on "micro3" for remote storage pool
 
    Would you like to set up CephFS remote storage? (yes/no) [default=yes]:  yes
+   Configure a dedicated Ceph network? (yes/no) [default=no]: yes
+   Choose either an IPv4 or IPv6 subnet (CIDR notation) to describe your Ceph dedicated cluster: 192.168.0.0/24
+   Interface "enp7s0" ("192.168.0.3") detected on cluster member "micro2"
+   Interface "enp7s0" ("192.168.0.4") detected on cluster member "micro3"
+   Interface "enp7s0" ("192.168.0.2") detected on cluster member "micro1"
    Configure distributed networking? (yes/no) [default=yes]:  yes
    Select an available interface per system to provide external connectivity for distributed network(s):
    Space to select; enter to confirm; type to filter results.
@@ -540,7 +573,19 @@ You can now inspect your cluster setup.
         total space: 29.67GiB
       used by: {}
 
-#. Inspect the network setup:
+#. Inspect the Ceph network setup:
+
+   .. terminal::
+      :input: microceph.ceph config dump
+      :user: root
+      :host: micro1
+      :scroll:
+
+      WHO     MASK  LEVEL     OPTION                       VALUE           RO
+      global        advanced  cluster_network              192.168.0.0/24  *
+      global        advanced  osd_pool_default_crush_rule  2
+
+#. Inspect the OVN network setup:
 
    .. terminal::
       :input: lxc network list
@@ -891,3 +936,44 @@ You can, however, create a different network to isolate some instances from othe
       14 packets transmitted, 0 received, 100% packet loss, time 13301ms
 
    You will see that ``u2`` is not reachable, because it is on a different OVN subnet.
+
+#. Inspect your Ceph related network traffic
+
+   .. terminal::
+      :input: lxc launch ubuntu:22.04 u5 -s remote
+      :user: root
+      :host: micro1
+      :scroll:
+
+      Creating c1
+      Starting c1
+
+#. At the same time, observe the Ceph traffic on the ``enp7s0`` interface (on any cluster member) using ``tcpdump``:
+
+   .. terminal::
+      :input: tcpdump -i enp7s0
+      :user: root
+      :host: micro2
+      :scroll:
+
+      17:48:48.600971 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329386555:329422755, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909539], length 36200
+      17:48:48.600971 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329422755:329451715, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 28960
+      17:48:48.601012 IP micro1.48746 > 10.33.3.4.6804: Flags [.], ack 329386555, win 24317, options [nop,nop,TS val 3647909564 ecr 3552095031], length 0
+      17:48:48.601089 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329451715:329516875, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 65160
+      17:48:48.601089 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329516875:329582035, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 65160
+      17:48:48.601089 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329582035:329624764, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 42729
+      17:48:48.601204 IP micro1.48746 > 10.33.3.4.6804: Flags [.], ack 329624764, win 23357, options [nop,nop,TS val 3647909564 ecr 3552095031], length 0
+      17:48:48.601206 IP 10.33.3.4.6803 > micro1.33328: Flags [P.], seq 938255:938512, ack 359644195, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909540], length 257
+      17:48:48.601310 IP micro1.48746 > 10.33.3.4.6804: Flags [P.], seq 245889462:245889506, ack 329624764, win 24576, options [nop,nop,TS val 3647909564 ecr 3552095031], length 44
+      17:48:48.602839 IP micro1.48746 > 10.33.3.4.6804: Flags [P.], seq 245889506:245889707, ack 329624764, win 24576, options [nop,nop,TS val 3647909566 ecr 3552095031], length 201
+      17:48:48.602947 IP 10.33.3.4.6804 > micro1.48746: Flags [.], ack 245889707, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909564], length 0
+      17:48:48.602975 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329624764:329624808, ack 245889707, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909564], length 44
+      17:48:48.603028 IP 10.33.3.4.6803 > micro1.33328: Flags [P.], seq 938512:938811, ack 359644195, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909540], length 299
+      17:48:48.603053 IP micro1.33328 > 10.33.3.4.6803: Flags [.], ack 938811, win 1886, options [nop,nop,TS val 3647909566 ecr 3552095031], length 0
+      17:48:48.604594 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359644195:359709355, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160
+      17:48:48.604644 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359709355:359774515, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160
+      17:48:48.604688 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359774515:359839675, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160
+      17:48:48.604733 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359839675:359904835, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160
+      17:48:48.604751 IP 10.33.3.4.6803 > micro1.33328: Flags [.], ack 359709355, win 24317, options [nop,nop,TS val 3552095035 ecr 3647909568], length 0
+      17:48:48.604757 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359904835:359910746, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095035], length 5911
+      17:48:48.604797 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359910746:359975906, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095035], length 65160