diff --git a/doc/explanation/microcloud.rst b/doc/explanation/microcloud.rst index 5a3827e07..767aee084 100644 --- a/doc/explanation/microcloud.rst +++ b/doc/explanation/microcloud.rst @@ -75,6 +75,9 @@ Disk sizes can vary. For distributed storage, MicroCloud uses MicroCeph, which is a lightweight way of deploying a Ceph cluster. To use distributed storage, you must have at least three disks (attached to at least three different machines). +It is also possible to specify a Ceph network through the initialization wizard or through the preseed file. +If you choose to have a dedicated Ceph network, each of your cluster member must have at least one network interface with +an IP address in the same subnet as the Ceph network you specified. Troubleshooting --------------- diff --git a/doc/how-to/initialise.rst b/doc/how-to/initialise.rst index 77608549b..ce5b84368 100644 --- a/doc/how-to/initialise.rst +++ b/doc/how-to/initialise.rst @@ -73,6 +73,9 @@ Complete the following steps to initialise MicroCloud: Wiping a disk will destroy all data on it. #. You can choose to optionally set up a CephFS distributed file system. + #. Select whether you want to set up a dedicated Ceph network. + + If you choose ``yes``, you must enter the network subnet in CIDR notation. It could either be an IPv4 or an IPv6 subnet. #. Select whether you want to set up distributed networking (using MicroOVN). If you choose ``yes``, configure the distributed networking: diff --git a/doc/how-to/preseed.yaml b/doc/how-to/preseed.yaml index 62ccca0a7..964ca0ece 100644 --- a/doc/how-to/preseed.yaml +++ b/doc/how-to/preseed.yaml @@ -24,6 +24,10 @@ systems: - name: micro04 ovn_uplink_interface: eth1 +# `ceph` is optional and represents the Ceph global configuration +ceph: + network: 10.0.1.0/24 + # `ovn` is optional and represents the OVN & uplink network configuration for LXD. ovn: ipv4_gateway: 192.0.2.1/24 diff --git a/doc/tutorial/get_started.rst b/doc/tutorial/get_started.rst index 726bee227..c726432a0 100644 --- a/doc/tutorial/get_started.rst +++ b/doc/tutorial/get_started.rst @@ -128,7 +128,7 @@ Complete the following steps to create the required disks in a LXD storage pool: | custom | remote3 | | block | 0 | +--------+---------+-------------+--------------+---------+ -3. Create a network +3. Create networks ------------------- MicroCloud requires an uplink network that the cluster members can use for external connectivity. @@ -145,6 +145,17 @@ Complete the following steps to set up this network: lxc network get microbr0 ipv4.address lxc network get microbr0 ipv6.address +You also have the option to create a dedicated network for the Ceph traffic: + +1. Create a bridge network without any parameters:: + + lxc network create cephbr0 + +#. Enter the following commands to find out the assigned IPv4 and IPv6 addresses for the network and note them down:: + + lxc network get cephbr0 ipv4.address + lxc network get cephbr0 ipv6.address + 4. Create and configure your VMs -------------------------------- @@ -183,6 +194,13 @@ Complete the following steps: lxc config device add micro3 eth1 nic network=microbr0 name=eth1 lxc config device add micro4 eth1 nic network=microbr0 name=eth1 +#. If you created a `cephbr0` network, create and add network interfaces that use the dedicated Ceph network to each VM:: + + lxc config device add micro1 eth2 nic network=cephbr0 name=eth2 + lxc config device add micro2 eth2 nic network=cephbr0 name=eth2 + lxc config device add micro3 eth2 nic network=cephbr0 name=eth2 + lxc config device add micro4 eth2 nic network=cephbr0 name=eth2 + #. Start the VMs:: lxc start micro1 @@ -218,6 +236,14 @@ Complete the following steps on each VM (``micro1``, ``micro2``, ``micro3``, and ip link set enp6s0 up + +#. If you set a `cephbr0` network, bring the other network interface up and give it an IP address within the network subnet:: + + # If the `cephbr0` gateway address is `192.168.0.1/24` (subnet should be `192.168.0.0/24`) + ip link set enp7s0 up + # `X` should be a number between 2 and 254, different for each VM + ip addr add 192.168.0.X/24 dev enp7s0 + #. Install the required snaps:: snap install microceph --channel=quincy/stable --cohort="+" @@ -272,6 +298,8 @@ Complete the following steps: #. Select all listed disks (these should be ``remote1``, ``remote2``, and ``remote3``). #. You don't need to wipe any disks (because we just created them). #. Select ``yes`` to optionally configure the CephFS distributed file system. + #. Select ``yes`` to configure a dedicated Ceph network. + #. Enter the IPv4 or IPv6 subnet address that you noted down for your ``cephbr0`` network (like ``192.168.0.0/24`` for example). #. Select ``yes`` to configure distributed networking. #. Select all listed network interfaces (these should be ``enp6s0`` on the four different VMs). #. Specify the IPv4 address that you noted down for your ``microbr0`` network as the IPv4 gateway. @@ -385,6 +413,11 @@ See the full initialisation process here: Using 1 disk(s) on "micro3" for remote storage pool Would you like to set up CephFS remote storage? (yes/no) [default=yes]: yes + Configure a dedicated Ceph network? (yes/no) [default=no]: yes + Choose either an IPv4 or IPv6 subnet (CIDR notation) to describe your Ceph dedicated cluster: 192.168.0.0/24 + Interface "enp7s0" ("192.168.0.3") detected on cluster member "micro2" + Interface "enp7s0" ("192.168.0.4") detected on cluster member "micro3" + Interface "enp7s0" ("192.168.0.2") detected on cluster member "micro1" Configure distributed networking? (yes/no) [default=yes]: yes Select an available interface per system to provide external connectivity for distributed network(s): Space to select; enter to confirm; type to filter results. @@ -540,7 +573,19 @@ You can now inspect your cluster setup. total space: 29.67GiB used by: {} -#. Inspect the network setup: +#. Inspect the Ceph network setup: + + .. terminal:: + :input: microceph.ceph config dump + :user: root + :host: micro1 + :scroll: + + WHO MASK LEVEL OPTION VALUE RO + global advanced cluster_network 192.168.0.0/24 * + global advanced osd_pool_default_crush_rule 2 + +#. Inspect the OVN network setup: .. terminal:: :input: lxc network list @@ -891,3 +936,44 @@ You can, however, create a different network to isolate some instances from othe 14 packets transmitted, 0 received, 100% packet loss, time 13301ms You will see that ``u2`` is not reachable, because it is on a different OVN subnet. + +#. Inspect your Ceph related network traffic + + .. terminal:: + :input: lxc launch ubuntu:22.04 u5 -s remote + :user: root + :host: micro1 + :scroll: + + Creating c1 + Starting c1 + +#. At the same time, observe the Ceph traffic on the ``enp7s0`` interface (on any cluster member) using ``tcpdump``: + + .. terminal:: + :input: tcpdump -i enp7s0 + :user: root + :host: micro2 + :scroll: + + 17:48:48.600971 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329386555:329422755, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909539], length 36200 + 17:48:48.600971 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329422755:329451715, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 28960 + 17:48:48.601012 IP micro1.48746 > 10.33.3.4.6804: Flags [.], ack 329386555, win 24317, options [nop,nop,TS val 3647909564 ecr 3552095031], length 0 + 17:48:48.601089 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329451715:329516875, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 65160 + 17:48:48.601089 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329516875:329582035, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 65160 + 17:48:48.601089 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329582035:329624764, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 42729 + 17:48:48.601204 IP micro1.48746 > 10.33.3.4.6804: Flags [.], ack 329624764, win 23357, options [nop,nop,TS val 3647909564 ecr 3552095031], length 0 + 17:48:48.601206 IP 10.33.3.4.6803 > micro1.33328: Flags [P.], seq 938255:938512, ack 359644195, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909540], length 257 + 17:48:48.601310 IP micro1.48746 > 10.33.3.4.6804: Flags [P.], seq 245889462:245889506, ack 329624764, win 24576, options [nop,nop,TS val 3647909564 ecr 3552095031], length 44 + 17:48:48.602839 IP micro1.48746 > 10.33.3.4.6804: Flags [P.], seq 245889506:245889707, ack 329624764, win 24576, options [nop,nop,TS val 3647909566 ecr 3552095031], length 201 + 17:48:48.602947 IP 10.33.3.4.6804 > micro1.48746: Flags [.], ack 245889707, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909564], length 0 + 17:48:48.602975 IP 10.33.3.4.6804 > micro1.48746: Flags [P.], seq 329624764:329624808, ack 245889707, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909564], length 44 + 17:48:48.603028 IP 10.33.3.4.6803 > micro1.33328: Flags [P.], seq 938512:938811, ack 359644195, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909540], length 299 + 17:48:48.603053 IP micro1.33328 > 10.33.3.4.6803: Flags [.], ack 938811, win 1886, options [nop,nop,TS val 3647909566 ecr 3552095031], length 0 + 17:48:48.604594 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359644195:359709355, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604644 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359709355:359774515, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604688 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359774515:359839675, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604733 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359839675:359904835, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604751 IP 10.33.3.4.6803 > micro1.33328: Flags [.], ack 359709355, win 24317, options [nop,nop,TS val 3552095035 ecr 3647909568], length 0 + 17:48:48.604757 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359904835:359910746, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095035], length 5911 + 17:48:48.604797 IP micro1.33328 > 10.33.3.4.6803: Flags [P.], seq 359910746:359975906, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095035], length 65160