Document Network Configurators (#363)

* Document Network Configurators Signed-off-by: Andrea Mazzotti <[email protected]>
rancher · Aug 13, 2024 · 3f7c7f8 · 3f7c7f8
1 parent 708ea30
commit 3f7c7f8
Show file tree

Hide file tree

Showing 7 changed files with 306 additions and 11 deletions.
diff --git a/docs/machineregistration-reference.md b/docs/machineregistration-reference.md
@@ -52,9 +52,14 @@ Both yip and cloud-init syntax are supported. See the [Cloud Config Reference](c
 #### config.network
 
 Contains the Declarative Networking configuration, supporting integration with [CAPI IPAM Providers](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220125-ipam-integration.md#ipam-provider).  
-Supports [nmstate](https://nmstate.io) syntax.  
 See the [Declarative Networking Reference](networking.md) for full information.  
 
+| Key               | Type      | Default value | Description                                                                                                   |
+|-------------------|-----------|---------------|---------------------------------------------------------------------------------------------------------------|
+| configurator      | string    | nmc           | The network configurator type to use (`nmc`, `nmstate`, or `nmconnections`)                                   |
+| ipAddresses       | objRefMap | empty         | A map of `IPPool` references. Map keys can be used for IPAddress substitution in the network config template. |
+| config            | obj       | empty         | The network config template. Syntax varies depending on the `configurator` in use.                            |
+
 #### config.elemental.registration
 Contains the configuration used for the connection and the initial registration to the <Vars name="elemental_operator_name" />.
 

diff --git a/docs/networking.md b/docs/networking.md
@@ -8,6 +8,9 @@ title: ''
 </head>
 
 import RegistrationWithNetwork from "!!raw-loader!@site/examples/network/machineregistration.yaml"
+import RegistrationWithNetworkNmc from "!!raw-loader!@site/examples/network/machineregistration-nmc.yaml"
+import RegistrationWithNetworkNmstate from "!!raw-loader!@site/examples/network/machineregistration-nmstate.yaml"
+import RegistrationWithNetworkNmconnections from "!!raw-loader!@site/examples/network/machineregistration-nmconnections.yaml"
 
 ## Network configuration with Elemental
 
@@ -20,8 +23,12 @@ The [MachineRegistration](machineregistration-reference) supports Declarative Ne
 - An IPAM Provider of your choice is installed on the Rancher management cluster.  
   For example the [InCluster IPAM Provider](https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster).
 
-- [nmstatectl](https://github.com/nmstate/nmstate/releases) and [NetworkManager](https://networkmanager.dev) need to be installed on the Elemental OS in use.  
-  Elemental provided images already include these dependencies, this only applies when building custom images.  
+- [NetworkManager](https://networkmanager.dev) needs to be installed on OS images and it can be directly configured using the `nmconnections` network [configurator](#configurators).  
+  Already included in Elemental provided images.  
+
+- (optionally) [nmc](https://github.com/suse-edge/nm-configurator/releases) can be used with `nmc` network [configurator](#configurators).  
+
+- (optionally) [nmstatectl](https://github.com/nmstate/nmstate/releases) can be used with `nmstate` network [configurator](#configurators).
 
 ### How to install the CAPI IPAM Provider
 
@@ -72,7 +79,7 @@ Note that this solution may eventually lead to conflicts with the applied CRDs a
 The `network` section of the `MachineRegistration` allows users to define:
 
 1. A map of IPPool references.
-1. A nmstate configuration template.
+1. A network config template (in this case `nmc` configurator is in use).  
 
 For example:
 
@@ -129,7 +136,7 @@ config:
         enabled: false
 ```
 
-The snippet above is almost 1:1 [nmstate syntax](https://nmstate.io/examples.html#nmstate-state-examples), with the only exception of the `{inventory-ip}` placeholder.  
+The snippet above is almost 1:1 [nm-configurator syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations), with the only exception of the `{inventory-ip}` placeholder.  
 During the installation or reset phases of Elemental machines, the `elemental-operator` will claim one IP Address from the referenced IP Pool, and substitute the `{inventory-ip}` placeholder with a real IP Address.  
 
 ### Claimed IPAddresses
@@ -162,3 +169,52 @@ status:
 Whenever a `MachineInventory` is deleted, the default (DHCP) network configuration will be restored and the IPs assigned will be released.  
 
 For more information and details on how troubleshoot issues, please consult the [documentation](./troubleshooting-network.md).
+
+### Configurators
+
+On the Elemental machine, `elemental-register` can configure the `NetworkManager` in different ways.  
+The configurator in use is defined in the [MachineRegistration.spec.network](./machineregistration-reference.md#confignetwork):  
+
+- [nmc](https://github.com/suse-edge/nm-configurator)
+- [nmstate](https://nmstate.io/)
+- [nmconnections](https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-settings-keyfile.html)
+
+<Tabs>
+
+<TabItem value="nmc" label="nmc" default>
+
+The `nmc` configurator uses the [nm-configurator unified syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations) to generate NetworkManager's connection files.  
+
+<details>
+  <summary>example MachineRegistration using nmc configurator</summary>
+<CodeBlock language="yaml" showLineNumbers>{RegistrationWithNetworkNmc}</CodeBlock>
+</details>
+
+</TabItem>
+
+<TabItem value="nmstate" label="nmstate" default>
+
+The `nmstate` configurator uses [nmstate syntax](https://nmstate.io/examples.html) to generate NetworkManager's connection files.  
+Note that [nmstatectl](https://github.com/nmstate/nmstate/releases) needs to be installed on the Elemental system to use this configurator. This is not included by default in Elemental images, but can be installed when building a [custom image](./custom-images.md).
+
+<details>
+  <summary>example MachineRegistration using nmstate configurator</summary>
+<CodeBlock language="yaml" showLineNumbers>{RegistrationWithNetworkNmstate}</CodeBlock>
+</details>
+
+</TabItem>
+
+<TabItem value="nmconnections" label="nmconnections" default>
+
+The `nmconnections` configurator is the simplest option available and allows the user to directly write `nmconnection` files.  
+Defining these files for complex network setups may be challenging, but it's always possible to use [nmcli](https://networkmanager.dev/docs/api/latest/nmcli.html), or even [nmstate](https://nmstate.io), or [nm-configurator](https://github.com/suse-edge/nm-configurator), and use the generated `nmconnection` files as a template.  
+This configurator only needs `NetworkManager`, without any extra dependency.  
+
+<details>
+  <summary>example MachineRegistration using nmconnections configurator</summary>
+<CodeBlock language="yaml" showLineNumbers>{RegistrationWithNetworkNmconnections}</CodeBlock>
+</details>
+
+</TabItem>
+
+</Tabs>
diff --git a/docs/troubleshooting-network.md b/docs/troubleshooting-network.md
@@ -15,7 +15,7 @@ Given the following sample registration:
 
 <CodeBlock language="yaml" title="example MachineRegistration using Declarative Networking" showLineNumbers>{RegistrationWithNetwork}</CodeBlock>
 
-We can expect each Elemental Machine to be configured using the defined `nmstate` template.  
+We can expect each Elemental Machine to be configured using the defined `nm-configurator` `_all.yaml` template.  
 
 At the very first boot, the `elemental-register` will try to contact the Rancher API to register a new `MachineInventory`.  
 At this stage the machine's network is not configured and will default to DHCP. It is a requirement that the machine is able to contact the Rancher API in this setup, otherwise the registration can not take place.  
@@ -89,7 +89,7 @@ You will notice that the `MachineInventory` carries the same `network.config` as
       inventory-ip: 192.168.122.150
 ```
 
-This `inventory-ip` will then be substituted in the `nmstate` config whenever `{inventory-ip}` has been defined.  
+This `inventory-ip` will then be substituted in the `nm-configurator` config whenever `{inventory-ip}` has been defined.  
 
 Also note that the `MachineInventory` references and owns each `IPAddressClaim` associated with it. Each claim follow the predictable `$MachineIventoryName-$IPPoolRefKey` naming convention: `m-e5331e3b-1e1b-4ce7-b080-235ed9a6d07c-inventory-ip`.  
 These claims will follow the lifecycle of the `MachineInventory` object and be deleted on cascade, for example during the [reset workflow](./reset.md).  
@@ -98,7 +98,7 @@ If the `IPAddresses` can not be claimed, the `NetworkConfigReady` condition will
 
 ## On the machine side
 
-During the installation phase, the `elemental-register` process running on the machine will receive the `nmstate` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nmstate` configuration:
+During the installation phase, the `elemental-register` process running on the machine will receive the `nm-configurator` `_all.yaml` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nm-configurator` configuration:
 
 ```yaml
 config:
@@ -129,10 +129,10 @@ config:
       table-id: 254
 ```
 
-The `elemental-register` will then invoke `nmstatectl apply` to apply this configuration into the running system.  
+The `elemental-register` will then invoke `nmc generate` and `nmc apply` to apply this configuration into the running system.  
 From this moment until reset, the machine will always use the applied configuration.  
 
-Also note that outside of installation and reset, `nmstate` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmstatectl` rather than the `nmstate` configuration itself.  
+Also note that outside of installation and reset, `nm-configurator` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmc` rather than the `nmc` configuration itself.  
 
 For example on any running system, you will find a [yip](https://github.com/rancher/yip) configuration file (`/oem/94_custom.yaml`) to apply the desired `nmconnections`, for example:  
 
@@ -174,7 +174,7 @@ stages:
                 [proxy]
 
                 [user]
-                nmstate.interface.description=Main-NIC
+                nm-configurator.interface.description=Main-NIC
               encoding: ""
               ownerstring: ""
           if: '[ ! -f /run/elemental/recovery_mode ]'

diff --git a/examples/network/machineregistration-nmc.yaml b/examples/network/machineregistration-nmc.yaml
@@ -0,0 +1,84 @@
+apiVersion: ipam.cluster.x-k8s.io/v1alpha2
+kind: InClusterIPPool
+metadata:
+  name: elemental-inventory-pool
+  namespace: fleet-default
+spec:
+  addresses:
+    - 192.168.122.150-192.168.122.200
+  prefix: 24
+  gateway: 192.168.122.1
+---
+apiVersion: ipam.cluster.x-k8s.io/v1alpha2
+kind: InClusterIPPool
+metadata:
+  name: elemental-secondary-pool
+  namespace: fleet-default
+spec:
+  addresses:
+    - 172.16.0.150-172.16.0.200
+  prefix: 24
+  gateway: 172.16.0.1
+---
+apiVersion: elemental.cattle.io/v1beta1
+kind: MachineRegistration
+metadata:
+  name: fire-nodes
+  namespace: fleet-default
+spec:
+  machineName: m-${System Information/UUID}
+  config:
+    network:
+      configurator: nmc
+      ipAddresses:
+        inventory-ip:
+          apiGroup: ipam.cluster.x-k8s.io
+          kind: InClusterIPPool
+          name: elemental-inventory-pool
+        secondary-ip:
+          apiGroup: ipam.cluster.x-k8s.io
+          kind: InClusterIPPool
+          name: elemental-secondary-pool
+      config:
+        dns-resolver:
+          config:
+            server:
+            - 192.168.122.1
+            search: []
+        routes:
+          config:
+          - destination: 0.0.0.0/0
+            next-hop-interface: enp1s0
+            next-hop-address: 192.168.122.1
+            metric: 150
+            table-id: 254
+          - destination: 172.16.0.1/24
+            next-hop-interface: enp8s0
+            next-hop-address: 172.16.0.1
+            metric: 150
+            table-id: 254
+        interfaces:
+          - name: enp1s0
+            type: ethernet
+            description: Main-NIC
+            state: up
+            ipv4:
+              enabled: true
+              dhcp: false
+              address:
+              - ip: "{inventory-ip}"
+                prefix-length: 24
+            ipv6:
+              enabled: false
+          - name: enp8s0
+            type: ethernet
+            description: Secondary-NIC
+            state: up
+            ipv4:
+              enabled: true
+              dhcp: false
+              address:
+              - ip: "{secondary-ip}"
+                prefix-length: 24
+            ipv6:
+              enabled: false
diff --git a/examples/network/machineregistration-nmconnections.yaml b/examples/network/machineregistration-nmconnections.yaml
@@ -0,0 +1,65 @@
+apiVersion: ipam.cluster.x-k8s.io/v1alpha2
+kind: InClusterIPPool
+metadata:
+  name: elemental-inventory-pool
+  namespace: fleet-default
+spec:
+  addresses:
+    - 192.168.122.150-192.168.122.200
+  prefix: 24
+  gateway: 192.168.122.1
+---
+apiVersion: ipam.cluster.x-k8s.io/v1alpha2
+kind: InClusterIPPool
+metadata:
+  name: elemental-secondary-pool
+  namespace: fleet-default
+spec:
+  addresses:
+    - 172.16.0.150-172.16.0.200
+  prefix: 24
+  gateway: 172.16.0.1
+---
+apiVersion: elemental.cattle.io/v1beta1
+kind: MachineRegistration
+metadata:
+  name: fire-nodes
+  namespace: fleet-default
+spec:
+  machineName: test-${System Information/UUID}
+  config:
+    network:
+      configurator: "nmconnections"
+      ipAddresses:
+        inventory-ip:
+          apiGroup: ipam.cluster.x-k8s.io
+          kind: InClusterIPPool
+          name: elemental-inventory-pool
+        secondary-ip:
+          apiGroup: ipam.cluster.x-k8s.io
+          kind: InClusterIPPool
+          name: elemental-secondary-pool
+      config:
+        enp1s0: |
+          [connection]
+          id=Wired connection 1
+          type=ethernet
+          interface-name=enp1s0
+          [ipv4]
+          address1={inventory-ip}/24,192.168.1.1
+          dns=192.168.122.1;
+          method=manual
+          route1=0.0.0.0/0,192.168.122.1
+          [ipv6]
+          method=disabled
+        enp8s0: |
+          [connection]
+          id=Wired connection 2
+          type=ethernet
+          interface-name=enp8s0
+          [ipv4]
+          address1={secondary-ip}/24,172.16.0.1
+          method=manual
+          route1=172.16.0.0/24,172.16.0.1,150
+          [ipv6]
+          method=disabled
diff --git a/examples/network/machineregistration-nmstate.yaml b/examples/network/machineregistration-nmstate.yaml
@@ -0,0 +1,84 @@
+apiVersion: ipam.cluster.x-k8s.io/v1alpha2
+kind: InClusterIPPool
+metadata:
+  name: elemental-inventory-pool
+  namespace: fleet-default
+spec:
+  addresses:
+    - 192.168.122.150-192.168.122.200
+  prefix: 24
+  gateway: 192.168.122.1
+---
+apiVersion: ipam.cluster.x-k8s.io/v1alpha2
+kind: InClusterIPPool
+metadata:
+  name: elemental-secondary-pool
+  namespace: fleet-default
+spec:
+  addresses:
+    - 172.16.0.150-172.16.0.200
+  prefix: 24
+  gateway: 172.16.0.1
+---
+apiVersion: elemental.cattle.io/v1beta1
+kind: MachineRegistration
+metadata:
+  name: fire-nodes
+  namespace: fleet-default
+spec:
+  machineName: m-${System Information/UUID}
+  config:
+    network:
+      configurator: nmstate
+      ipAddresses:
+        inventory-ip:
+          apiGroup: ipam.cluster.x-k8s.io
+          kind: InClusterIPPool
+          name: elemental-inventory-pool
+        secondary-ip:
+          apiGroup: ipam.cluster.x-k8s.io
+          kind: InClusterIPPool
+          name: elemental-secondary-pool
+      config:
+        dns-resolver:
+          config:
+            server:
+            - 192.168.122.1
+            search: []
+        routes:
+          config:
+          - destination: 0.0.0.0/0
+            next-hop-interface: enp1s0
+            next-hop-address: 192.168.122.1
+            metric: 150
+            table-id: 254
+          - destination: 172.16.0.1/24
+            next-hop-interface: enp8s0
+            next-hop-address: 172.16.0.1
+            metric: 150
+            table-id: 254
+        interfaces:
+          - name: enp1s0
+            type: ethernet
+            description: Main-NIC
+            state: up
+            ipv4:
+              enabled: true
+              dhcp: false
+              address:
+              - ip: "{inventory-ip}"
+                prefix-length: 24
+            ipv6:
+              enabled: false
+          - name: enp8s0
+            type: ethernet
+            description: Secondary-NIC
+            state: up
+            ipv4:
+              enabled: true
+              dhcp: false
+              address:
+              - ip: "{secondary-ip}"
+                prefix-length: 24
+            ipv6:
+              enabled: false