diff --git a/docs/machineregistration-reference.md b/docs/machineregistration-reference.md index f9506d035..c6b565ec2 100644 --- a/docs/machineregistration-reference.md +++ b/docs/machineregistration-reference.md @@ -52,9 +52,14 @@ Both yip and cloud-init syntax are supported. See the [Cloud Config Reference](c #### config.network Contains the Declarative Networking configuration, supporting integration with [CAPI IPAM Providers](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220125-ipam-integration.md#ipam-provider). -Supports [nmstate](https://nmstate.io) syntax. See the [Declarative Networking Reference](networking.md) for full information. +| Key | Type | Default value | Description | +|-------------------|-----------|---------------|---------------------------------------------------------------------------------------------------------------| +| configurator | string | nmc | The network configurator type to use (`nmc`, `nmstate`, or `nmconnections`) | +| ipAddresses | objRefMap | empty | A map of `IPPool` references. Map keys can be used for IPAddress substitution in the network config template. | +| config | obj | empty | The network config template. Syntax varies depending on the `configurator` in use. | + #### config.elemental.registration Contains the configuration used for the connection and the initial registration to the . diff --git a/docs/networking.md b/docs/networking.md index 8b2a9cfad..0c7ec9a76 100644 --- a/docs/networking.md +++ b/docs/networking.md @@ -8,6 +8,9 @@ title: '' import RegistrationWithNetwork from "!!raw-loader!@site/examples/network/machineregistration.yaml" +import RegistrationWithNetworkNmc from "!!raw-loader!@site/examples/network/machineregistration-nmc.yaml" +import RegistrationWithNetworkNmstate from "!!raw-loader!@site/examples/network/machineregistration-nmstate.yaml" +import RegistrationWithNetworkNmconnections from "!!raw-loader!@site/examples/network/machineregistration-nmconnections.yaml" ## Network configuration with Elemental @@ -20,8 +23,12 @@ The [MachineRegistration](machineregistration-reference) supports Declarative Ne - An IPAM Provider of your choice is installed on the Rancher management cluster. For example the [InCluster IPAM Provider](https://github.com/kubernetes-sigs/cluster-api-ipam-provider-in-cluster). -- [nmstatectl](https://github.com/nmstate/nmstate/releases) and [NetworkManager](https://networkmanager.dev) need to be installed on the Elemental OS in use. - Elemental provided images already include these dependencies, this only applies when building custom images. +- [NetworkManager](https://networkmanager.dev) needs to be installed on OS images and it can be directly configured using the `nmconnections` network [configurator](#configurators). + Already included in Elemental provided images. + +- (optionally) [nmc](https://github.com/suse-edge/nm-configurator/releases) can be used with `nmc` network [configurator](#configurators). + +- (optionally) [nmstatectl](https://github.com/nmstate/nmstate/releases) can be used with `nmstate` network [configurator](#configurators). ### How to install the CAPI IPAM Provider @@ -72,7 +79,7 @@ Note that this solution may eventually lead to conflicts with the applied CRDs a The `network` section of the `MachineRegistration` allows users to define: 1. A map of IPPool references. -1. A nmstate configuration template. +1. A network config template (in this case `nmc` configurator is in use). For example: @@ -129,7 +136,7 @@ config: enabled: false ``` -The snippet above is almost 1:1 [nmstate syntax](https://nmstate.io/examples.html#nmstate-state-examples), with the only exception of the `{inventory-ip}` placeholder. +The snippet above is almost 1:1 [nm-configurator syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations), with the only exception of the `{inventory-ip}` placeholder. During the installation or reset phases of Elemental machines, the `elemental-operator` will claim one IP Address from the referenced IP Pool, and substitute the `{inventory-ip}` placeholder with a real IP Address. ### Claimed IPAddresses @@ -162,3 +169,52 @@ status: Whenever a `MachineInventory` is deleted, the default (DHCP) network configuration will be restored and the IPs assigned will be released. For more information and details on how troubleshoot issues, please consult the [documentation](./troubleshooting-network.md). + +### Configurators + +On the Elemental machine, `elemental-register` can configure the `NetworkManager` in different ways. +The configurator in use is defined in the [MachineRegistration.spec.network](./machineregistration-reference.md#confignetwork): + +- [nmc](https://github.com/suse-edge/nm-configurator) +- [nmstate](https://nmstate.io/) +- [nmconnections](https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-settings-keyfile.html) + + + + + +The `nmc` configurator uses the [nm-configurator unified syntax](https://github.com/suse-edge/nm-configurator?tab=readme-ov-file#unified-configurations) to generate NetworkManager's connection files. + +
+ example MachineRegistration using nmc configurator +{RegistrationWithNetworkNmc} +
+ + + + + +The `nmstate` configurator uses [nmstate syntax](https://nmstate.io/examples.html) to generate NetworkManager's connection files. +Note that [nmstatectl](https://github.com/nmstate/nmstate/releases) needs to be installed on the Elemental system to use this configurator. This is not included by default in Elemental images, but can be installed when building a [custom image](./custom-images.md). + +
+ example MachineRegistration using nmstate configurator +{RegistrationWithNetworkNmstate} +
+ + + + + +The `nmconnections` configurator is the simplest option available and allows the user to directly write `nmconnection` files. +Defining these files for complex network setups may be challenging, but it's always possible to use [nmcli](https://networkmanager.dev/docs/api/latest/nmcli.html), or even [nmstate](https://nmstate.io), or [nm-configurator](https://github.com/suse-edge/nm-configurator), and use the generated `nmconnection` files as a template. +This configurator only needs `NetworkManager`, without any extra dependency. + +
+ example MachineRegistration using nmconnections configurator +{RegistrationWithNetworkNmconnections} +
+ + + + diff --git a/docs/troubleshooting-network.md b/docs/troubleshooting-network.md index 59595522b..c05fb112e 100644 --- a/docs/troubleshooting-network.md +++ b/docs/troubleshooting-network.md @@ -15,7 +15,7 @@ Given the following sample registration: {RegistrationWithNetwork} -We can expect each Elemental Machine to be configured using the defined `nmstate` template. +We can expect each Elemental Machine to be configured using the defined `nm-configurator` `_all.yaml` template. At the very first boot, the `elemental-register` will try to contact the Rancher API to register a new `MachineInventory`. At this stage the machine's network is not configured and will default to DHCP. It is a requirement that the machine is able to contact the Rancher API in this setup, otherwise the registration can not take place. @@ -89,7 +89,7 @@ You will notice that the `MachineInventory` carries the same `network.config` as inventory-ip: 192.168.122.150 ``` -This `inventory-ip` will then be substituted in the `nmstate` config whenever `{inventory-ip}` has been defined. +This `inventory-ip` will then be substituted in the `nm-configurator` config whenever `{inventory-ip}` has been defined. Also note that the `MachineInventory` references and owns each `IPAddressClaim` associated with it. Each claim follow the predictable `$MachineIventoryName-$IPPoolRefKey` naming convention: `m-e5331e3b-1e1b-4ce7-b080-235ed9a6d07c-inventory-ip`. These claims will follow the lifecycle of the `MachineInventory` object and be deleted on cascade, for example during the [reset workflow](./reset.md). @@ -98,7 +98,7 @@ If the `IPAddresses` can not be claimed, the `NetworkConfigReady` condition will ## On the machine side -During the installation phase, the `elemental-register` process running on the machine will receive the `nmstate` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nmstate` configuration: +During the installation phase, the `elemental-register` process running on the machine will receive the `nm-configurator` `_all.yaml` config template and the list of claimed IPAddresses with their keys. This information will be digested to an applicable `nm-configurator` configuration: ```yaml config: @@ -129,10 +129,10 @@ config: table-id: 254 ``` -The `elemental-register` will then invoke `nmstatectl apply` to apply this configuration into the running system. +The `elemental-register` will then invoke `nmc generate` and `nmc apply` to apply this configuration into the running system. From this moment until reset, the machine will always use the applied configuration. -Also note that outside of installation and reset, `nmstate` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmstatectl` rather than the `nmstate` configuration itself. +Also note that outside of installation and reset, `nm-configurator` is no longer used, since the `elemental-register` will persist the `/etc/NetworkManager/system-connection/*.nmconnection` files generated by `nmc` rather than the `nmc` configuration itself. For example on any running system, you will find a [yip](https://github.com/rancher/yip) configuration file (`/oem/94_custom.yaml`) to apply the desired `nmconnections`, for example: @@ -174,7 +174,7 @@ stages: [proxy] [user] - nmstate.interface.description=Main-NIC + nm-configurator.interface.description=Main-NIC encoding: "" ownerstring: "" if: '[ ! -f /run/elemental/recovery_mode ]' diff --git a/examples/network/machineregistration-nmc.yaml b/examples/network/machineregistration-nmc.yaml new file mode 100644 index 000000000..d6187dcf7 --- /dev/null +++ b/examples/network/machineregistration-nmc.yaml @@ -0,0 +1,84 @@ +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-inventory-pool + namespace: fleet-default +spec: + addresses: + - 192.168.122.150-192.168.122.200 + prefix: 24 + gateway: 192.168.122.1 +--- +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-secondary-pool + namespace: fleet-default +spec: + addresses: + - 172.16.0.150-172.16.0.200 + prefix: 24 + gateway: 172.16.0.1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: fire-nodes + namespace: fleet-default +spec: + machineName: m-${System Information/UUID} + config: + network: + configurator: nmc + ipAddresses: + inventory-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-inventory-pool + secondary-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-secondary-pool + config: + dns-resolver: + config: + server: + - 192.168.122.1 + search: [] + routes: + config: + - destination: 0.0.0.0/0 + next-hop-interface: enp1s0 + next-hop-address: 192.168.122.1 + metric: 150 + table-id: 254 + - destination: 172.16.0.1/24 + next-hop-interface: enp8s0 + next-hop-address: 172.16.0.1 + metric: 150 + table-id: 254 + interfaces: + - name: enp1s0 + type: ethernet + description: Main-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{inventory-ip}" + prefix-length: 24 + ipv6: + enabled: false + - name: enp8s0 + type: ethernet + description: Secondary-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{secondary-ip}" + prefix-length: 24 + ipv6: + enabled: false diff --git a/examples/network/machineregistration-nmconnections.yaml b/examples/network/machineregistration-nmconnections.yaml new file mode 100644 index 000000000..792ec07dd --- /dev/null +++ b/examples/network/machineregistration-nmconnections.yaml @@ -0,0 +1,65 @@ +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-inventory-pool + namespace: fleet-default +spec: + addresses: + - 192.168.122.150-192.168.122.200 + prefix: 24 + gateway: 192.168.122.1 +--- +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-secondary-pool + namespace: fleet-default +spec: + addresses: + - 172.16.0.150-172.16.0.200 + prefix: 24 + gateway: 172.16.0.1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: fire-nodes + namespace: fleet-default +spec: + machineName: test-${System Information/UUID} + config: + network: + configurator: "nmconnections" + ipAddresses: + inventory-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-inventory-pool + secondary-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-secondary-pool + config: + enp1s0: | + [connection] + id=Wired connection 1 + type=ethernet + interface-name=enp1s0 + [ipv4] + address1={inventory-ip}/24,192.168.1.1 + dns=192.168.122.1; + method=manual + route1=0.0.0.0/0,192.168.122.1 + [ipv6] + method=disabled + enp8s0: | + [connection] + id=Wired connection 2 + type=ethernet + interface-name=enp8s0 + [ipv4] + address1={secondary-ip}/24,172.16.0.1 + method=manual + route1=172.16.0.0/24,172.16.0.1,150 + [ipv6] + method=disabled diff --git a/examples/network/machineregistration-nmstate.yaml b/examples/network/machineregistration-nmstate.yaml new file mode 100644 index 000000000..faad1909d --- /dev/null +++ b/examples/network/machineregistration-nmstate.yaml @@ -0,0 +1,84 @@ +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-inventory-pool + namespace: fleet-default +spec: + addresses: + - 192.168.122.150-192.168.122.200 + prefix: 24 + gateway: 192.168.122.1 +--- +apiVersion: ipam.cluster.x-k8s.io/v1alpha2 +kind: InClusterIPPool +metadata: + name: elemental-secondary-pool + namespace: fleet-default +spec: + addresses: + - 172.16.0.150-172.16.0.200 + prefix: 24 + gateway: 172.16.0.1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: fire-nodes + namespace: fleet-default +spec: + machineName: m-${System Information/UUID} + config: + network: + configurator: nmstate + ipAddresses: + inventory-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-inventory-pool + secondary-ip: + apiGroup: ipam.cluster.x-k8s.io + kind: InClusterIPPool + name: elemental-secondary-pool + config: + dns-resolver: + config: + server: + - 192.168.122.1 + search: [] + routes: + config: + - destination: 0.0.0.0/0 + next-hop-interface: enp1s0 + next-hop-address: 192.168.122.1 + metric: 150 + table-id: 254 + - destination: 172.16.0.1/24 + next-hop-interface: enp8s0 + next-hop-address: 172.16.0.1 + metric: 150 + table-id: 254 + interfaces: + - name: enp1s0 + type: ethernet + description: Main-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{inventory-ip}" + prefix-length: 24 + ipv6: + enabled: false + - name: enp8s0 + type: ethernet + description: Secondary-NIC + state: up + ipv4: + enabled: true + dhcp: false + address: + - ip: "{secondary-ip}" + prefix-length: 24 + ipv6: + enabled: false diff --git a/examples/network/machineregistration.yaml b/examples/network/machineregistration.yaml index 867bdf417..2a52ce623 100644 --- a/examples/network/machineregistration.yaml +++ b/examples/network/machineregistration.yaml @@ -18,6 +18,7 @@ spec: machineName: m-${System Information/UUID} config: network: + configurator: nmc ipAddresses: inventory-ip: apiGroup: ipam.cluster.x-k8s.io