cloud-init is reseting DNS network config from dhcp after Fedora 40 -> 41 upgrade on Hetzner cloud

[pschiffe]

Bug report

After upgrade from Fedora 40 to Fedora 41 (cloud-init-24.1.4-3.fc40.noarch -> cloud-init-24.2-3.fc41.noarch) on Hetzner cloud; the custom DNS configuration done by network manager is overridden during reboot by cloud-init. The log also show some exceptions, see below.

Before with Fedora 40 and cloud-init-24.1.4-3.fc40.noarch the log just showed, without errors:

2024-11-12 18:50:05,852 - stages.py[DEBUG]: Allowed events: {<EventScope.NETWORK: 'network'>: {<EventType.BOOT_NEW_INSTANCE: 'boot-new-instance'>}}
2024-11-12 18:50:05,852 - stages.py[DEBUG]: Event Denied: scopes=['network'] EventType=boot-legacy
2024-11-12 18:50:05,852 - stages.py[DEBUG]: No network config applied. Neither a new instance nor datasource network update allowed
2024-11-12 18:50:05,852 - stages.py[DEBUG]: applying net config names for {'config': [{'mac_address': '96:00:02:9f:7f:6e', 'name': 'eth0', 'subnets': [{'ipv4': True, 'type': 'dhcp'}, {'address': '2a01:4f8:c013:ce8::1/64', 'dns_nameservers': ['2a01:4ff:ff00::add:2', '2a01:4ff:ff00::add:1'], 'gateway': 'fe80::1', 'ipv6': True, 'type': 'static'}], 'type': 'physical'}], 'version': 1}

And the custom DNS configuration done by nmcli wasn't changed. Now there are exceptions and the DNS config is reset.

Steps to reproduce the problem

I didn't do any changes to cloud-init. There's configuration provided by Hetzner:

$ cat /etc/cloud/cloud.cfg.d/90-hetznercloud.cfg 
# The top level settings are used as module
# and system configuration.

# A set of users which may be applied and/or used by various modules
# when a 'default' entry is found it will reference the 'default_user'
# from the distro configuration specified below
users:
   - default

# If this is set, 'root' will not be able to ssh in and they
# will get a message to login instead as the default $user
disable_root: false

mount_default_fields: [~, ~, 'auto', 'defaults,nofail', '0', '2']
resize_rootfs_tmp: /dev
ssh_pwauth:   0

# This will cause the set+update hostname module to not operate (if true)
preserve_hostname: false

# Example datasource config
# datasource:
#    Ec2:
#      metadata_urls: [ 'blah.com' ]
#      timeout: 5 # (defaults to 50 seconds)
#      max_wait: 10 # (defaults to 120 seconds)

# The modules that run in the 'init' stage
cloud_init_modules:
 - migrator
 - seed_random
 - bootcmd
 - write-files
 - [ growpart, always ]
 - [ resizefs, always ]
 - disk_setup
 - mounts
 - set_hostname
 - update_hostname
 - [ update_etc_hosts, once-per-instance ]
 - ca-certs
 - rsyslog
 - users-groups
 - ssh

# The modules that run in the 'config' stage
cloud_config_modules:
 - ssh-import-id
 - locale
 - set-passwords
 - spacewalk
 - yum-add-repo
 - ntp
 - timezone
 - disable-ec2-metadata
 - runcmd

# The modules that run in the 'final' stage
cloud_final_modules:
 - package-update-upgrade-install
 - puppet
 - chef
 - mcollective
 - salt-minion
 - rightscale_userdata
 - scripts-vendor
 - scripts-per-once
 - scripts-per-boot
 - scripts-per-instance
 - scripts-user
 - ssh-authkey-fingerprints
 - keys-to-console
 - phone-home
 - final-message
 - power-state-change

# System and/or distro specific settings
# (not accessible to handlers/transforms)
system_info:
   # This will affect which distro class gets used
   distro: fedora
   # Default user name + that default users groups (if added/used)
   default_user:
     name: root
     lock_passwd: false
     shell: /bin/bash
   # Other config here will be given to the distro class and/or path classes
   paths:
      cloud_dir: /var/lib/cloud/
      templates_dir: /etc/cloud/templates/
   ssh_svcname: sshd

$ cat /etc/cloud/cloud.cfg.d/92-hetznercloud-ds.cfg 
# Hetzner cloud-init defaults
# This file was written during the Hetzner image generation process

datasource_list: [ Hetzner, None ]

Environment details

• Cloud-init version: cloud-init-24.2-3.fc41.noarch
• Operating System Distribution: Fedora 41
• Cloud provider, platform or installer type: Hetzner cloud

cloud-init logs

2025-01-10 11:05:23,189 - subp.py[DEBUG]: Running command ['ip', '--json', 'addr'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,200 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/hetzner/v1/metadata/instance-id' with {'url': 'http://169.254.169.254/hetzner/v1/metadata/instance-id', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 5.0, 'headers': {'User-Agent': 'Cloud-Init/24.2'}} configuration
2025-01-10 11:05:23,207 - distros[DEBUG]: DHCP client selected: dhcpcd
2025-01-10 11:05:23,208 - dhcp.py[DEBUG]: Performing a dhcp discovery on ens3
2025-01-10 11:05:23,208 - subp.py[DEBUG]: Running command ['ip', 'link', 'set', 'dev', 'ens3', 'up'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,215 - util.py[DEBUG]: Reading from /sys/class/net/ens3/type (quiet=False)
2025-01-10 11:05:23,215 - util.py[DEBUG]: Read 2 bytes from /sys/class/net/ens3/type
2025-01-10 11:05:23,215 - subp.py[DEBUG]: Running command ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3'] with allowed return codes [0] (shell=False, capture=True)
2025-01-10 11:05:23,433 - subp.py[DEBUG]: ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3'] took 0.2s to run
2025-01-10 11:05:23,433 - dhcp.py[DEBUG]: dhcpcd exited with code: 1 stderr: 'dhcpcd-10.0.8 starting\n        DUID 00:01:00:01:2f:12:a1:d3:96:00:02:9f:7f:6e\n        ens3: IAID 02:9f:7f:6e\n        dhcpcd_fork_cb: dhcpcd manager hungup' stdout: ''
2025-01-10 11:05:23,433 - DataSourceHetzner.py[ERROR]: Bailing, DHCP Exception: 
2025-01-10 11:05:23,433 - handlers.py[DEBUG]: finish: init-local/search-Hetzner: FAIL: no local data found from DataSourceHetzner
2025-01-10 11:05:23,433 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed
2025-01-10 11:05:23,433 - util.py[DEBUG]: Getting data from <class 'cloudinit.sources.DataSourceHetzner.DataSourceHetzner'> failed
Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 666, in dhcp_discovery
    out, err = subp.subp(
               ~~~~~~~~~^
        command,
        ^^^^^^^^
        timeout=self.timeout,
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/cloudinit/subp.py", line 295, in subp
    raise ProcessExecutionError(
        stdout=out, stderr=err, exit_code=rc, cmd=args
    )
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['/usr/sbin/dhcpcd', '--ipv4only', '--waitip', '--persistent', '--noarp', '--script=/bin/true', 'ens3']
Exit code: 1
Reason: -
Stdout: 
Stderr: dhcpcd-10.0.8 starting
        DUID 00:01:00:01:2f:12:a1:d3:96:00:02:9f:7f:6e
        ens3: IAID 02:9f:7f:6e
        dhcpcd_fork_cb: dhcpcd manager hungup

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 1053, in find_source
    if s.update_metadata_if_supported(
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        [EventType.BOOT_NEW_INSTANCE]
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ):
    ^
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 929, in update_metadata_if_supported
    result = self.get_data()
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 485, in get_data
    return_value = self._check_and_get_data()
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/__init__.py", line 417, in _check_and_get_data
    return self._get_data()
           ~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/cloudinit/sources/DataSourceHetzner.py", line 60, in _get_data
    with EphemeralDHCPv4(
    ...<17 lines>...
        )
  File "/usr/lib/python3.13/site-packages/cloudinit/net/ephemeral.py", line 306, in __enter__
    return self.obtain_lease()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/site-packages/cloudinit/net/ephemeral.py", line 329, in obtain_lease
    self.lease = maybe_perform_dhcp_discovery(
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.distro, self.iface, self.dhcp_log_func
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 105, in maybe_perform_dhcp_discovery
    return distro.dhcp_client.dhcp_discovery(interface, dhcp_log_func, distro)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/site-packages/cloudinit/net/dhcp.py", line 733, in dhcp_discovery
    raise NoDHCPLeaseError from error
cloudinit.net.dhcp.NoDHCPLeaseError