The /etc/hostname.* files should have the mtu on
a separate line otherwise it gives error:

  ifconfig: mtu: bad value

The lines are executed in order by ifconfig and
mtu should be on it's own line.

Fixes: canonicalGH-5413

)

When there is no IPv6 addr given in the customization configuration,
we shall set IPv6 type to dhcp6, then customized Linux network will be
set to dhcp IPv6 explicitly.

* Remove `unittest` constructs and remove base classes.
* Replace tests that don't test things with tests that do
* Add fstab and mounts combinations test

canonicalGH-5445

There are no call sites requesting not decoding the environment vars.
This change decodes then always, simplifying typing and logic.

Instead of a broad try/except, do properly check for conditions that
invalidate a mount location.

Support for jumbo frames requires that the underlying physical interfaces
and the parent bond interface all have the larger MTU configured, not just
the physical interfaces.

…canonical#5501)

At least one of (or both) 'baseurl' or 'metalink' should be provided for yum
repository specification. Add schema changes to enforce it. Without this,
with just 'metalink' property set, one would get the schema validator error

\---
Error: Cloud config schema errors: yum_repos.epel-release: 'baseurl' is a required property
\---

Signed-off-by: Ani Sinha <[email protected]>

On systemd, services are started by PID 1. When this doesn't happen, cloud-init
is in an unknown run state and should warn the user.

Reorder pid log to be able to reuse Distro information.

Add docstring deprecating util.is_Linux().

…5209)

Implement verify_clean_boot() to ignore certain expected logs
in a platform-specific way.

…anonical#5209)

Ensure ignore_warnings=True or ignore_errors=True is honored and
not overridden by supplemental warning texts appended.

…l#5209)

Avoid using warning level messages as there may be some
use-cases in the wild that need to invoke cloud-init boot
stages after boot for some reason unknown to upstream.

Provide a detailed warning message informing admins to file
issues against cloud-init to better represent those feature
needs before dropping this feature altogether.

…onical#5504)

Commit 604d80e introduced assertions expecting exit 2 from the
CLI when calling cloud-init init --local. Revert this test assertion
as only cloud-init status command exits (2) on deprecations/warnings.

Invoking cloud-init's boot stages on the commmand line will only exit
1 if critical errors are encountered to avoid degrading overall
systemd health as seen from cloud-init systemd units. When cloud-init
boot stages encounter recoverable_errors of any type, there is no
need to exit non-zero as those deprecation logs are not-critical to
the health of the system as a whole.

It is pretty consistently failing due to canonical#5373 with no fix in
sight.

Ensure DNS server addresses are parsed from the proper location
of network_data.json

Fixes canonical#5386

Co-authored-by: Alberto Contreras <[email protected]>

If DNS information is added to a NetworkManager managed interface where
the given protocol family is disabled, NetworkManager will be unable to
activate the interface.

canonical#5387

'mirrorlist' config can be specified instead or along with 'baseurl' in the yum
repository config. Add support for specifying mirrorlist instead of 'baseurl'.

Fixes canonicalGH-5520
Signed-off-by: Ani Sinha <[email protected]>

9929a00 added the ability to used a cached datasource when none is
found. This was supposed to be per-datasource, but the lack of cache
cleaning got applied universally. This commit makes it so cache will be
cleaned as it was before if fallback isn't implemented in datasource.

Fixes canonicalGH-5486

This is useful for logs we want hidden by default but can be turned
on via configuration.

Switch to pathlib where appropriate and call consistently

…5515)

With this change, the following config in cloud.cfg.d/ will select NoCloud in
network stage.

```
datasource_list: [ GCE, NoCloud, None ]
datasource:
  NoCloud:
    seedfrom: http://0.0.0.0:8000/
```

Previously a two or less datasources in the datasource_list were required to
get this behavior, which was undocumented and not intuitive.

The ds-identify already allowed inline user-data and meta-data to
trigger detection.

Add ds-identify unittests for seedfrom and inline user-data.
Add DataSourceNoCloud.ds_detect() unittests for seedfrom and inline
user-data.

The nocloud datasource logs messages that are sometimes confused by users
for errors. Clarify them.

Also, remove redundant information from the logs:

- simplify log wording
- only include seed and dsmode information in nocloud string when
  non-default values are used

ds-identify does not support the fs_label key. This key is only partially
supported.

Deprecate it. Users of custom labels may switch to cidata or CIDATA.

Note: Tools such as cloud-localds hard-code the label.

)

UP4W business logic is so that its data overrides user at a key (module) 
level.
That means the entire Landscape config is overriden if both agent data
and user data contains config for that module.
Yet, for better usability, computer tags must be assignable per instance.
That's not possible with agent.yaml, because it's meant to be global.
Its config data affects all Ubuntu WSL instances.

Thus this aims to make a special case for landscape.client.tags,
if present in user provided data (either Landscape or local user -
  whatever is picked up before merging with agent.yaml)
its value overwrites any tags set by agent.yaml.

Only landscape.client.tags are treated specially.
The pre-existing merge rules still apply for any other value present in
both agent.yaml and user provided data.

Fixes UDENG-2464

Commit 441d8f8 adds openeuler to the list of supported distros in
cc_spacewalk.py, but there is one omission.

This change adds an optional `source` key to the `write_files` module,
allowing users to specify a URI from which to load file contents. This
facilitates more flexible multi-part configurations, as file contents
can be managed via external sources such as independent Git
repositories.

Fixes canonicalGH-5500

Change default usr_lib_exec from /usr/libexec/ to /usr/lib
as azurelinux installs cloud-init tool binaries under
/usr/lib/cloud-int instead of /usr/libexec/cloud-init

After 7703634 log messages migrated
from:

"Detected platform DataSource..." to "Detected DataSource..."

Adapt integration tests accordingly.

Adapt to conform with 7703634

…nonical#5537)

Got swallowed by
https://github.com/canonical/cloud-init/pull/5116/commits/
da6b5c4

The former commit resulted in usage error from the wslpath
command thus we never found WSL specific data,
disabling cloud-init.

Create explicit file to define and pin the actions/labeler's version.

Upgrade breaking changes from action/labeler@v5.

The suggested qemu command line in our local execution example
is rather old.

Change the discouraged -hd* options to the new
-device instead.

Further add a chance to use KVM acceleration to speed up the
example.

Finally we had several occasions to be working on x86 only.
We dropped arguments that can not work on cross-arch and
furthermore added a hint at how one could again native
performance on these platforms.

Fixes canonicalGH-5050

SC-1835
Fixes canonicalGH-5334

Fixes canonicalGH-5513

…ical#5543)

Improve explaination on #cloud-boothook for end-users.

SC-1657
Fixes canonicalGH-4542

Co-authored-by: Calvin Mwadime <[email protected]>

Fixes canonicalGH-5467

Without this explicit config option, an untracked and unused
`doc/rtd/spelling_wordlist.txt` file is created while running
`tox -e doc-spelling`.

…cal#5545)

…#5521)

Formally document providing runtime configuration in system configuration.
Introduce names to identify previously unnamed NoCloud concepts.
Add more structure - discrete sections for:
- runtime configuration types
- discovery configuration
- configuration sources

Add group of pages for drop-in custom modules and
restructure existing docs under it.

Add doc for custom datasources and config modules.
    
SC-1836
Fixes canonicalGH-4649

Fixes canonicalGH-4739

…nonical#5568)

Unspecified base match in labeler assumes 'any' for each match
clause. When specifying base-branch and --any-glob-to-any-file either
one of these cases would result in a successful match which would label
all PRs again main as documentation. We need to explicitly specify
'all:' in our labeler match config to ensure BOTH:

 * matching file paths related to documentation
         -AND-
 * targeting a merge against 'main' branch

…5569)

Both were version gated and do not run.

…cal#5570)

Also drop undesirable former doc-autolabel.yml workflow

When instance id hasn't changed and datasource hasn't changed, don't
forcibly reload the configuration.

This enables support for network config v2 and v1 to NoCloud
when used with http / ftp / etc. 

BREAKING_CHANGE: Adds an additional network request to NoCloud.

String output changed in 7703634.
Instance-id doesn't change on LXD / Focal.

The handle function of cc_mounts was hard to grok and had one of the
highest cyclomatic complexity scores in the codebase. Functionally,
the code should be unchanged.

…#5573)

Python interpreter initialization and module import time 
contributes a significant amount of wall clock time to
cloud-init's runtime (and therefore to total boot time).

Cloud-init has four stages. Each stage starts its own Python
interpreter and loads the same libraries. To eliminate the
redundant work of starting an interpreter and loading libraries,
this changes cloud-init to run as a single process. Systemd
service ordering is retained by using the existing cloud-init
services as shims which use a synchronization protocol to start
each cloud-init stage and to communicate that each stage is
complete to the init system. Since multiple cloud-init processes
sit in the critical chain of starting the system, this reduces
boot time (including time to ssh login and time to cloud-init
completion).

Currently only systemd is supported, but the synchronization
protocol should be capable of supporting other init systems
as well with minor changes.

Note: This enables many additional follow-on improvements that
eliminate redundant work. However, these potential improvements
are temporarily ignored. This commit has been structured to
minimize the changes required to capture the majority of primary
performance savings while preserving correctness and the ability
to preserve backwards compatibility.

Since this changes the semantics of the existing cloud-init unit
files, this change takes the opportunity to rename one of its
systemd units which causes frequent user confusion. The unit named
cloud-init.service is often mistaken by users for being the only
cloud-init service, when it is simply one of four stages. This
stage is documented as the "network" stage, so this service will
be renamed to "cloud-init-network.service". A new notify service
is added as part of this implementation which contains the
cloud-init process. This unit is named "cloud-init-main.service".
 
Synchronization protocol
========================

- create one Unix socket for each systemd service stage
- send sd_notify()
- For each of the four stages (local, network, config, final):
   - when init system sends "start" to the Unix socket, start the
     stage
   - when running stage is complete, send "done" to Unix socket

File changes
============

socket.py (new)
---------------

- define a systemd-notify helper function
- define a context manager which implements a multi-socket
  synchronization protocol

cloud-init.service -> cloud-init-network.service (renamed)
----------------------------------------------------------

- renamed to cloud-network.service

cloud-{init-local,init-network,config,final}.services
-------------------------------------------

- change ExecStart to use netcat to connect to Unix socket and:
  - send a start message
  - wait for completion response
- note: a pure Python equivalent is possible for any downstreams
  which do not package openbsd's netcat

cloud-init-main.service (new)
-----------------------------

 - use service type to 'notify'
 - invoke cloud-init in single process mode
 - adopt systemd ordering requirements from cloud-init-local.service
 - adopt KillMode from cloud-final.service

main.py
-------

 - Add command line flag to indicate "all stages" mode
 - In this mode run each stage followed by an IPC
   synchronization protocol step

cloud-final.services
--------------------

- drop KillMode

cloud-init-local.services
-------------------------

- drop dependencies made redundant by ordering after
  cloud-init-main.service

Performance Impact
==================

On Ubuntu 24.04, Python's wall clock start up time as measured with
`time python3 -c 'import cloudinit.cmd.main' on a few cloud types:

lxc container: 0.256s
QEMU machine:  0.300s
gce instance:  0.367s
ec2 instance:  0.491s

This change eliminates x1 this start up time from time to ssh.
This change eliminates x3 this start up time from cloud-init's total
completion. Total benefit varies based on the platform that the
instance is hosted by, but all platforms will measurably benefit from
this change.

BREAKING_CHANGE: Run all four cloud-init services as a single systemd service.

Commit 5322dca introduced an assumption to read_seeded that
network-config must always be present for NoCloud datasource.
Since it is still considered and optional supplemental configuration
allow the read_seeed calls to succeed in the absence of network-config.

Avoids failures seen in tests/integration-tests/datasources/test_nocloud.py::
  test_nocloud_seedfrom_vendordata

This was previously unnecessary because:

a. LXD automatically appends the user.meta-data key to default meta-data.
b. In the presence of duplicate keys, PyYAML uses the last key.

This change is the cloud-init part of a set of changes that will enable cloud-init
to avoid depending on undefined behavior. In the future LXD may stop
appending user-defined meta-data to its default meta-data. This change
makes cloud-init forward compatible to LXD for when that change is
implemented.

canonicalGH-5575

)

Fixes canonicalGH-5445

Azure Guest Proxy Agent is a new feature in Azure
that offers a key exchange protocol to secure
communication between guest and host using eBPF.

Add opt-in feature which enables the Azure Guest Proxy Agent
when ovf-env.xml has ProvisionGuestProxyAgent=True.
Report provisioning failures if ProvisionGuestProxyAgent is enabled
but images do not have azure-proxy-agent installed or functional.

…l#5556)

- fix exception handling when retr fails
- test: Close connection on failure
- test: Ensure server is running before it is queried

User output and service names recently changed.

…anonical#5426)

When snap refresh.hold is set to forever, an admin is saying they do
not want generic automated refreshes of snaps performed by default.

This should be an indicator to cloud-init to avoid calling snap refresh
on such systems due to a `package_upgrade: true` present in user-data.

For network-limited environments with images which have the snap package
manager but don't want to wait and timeout on snap refresh, the following
user-data can be provided to still allow for package_upgrade: true,
and avoid a 20-30 second wait on snaps being unable to access certain
snap URLs.

   #cloud-config
   package_upgrade: true
   snap:
     commands:
       00: snap refresh --hold=forever

cloud-init now interrogates the state refresh.hold value by calling
  snap get system -d

If snap refresh --hold was called in that environment to set 'forever',
cloud-init will skip calling refresh and log the reason for skipping.

We cannot honor short time-based refresh.holds because the snap
services place a short hold in early boot anyway as systemd units
startup.

Fixes: canonicalGH-5290

…#5593)

Commit 5322dca added network-config support to nocloud's
read_optional_seed function. It persisted meta-data as
network-config. Add tests and fix to track network-config value.

…onical#5471)" (canonical#5596)

This reverts commit 2b6fe64.

When there is no IPv6 set to dhcp explicitly, NetworkManager keyfile
defaults to method=auto, may-fail=true. When there is Ipv6 set to dhcp
explictily, NetworkManager keyfile will be set to
method=auto, may-fail=false. The default settings are what we want, so
revert the previous change to keep IPv6 not set explicitly.

…nonical#5598)

chore: add comment explaining the NetworkManager may-fail setting

The value of may-fail in network manager keyfile is a source of
confusion as the default value of it is True for Network Manager and
False for network manager renderer implementation. Add a comment to
explain why the renderer sets may-fail to False in its implementation.

Also shift the format page higher in the explanation page list, since
this is a high traffic page.

Patch diff was invalid representing a 57 line diff for cloud-init.service
in the diff header. But, the actual diff context was only 56 lines.

This broke the ability to apply all quilt patches with quilt push -a.

Resulting in broken daily build recipe for focal with the following
error message:
  patch: **** malformed patch at line 252:

  Patch no-single-process.patch does not apply (enforce with -f)

…canonical#5602)

Recently noticed that doc file changes in nested subdirs were
not triggering documentation auto label.

Example of subdir match at
https://github.com/actions/labeler?tab=readme-ov-file#basic-examples

Fixes canonicalGH-5614

)

As noted in the systemd documentation, /etc is reserved for "System
units created by the administrator" while the lib directory should be
used by "System units installed by the distribution package manager".

Fixes canonicalGH-5613

When referencing a command from another environment, it will cause
errors when the other environment already exists. Fix it by avoiding
indirection in environment command definitions.

Additionally, simplify envoronment dependency management by defining two
lists of dependencies: a default one with pinned versions for all
environments, and an unpinned on for "tip" environments. Several
dependencies have been missed in the mypy envornments, so this should
make it easier by standardizing environment dependencies to be
consistent across environments.