patches:
debian/patches/expire-on-hashed-users.patch

In the dpkg-maintscript-helper manpage:
If the conffile has not been shipped for several versions, and you are
now modifying the maintainer scripts to clean up the obsolete file,
prior-version should be based on the version of the package that you
are now preparing, not the first version of the package that lacked the
conffile

Commit cda82fe warranted downstream patch refresh to
fix daily recipe builds against main.

patches:
debian/patches/retain-old-groups.patch

Add Akamai to the default templates to allow datasource discovery.

Co-authored-by: Chad Smith <[email protected]>

Co-authored-by: Chad Smith <[email protected]>

cc_apt_configure will use apt_pkg from this deb if present to
read apt configuration for Dir::Etc::sourceparts/sourcelist values.

When absent, rely on the command apt-config dump.

Commit 7ffd557 unintentionally redacted the second part of the
changeset moving Before=systemd-user-sessions.service from
cloud-config.service back into cloud-init.service.

Replace the dropped patchset.

* azure: check for primary interface when performing DHCP

For Savable PPS we rely on a connectivity check to IMDS to determine
if the NIC is primary.  However, in some cases, connectivity may be
delayed and we incorrectly assume the NIC is not primary.

Instead of relying on connectivity, check the DHCP-provided route
configuration for the presence of Wireserver and/or IMDS IPs.

- Return bool from _setup_ephemeral_networking() indicating primary
NIC or not.

- Remove _check_if_nic_is_primary() used in Savable PPS.

- Add a relevant diagnostic to assist debugging failures when a
secondary NIC is chosen.

- Remove some tests that are redundant.

Signed-off-by: Chris Patterson <[email protected]>

…nical#4553)

By default, cloud-init schema command will look for processed network
config at /run/cloud-init/network-config.json and report any
schema validation errors on the commandline.

To suport this feature:
- return False when validate_cloudconfig_file does not validate a
  file
- skip valdation on either empty network-config schema or network
  version 2
- add a new --schema-type param to schema subcommand to allow for
  providing source network-config YAML files and validate as
  network-config instead of cloud-config
- Add schema_type parameter to validate_cloudconfig_file allowing for
  plumbing 'network-config' type to schema validation call-sites
- distill messaging to report config type and cfg files. Avoid
  classifying network-config as 'cloud-config', avoid lowercase yaml
- When --config-file does not provide known header, label this cfg_type
  as UNKNOWN_CONFIG_HEADER for messaging

Additionally, drop unnecessary jsonschema.ValidationError imports
as that is already done at the module level.

…al#4482)

When reading netcfg from the environment, validate version 1 against
known JSON network-config schema and warnings on invalid schema.

Persist the processed network-config at
/run/cloud-init/network-config.json as a symlink to
/var/lib/cloud/instance/network-config.json.

…cal#4482)

Plug cloud_init_deppest_matches into any oneOf subschema calculation
for most likely schema error match.

This function will prefer the deepest match into the json_schema path
with the exception that it prefers error messages for the matching
subschema error for any instance containing a 'type' key.

This means, when the subschema defines a required type: key. The
cloud_init_deepest_match will raise the matching errors for the
specific subschema containing the matching type: XXX over any
other errors.

Also drop unhelpful shema validation errors:
  'blah' is not valid under any of the given schemas.

This is replaced with specific validation errors for each unmatched
subschema.

…cal#4482)

network-config schema now permits optional top-level network key.

We should generally be using oneOf declarations in JSON schema.
Fix anyOf_type_route schema definition.

Update route definition to oneOf from anyOf requiring either
'network' or 'destination' key, combined with a 'gateway' key.

After pytest-dev/pytest#11125 being included in
pytest==7.4, pytest can be unpinned.

Make use of new QEMU support in pycloudlib and bump pycloudlib
accordingly.

Set kernel commandline network configuration source to have precedence over
initramfs config in init-local timeframe.

…anonical#4556)

In test_logs.py, test_collect_logs_includes_optional_userdata() would leak the
journalctl subprocess call because it did not mock subprocess.call() like
test_collect_logs_creates_tarfile() does. This would cause the unit test to
take a long time to run because it was calling the actual journalctl command.

Fixes canonicalGH-4536

Provide an unauthenticated view of meeting notes for those without
google accounts.  Original meeting notes live in a shared google
drive document. To provide a point of reference for those in the
community without google accounts we have refactored the notes to
markdown and will host them in the cloud-init repository so that
both discourse trip reports and mailinglist threads can reference
these notes without have to bloat the email or discourse post with
details.

Co-authored-by: Sally Makin <[email protected]>

Fixes canonicalGH-3016

…anonical#4468)

When retrying to fetch metadata from IMDS there is a backoff for
logging to avoid flooding the logs.  In the current implementation
we have to wait until the next backoff log to see the transition.

Always log when the current error does not match the previous one.

Signed-off-by: Chris Patterson <[email protected]>

Also moved images out of the repo (including the ones from previous
years) and added to the Ubuntu asset manager for hosting to avoid
bloating the repo as time goes on!

…cal#4564)

Fixes canonicalGH-4487

Commit f780cf9 removed the modules-init key from status.json v1 key.
Don't use it as example test data.

This key had more meaning to a developer than to a user. Replace with
"recoverable_errors", and align internal variable names with external
user UI for code legibility.

If different meaning for duplicate keys is required, then a v2 can be
added. Drop versioning scheme and duplicate keys to reduce unnecessary
verbosity.

BREAKING CHANGE: cloud-init status --json output

…l#4500)

The detail in this key is duplicate, and changing the value of this key
during error condition is neither obvious nor documented. Make this key
behave the same regardless of error condition.

BREAKING CHANGE: status.json

Summary
=======
This commit `cloud-init status` to include:

  1. A new exit code (2)
  2. Additional running states, exported under a new key "extended_status"
  3. External representation of all internal errors:
    - aggregate recoverable errors
    - per-stage recoverable errors
    - per-stage non-recoverable errors (aggregate key already exists)

Current state: recoverable errors vs non-recoverable errors
===========================================================

critical failure
----------------
If cloud-init is unable to complete, the service returns with exit code
1, and error messages are visible in the log files and in output of
`cloud-init status --format json` under the top level 'error' key.

recoverable failure
-------------------
In the case that cloud-init is able to complete yet something goes awry,
the service returns with exit code 0 and messages are visible in the log
files.

Future state: recoverable errors vs non-recoverable errors
==========================================================

critical failure
----------------
If cloud-init is unable to complete, error messages will now
additionally be visible in output of `cloud-init status --format json`
within the 'error' key nested under the module-level keys: 'init-local',
'init', 'modules-config', 'modules-final'.

recoverable failure
-------------------
In the case that cloud-init is able to complete yet something goes awry,
the service will now return with exit code 2, and error messages will be
visible in the output of `cloud-init status --format` json under the top
level 'recoverable_errors' key as well as within the 'error' key nested
under the module-level keys: 'init-local', 'init', 'modules-config',
'modules-final'.

Implementation
==============

Cloud-init error codes
----------------------
 0 - success
 1 - unrecoverable error
 2 - recoverable error (new)

This new exit code indicates recoverable errors. If cloud-init exits
with exit code (2), cloud-init was able to complete gracefully, however
something went wrong and the user should investigate.

Additional states
-----------------
For backwards compatibility, the output of `cloud-init status` remains
unchanged. A new key 'extended_status' is included in the output:

$ cloud-init status --format json | jq .status
"done"

$ cloud-init status --format json | jq .extended_status
"degraded done"

See Appendix A for list of possible states.

Exported errors: Aggregated errors
----------------------------------
When a recoverable error occurs, the internal cloud-init state
information is made visible under a top level aggregate key
'recoverable_errors' with errors sorted by error level:

$ cloud-init status --format json | jq .recoverable_errors
{
  "WARNING": [
    "Failed at merging in cloud config part from part-001: empty cloud config",
    "No template found in /etc/cloud/templates for template named sources.list.ubuntu.deb822",
    "No template found in /etc/cloud/templates for template named sources.list",
    "No template found, not rendering /etc/apt/sources.list.d/ubuntu.sources"
  ]
}

See Appendix B for list of possible error levels.

Exported errors: Per-stage errors
---------------------------------
The keys 'errors' and 'recoverable_errors' are also exported for each
stage to allow attribution of recoverable and non-recoverable errors
to their source.

$ cloud-init status --format json | jq .init.recoverable_errors
{
  "WARNING": [
    "Failed at merging in cloud config part from part-001: empty cloud config"
  ]
}

Note: Only cloud-init stages which have completed are listed in the
output of `cloud-init status --format json`.

See Appendix C for list of possible cloud-init stages.

Limitations of internal errors
==============================
- Exported recoverable errors represent logged messages, which are not
  guaranteed to be stable between releases. The contents of the
  'errors' and 'recoverable_errors' keys are not guaranteed to have
  stable output!
- Exported errors and recoverable errors may occur at different stages
  since users may reorder configuration modules to run at different
  stages via cloud.cfg.

Appendices
==========

Appendix A: Extended states
---------------------------
  "not running"
  "running"
  "done"
  "error"
  "degraded done"
  "degraded running"
  "disabled"

Appendix B: Error levels
------------------------
Reported recoverable error messages are grouped by the level at which
they are logged. Complete list of levels:

  WARNING
  DEPRECATED
  ERROR
  CRITICAL

Appendix C: Stages of cloud-init
--------------------------------
The json representation of cloud-init stages (in run order) is:

  "init-local"
  "init"
  "modules-config"
  "modules-final"

This commit implements design specification US057[1].

[1] https://discourse.ubuntu.com/t/spec-improve-error-and-warning-visibility/39765

patches:
debian/patches/do-not-block-user-login.patch

patches:
debian/patches/status-do-not-remove-duplicated-data.patch

This patch was inadvertently dropped in 5d4a3cf.

   - Disable apt source list generation with DEB822 style

fix: Handle systemctl commands when dbus not ready

During `cloud-init status`, we check systemctl to ensure the status
we're reporting is accurate. However, we can get an error from systemctl
if dbus isn't ready yet. This commit will either ignore the error if we
can assume that cloud-init is still running, or retry until we get a
proper response from systemctl.

Fixes canonicalGH-4676

Bump the version in cloudinit/version.py to 23.4.1 and update ChangeLog.

patches:
debian/patches/retain-apt-pre-deb822.patch
debian/patches/status-do-not-remove-duplicated-data.patch

Retain exit code in cloud-init status for recoverable errors.
(LP: #2048522).

)

Fixes: canonicalGH-4760

Bump the version in cloudinit/version.py to 23.4.2 and update ChangeLog.

patches:
debian/patches/retain-netplan-world-readable.patch
debian/patches/status-do-not-remove-duplicated-data.patch
debian/patches/status-retain-recoverable-error-exit-code.patch

netplan 106.1-7 requires that netplan config parts are root-readonly.

Any files with more open permissions will generate warnings on the
netplan cli.

This change in behavior was introduced in netplan.io to
mitigate some security concerns where netplan files could contain
sensitive information.

Align with this security guidance and ensure cloud-init writes
/etc/netplan/50-cloud-init.yaml as root 600 instea of world-readable.

LP: #2053157

…nonical#4727)

If a cloud or user sets create_hostname_file to false (as GCP does)
the examples in our docs for setting the hostname will not work.
This change updates those examples to also set create_hostname_file
to true.

In bugs such as LP: #2046483, cloud-init status --wait produces
a traceback early when dbus or other resources are not yet available
resulting in an early unblock from status --wait.

Add an integration test inserting a systemd unit before
cloud-init-local.service that will block until cloud-init is done.

Assert that the timestamp of marker file created when this test unit
unblocks, occurs after cloud-init.target was reached.

This reverts commit 9c6bbc3.

Add ProvisionGuestProxyAgent Boolean configuration setting into the OvfEnv class.
This PR is only logging the value of ProvisionGuestProxyAgent.