Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deprecation version boundary #5411

Merged
merged 3 commits into from
Jun 27, 2024

Conversation

holmanb
Copy link
Member

@holmanb holmanb commented Jun 18, 2024

Additional Context

Add deprecation version knob for stable release distros. This allows distros to set an "initial release" version for the cloud-init release on a stable distro, and any newly added deprecation in future releases should log at an INFO level.

with the following cloud-config:

#cloud-config
grub-dpkg:
  enabled:
    false
users:
  - name: osadmin
    lock-passwd: false
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    ssh-authorized-keys:
      - ssh-rsa "AAAAB3NzaC1yc2EAAAADAQABAAABAQCc/K1T62elOkJKc94sWovn06RE7D274Fxx5n+eT/oClAwk7632awMtWjO+lIRpzOlEcYCR6wexlmL9mZNEFJoROMMRyCxRilNAiDKOq0X5j1HXY2ky9KjtLxo2b8yXFRBcYKK4W8aHU94NkNek81jdi9lC+L6jUrZ25d7xylqnEbd4TgrBphiowdh5B3RY+j6ePthLWWhHopRcVDgq91yYacelCCIfSDaGBya9iXPZSwoKtPin4n5PdSGmSOA3fIkvV5JjLS1QbuY5tzhXKuH9MdAqrvutF1bprs/GDf0IY8DO7DDoo8rXjhhgGtCdg1i5UitZIAM3hn/Vk5F//5eJ JOOS8312@EB-OR6120158"

this PR yields the following schema annotate output:

# cloud-init schema --system --annotate
Found cloud-config data types: user-data, network-config

1. user-data at /var/lib/cloud/instances/cb88d1af-f827-4381-bc89-4fad787bcaaf/cloud-config.txt:
#cloud-config

# from 1 files
# part-001

---
grub-dpkg:		# D1
    enabled: false
users:
-   lock-passwd: false		# D2
    name: osadmin
    ssh-authorized-keys:		# D3
    - ssh-rsa "AAAAB3NzaC1yc2EAAAADAQABAAABAQCc/K1T62elOkJKc94sWovn06RE7D274Fxx5n+eT/oClAwk7632awMtWjO+lIRpzOlEcYCR6wexlmL9mZNEFJoROMMRyCxRilNAiDKOq0X5j1HXY2ky9KjtLxo2b8yXFRBcYKK4W8aHU94NkNek81jdi9lC+L6jUrZ25d7xylqnEbd4TgrBphiowdh5B3RY+j6ePthLWWhHopRcVDgq91yYacelCCIfSDaGBya9iXPZSwoKtPin4n5PdSGmSOA3fIkvV5JjLS1QbuY5tzhXKuH9MdAqrvutF1bprs/GDf0IY8DO7DDoo8rXjhhgGtCdg1i5UitZIAM3hn/Vk5F//5eJ
        JOOS8312@EB-OR6120158"
    sudo:
    - ALL=(ALL) NOPASSWD:ALL
...

# Deprecations: -------------
# D1: An alias for ``grub_dpkg`` Deprecated in version 22.2. Use ``grub_dpkg`` instead.
# D2: Default: ``true`` Deprecated in version 22.3. Use ``lock_passwd`` instead.
# D3:  Deprecated in version 18.3. Use ``ssh_authorized_keys`` instead.


  Valid schema user-data

2. network-config at /var/lib/cloud/instances/cb88d1af-f827-4381-bc89-4fad787bcaaf/network-config.json:
  Valid schema network-config

and the deprecated key use is logged at level DEPRECATED:

# cloud-init status  --long
status: done
extended_status: degraded done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:04 +0000
detail: DataSourceLXD
errors: []
recoverable_errors:
DEPRECATED:
	- Deprecated cloud-config provided:grub-dpkg: An alias for ``grub_dpkg`` Deprecated in version 22.2. Use ``grub_dpkg`` instead., users.0.lock-passwd: Default: ``true`` Deprecated in version 22.3. Use ``lock_passwd`` instead., users.0.ssh-authorized-keys:  Deprecated in version 18.3. Use ``ssh_authorized_keys`` instead.

when DEPRECATION_INFO_BOUNDARY is set to 22.2

diff --git a/cloudinit/features.py b/cloudinit/features.py
index 9d88e138c..7f6900969 100644
--- a/cloudinit/features.py
+++ b/cloudinit/features.py
@@ -87,7 +87,7 @@ On Debian and Ubuntu systems, cc_apt_configure will write a deb822 compatible
 to write /etc/apt/sources.list directly.
 """
 
-DEPRECATION_INFO_BOUNDARY = "devel"
+DEPRECATION_INFO_BOUNDARY = "22.2"
 """
 DEPRECATION_INFO_BOUNDARY is used by distros to configure at which version to
 start logging deprecations at a level higher than INFO.

all deprecations are still reported

root@me:~# cloud-init schema --annotate --system
Found cloud-config data types: user-data, network-config

1. user-data at /var/lib/cloud/instances/5362d6fe-c68b-4e53-9920-49795eac9c28/cloud-config.txt:
#cloud-config

# from 1 files
# part-001

---
grub-dpkg:		# D1
    enabled: false
users:
-   lock-passwd: false		# D2
    name: osadmin
    ssh-authorized-keys:		# D3
    - ssh-rsa "AAAAB3NzaC1yc2EAAAADAQABAAABAQCc/K1T62elOkJKc94sWovn06RE7D274Fxx5n+eT/oClAwk7632awMtWjO+lIRpzOlEcYCR6wexlmL9mZNEFJoROMMRyCxRilNAiDKOq0X5j1HXY2ky9KjtLxo2b8yXFRBcYKK4W8aHU94NkNek81jdi9lC+L6jUrZ25d7xylqnEbd4TgrBphiowdh5B3RY+j6ePthLWWhHopRcVDgq91yYacelCCIfSDaGBya9iXPZSwoKtPin4n5PdSGmSOA3fIkvV5JjLS1QbuY5tzhXKuH9MdAqrvutF1bprs/GDf0IY8DO7DDoo8rXjhhgGtCdg1i5UitZIAM3hn/Vk5F//5eJ
        JOOS8312@EB-OR6120158"
    sudo:
    - ALL=(ALL) NOPASSWD:ALL
...

# Deprecations: -------------
# D1: An alias for ``grub_dpkg`` Deprecated in version 22.2. Use ``grub_dpkg`` instead.
# D2: Default: ``true`` Deprecated in version 22.3. Use ``lock_passwd`` instead.
# D3:  Deprecated in version 18.3. Use ``ssh_authorized_keys`` instead.


  Valid schema user-data

2. network-config at /var/lib/cloud/instances/5362d6fe-c68b-4e53-9920-49795eac9c28/network-config.json:
  Valid schema network-config

however only older deprecations than this version are logged at a DEPRECATED level:

root@me:~# cloud-init status --long
status: done
extended_status: degraded done
boot_status_code: enabled-by-generator
last_update: Thu, 01 Jan 1970 00:00:03 +0000
detail: DataSourceLXD
errors: []
recoverable_errors:
DEPRECATED:
	- Deprecated cloud-config provided:grub-dpkg: An alias for ``grub_dpkg`` Deprecated in version 22.2. Use ``grub_dpkg`` instead.

nice to have:

  • log the release boundary
  • user configurable release boundary

Test Steps

Checklist

Merge type

  • Squash merge using "Proposed Commit Message"
  • Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

Copy link
Member

@TheRealFalcon TheRealFalcon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting the bikeshed 😉

Looks good so far though

cloudinit/settings.py Outdated Show resolved Hide resolved
cloudinit/util.py Show resolved Hide resolved
cloudinit/settings.py Outdated Show resolved Hide resolved
@holmanb holmanb force-pushed the holmanb/deprecation-version-boundary branch 2 times, most recently from 7b41c13 to 286994a Compare June 19, 2024 23:06
@holmanb holmanb changed the title wip deprecation version boundary Jun 26, 2024
@holmanb holmanb force-pushed the holmanb/deprecation-version-boundary branch from 286994a to 760c85f Compare June 26, 2024 23:27
@holmanb holmanb marked this pull request as ready for review June 27, 2024 00:01
Copy link
Collaborator

@blackboxsw blackboxsw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good some unittest fixes to mock the feature DEPRECATION_INFO_BOUNDARY but otherwise +1

  • Would like to see a unit test that exercise a DEPRECATION_INFO_BOUNDARY of something != "devel" to provide examples to other downstream of what version strings appear acceptable and the behavior expected in those cases.

start logging deprecations at a level higher than INFO.

The default value "devel" tells cloud-init to log all deprecations higher
than INFO. This value may be overriden by downstreams in order to maintain
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
than INFO. This value may be overriden by downstreams in order to maintain
than INFO. This value may be overriden by downstreams in order to maintain

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the difference between the suggestion and current values?

DEPRECATION_INFO_BOUNDARY = "devel"
"""
DEPRECATION_INFO_BOUNDARY is used by distros to configure at which version to
start logging deprecations at a level higher than INFO.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to mention the logs greater than INFO level will generate an exit(2) from cloud-init status


<value> :: = <default> | <version>
<default> ::= "devel"
<version> ::= <major> "." <minor> "." <patch>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patch is optional. Maybe document that aspect? We may want to even limit this to just major.minor as some originally published downstream stable versions of cloud-init may contain versions such as 22.1-14-g2e17a0d6-0ubuntu1~22.04.5. We definitely don't want anything beyond digits and periods.

Suggested change
<version> ::= <major> "." <minor> "." <patch>
<version> ::= <major> "." <minor> [ "." <patch> ]
Suggested change
<version> ::= <major> "." <minor> "." <patch>
<version> ::= <digit> "." <digit> [ "." <digit> ]

Now that we are accepting a slightly more complex schema than booleans in features.py, I bet if we get more complex values for other 'features', some sort of schema validation of the configured values could be helpful to avoid the potential of downstreams. It's definitely not worth the investment on this PR, but a discussion topic we could bring up if we get more complex configurable feature values.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patch is optional. Maybe document that aspect?

+1 Good suggestion

We may want to even limit this to just major.minor as some originally published downstream stable versions of cloud-init may contain versions such as 22.1-14-g2e17a0d6-0ubuntu1~22.04.5

This version doesn't match the documented pattern, and even if a downstream ignores the documentation, Version.from_str() won't accept such a format so they would see the following exception in any tests that they run

>>> util.Version.from_str("24.1-0")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/holmanb/ci-a/cloudinit/util.py", line 3166, in from_str
    return cls(*(list(map(int, version.split(".")))))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: invalid literal for int() with base 10: '1-0'

Also, lets say a downstream wants to base their initial release on patch version 24.1.13 and wants to include a deprecation that was added in 24.1.12. Limiting to minor-only versions unnecessarily disallows that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we are accepting a slightly more complex schema than booleans in features.py, I bet if we get more complex values for other 'features', some sort of schema validation of the configured values could be helpful to avoid the potential of downstreams. It's definitely not worth the investment on this PR, but a discussion topic we could bring up if we get more complex configurable feature values.

Eh, maybe. Personally I think that trying to prevent errors introduced by downstreams or third-parties in upstream code is a lot lower priority than validating user-introduced errors and documenting the possible cloud.cfg values.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patch is optional. Maybe document that aspect?

+1 Good suggestion

FWIW minor is also optional, but I don't think that is worth documenting since it's not likely to be used or useful.

Version.from_str(features.DEPRECATION_INFO_BOUNDARY)
):
info_deprecations.append(
SchemaProblem(path, schema_error.message)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for my info, what's the perspective on dropping the local var declaration 'problem =' which can be used in both info_deprecations and errors append operations? Just to aid readability at the append call site?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original code created a tuple with a single element and then appended this single element to a list:

problem = (SchemaProblem(path, schema_error.message),)
...
deprecations += problem

which takes some extra mental gymnastics to realize the end goal is to append to a list

deprecations.append(SchemaProblem(path, schema_error.message))

I consider the legibility difference between:

if <blah>:
    info_deprecations.append(SchemaProblem(path, schema_error.message))
else:
    deprecations.append(SchemaProblem(path, schema_error.message))

and

problem = SchemaProblem(path, schema_error.message)
if <blah>:
    info_deprecations.append(problem)
else:
    deprecations.append(problem)

to be negligible. The second has less total calls to grok, but the first has an extra variable to track, and since the calls are simple and identical it's quick realize that they do the same thing.

what's the perspective on dropping the local var declaration 'problem ='

It's really just a style preference. When I find differences like this negligible I typically lean towards the option that involves assigning fewer variables for a few of reasons:

  1. less state / context to track -> easier to mentally follow
  2. easier to refactor into simpler / cleaner code in the future
  3. less variable names that I need to think up (obviously doesn't apply in this case because pre-existing code)
  4. less lines of code to read / maintain

While these reasons reasons seem bit trivial, the impacts add up as code size and complexity grows.

I suppose I could have done this PR without making this change, but the actual reason that this happened due to an earlier iteration subclassing a different type and having to move the assignment within the if isinstance() call which necessitated modifying this variable anyways.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good intent/reasoning here thanks.

@holmanb
Copy link
Member Author

holmanb commented Jun 27, 2024

@blackboxsw I just need to add a unittest for this new behavior and then I think it's ready

@blackboxsw
Copy link
Collaborator

@blackboxsw I just need to add a unittest for this new behavior and then I think it's ready

Since I added the request for more testing, I figured I'd setup a couple of minor tests to exercise the boundary to get better context on behavior here. Review away, if you like the tests, let's keep em, or of something needs to change go for it.

@blackboxsw blackboxsw force-pushed the holmanb/deprecation-version-boundary branch from a30b048 to 01bc508 Compare June 27, 2024 18:17
tests/unittests/test_log.py Outdated Show resolved Hide resolved
holmanb and others added 3 commits June 27, 2024 14:53
This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.
…#5411)

This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.
@holmanb holmanb force-pushed the holmanb/deprecation-version-boundary branch from 048551b to 7bb1a72 Compare June 27, 2024 20:55
@holmanb holmanb merged commit 7f98af9 into canonical:main Jun 27, 2024
23 checks passed
holmanb added a commit that referenced this pull request Jun 27, 2024
This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.
holmanb added a commit that referenced this pull request Jun 27, 2024
This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.
holmanb added a commit that referenced this pull request Jun 28, 2024
This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.
holmanb added a commit that referenced this pull request Jun 28, 2024
This will all stable distros to define whether a key is deprecated
based on the version of cloud-init which deprecated the key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants