fix (openstack): Append interface / scope_id for IPv6 link-local metadata address #5419

frittentheke · 2024-06-20T16:06:57Z

Proposed Commit Message

 OpenStack: Append interface for IPv6 link-local metadata address

Link-Local IPv6 addresses (fe80::/10) without a scope_id or interface are invalid,
see [1]. This made urllib3 reject an URL of `http://[fe80::a9fe:a9fe]` as invalid
argument.
This change now appends an interface in the proper URL-encoded format ([2]).

[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/netinet_in.h.html
[2] https://datatracker.ietf.org/doc/html/rfc6874

Fixes GH-5422

Additional Context

Test Steps

Start VM on OpenStack with only IPv6.
#1805 (comment)

Checklist

My code follows the process laid out in the documentation
I have updated or added any unit tests accordingly
I have updated or added any documentation accordingly

Merge type

Squash merge using "Proposed Commit Message"
Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

holmanb

This requires an interface together with the address itself
to be valid.

Why?

Also, please sign the CLA and add your name to the signers list so that we can accept this contribution.

holmanb · 2024-06-20T19:03:10Z

cloudinit/sources/DataSourceOpenStack.py

 # Various defaults/constants...
-DEF_MD_URLS = ["http://[fe80::a9fe:a9fe]", "http://169.254.169.254"]
+DEF_MD_URLS = [
+    "http://[fe80::a9fe:a9fe%{iface}]".format(iface=net.find_fallback_nic()),


We don't want to call this function outside of a function definition. This would call net.find_fallback_nic() on all datasources (including non-openstack).

@holmanb do you like it better when I move the addresses to the only function they are used (and likely mangled)?

Should / could I also use self.distro.fallback_interface ?

@holmanb do you like it better when I move the addresses to the only function they are used (and likely mangled)?

Yes. This isn't referenced externally so that should be fine.

Should / could I also use self.distro.fallback_interface ?

Please do, that would be preferred.

frittentheke · 2024-06-20T19:40:04Z

Why?

Because urllib3 obviously does not accept a link-local address without zone identifier
This RFC vaguely implies this is required ... https://datatracker.ietf.org/doc/html/rfc4007#section-6
This post from RIPE does explains it nicely ... https://labs.ripe.net/author/philip_homburg/whats-the-deal-with-ipv6-link-local-addresses/

The Open Group Base Specifications Issue 7, 2018 edition, IEEE Std 1003.1-2017 (Revision of IEEE Std 1003.1-2008) has this to say about the sin6_scope_id field:
The sin6_scope_id field is a 32-bit integer that identifies a set of interfaces as appropriate for the scope of the address carried in the sin6_addr field. For a link scope sin6_addr, the application shall ensure that sin6_scope_id is a link index. For a site scope sin6_addr, the application shall ensure that sin6_scope_id is a site index. The mapping of sin6_scope_id to an interface or set of interfaces is implementation-defined.
In short, it is not possible to use an IPv6 link-local address in sockets API without selecting an appropriate interface as well. This is a significant departure from the past: knowing an IP address is no longer enough.

And it's documented by OpenStack https://github.com/openstack/neutron/blob/305e1451bbc980258d7f1accdbba83e5b08ab59e/releasenotes/notes/metadata-ipv6-b3607f6932da7226.yaml#L22

cloudinit/sources/DataSourceOpenStack.py

tomra · 2024-06-21T08:44:41Z

Hi @frittentheke thank you for your bug report and your PR.
I have tested your code and it works as expected.

Here the cloud-init.log output:

2024-06-21 08:35:51,344 - url_helper.py[DEBUG]: [0/1] open 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack' with {'url': 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 10.0, 'headers': {'User-Agent': 'Cloud-Init/24.1.3-0ubuntu1~22.04.1'}} configuration
2024-06-21 08:35:51,962 - url_helper.py[DEBUG]: Read from http://[fe80::a9fe:a9fe%25enp3s0]/openstack (200, 105b) after 1 attempts
2024-06-21 08:35:51,962 - DataSourceOpenStack.py[DEBUG]: Using metadata source: 'http://[fe80::a9fe:a9fe%25enp3s0]'
2024-06-21 08:35:52,977 - url_helper.py[DEBUG]: Read from http://[fe80::a9fe:a9fe%25enp3s0]/openstack/2018-08-27/vendor_data.json (200, 2b) after 1 attempts

@holmanb thanks for the code review.

holmanb

One minor nitpick, otherwise looks good!

cloudinit/sources/DataSourceOpenStack.py

Link-Local IPv6 addresses (fe80::/10) without a scope_id or interface are invalid, see [1]. This made urllib3 reject an URL of `http://[fe80::a9fe:a9fe]` as invalid argument. This change now appends an interface in the proper URL-encoded format ([2]). [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/netinet_in.h.html [2] https://datatracker.ietf.org/doc/html/rfc6874 Fixes canonicalGH-5422 Signed-off-by: Christian Rohmann <[email protected]>

frittentheke · 2024-06-21T19:45:21Z

One minor nitpick, otherwise looks good!

Fixed/removed @holmanb

frittentheke · 2024-06-24T19:06:13Z

Thanks for fixing the test @holmanb and for the quick review process!

holmanb

The code looks good to me. Thanks @frittentheke for this contribution!

…ata address (#5419) Link-Local IPv6 addresses (fe80::/10) without a scope_id or interface are invalid, see [1]. This made urllib3 reject an URL of `http://[fe80::a9fe:a9fe]` as invalid argument. This change now appends an interface in the proper URL-encoded format ([2]). [1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/netinet_in.h.html [2] https://datatracker.ietf.org/doc/html/rfc6874 Fixes GH-5422 Signed-off-by: Christian Rohmann <[email protected]>

ani-sinha · 2024-10-15T08:44:23Z

@holmanb when we apply this patch to 24.1.4, we are failing a test which surprisingly passes when we use Ubuntu's 24.3.1 version of cloud-init under identical setup:

[K         Starting [0;1;39mcloud-init.service[0m - Cloud-init: Network Stage...
[   51.425393] cloud-init[1155]: Cloud-init v. 24.2-1.el10.xiachen202410081653 running 'init' at Sat, 12 Oct 2024 07:35:28 +0000. Up 51.40 seconds.
[   51.458179] cloud-init[1155]: ci-info: +++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++
[   51.466753] cloud-init[1155]: ci-info: +--------+------+------------------------------+-----------+--------+-------------------+
[   51.475232] cloud-init[1155]: ci-info: | Device |  Up  |           Address            |    Mask   | Scope  |     Hw-Address    |
[   51.483694] cloud-init[1155]: ci-info: +--------+------+------------------------------+-----------+--------+-------------------+
[   51.492234] cloud-init[1155]: ci-info: | enp3s0 | True | fd2e:6f44:5dd8:c956::9f/128  |     .     | global | fa:16:3e:be:3d:62 |
[   51.500750] cloud-init[1155]: ci-info: | enp3s0 | True | fe80::f816:3eff:febe:3d62/64 |     .     |  link  | fa:16:3e:be:3d:62 |
[   51.509193] cloud-init[1155]: ci-info: |   lo   | True |          127.0.0.1           | 255.0.0.0 |  host  |         .         |
[   51.517523] cloud-init[1155]: ci-info: |   lo   | True |           ::1/128            |     .     |  host  |         .         |
[   51.525862] cloud-init[1155]: ci-info: +--------+------+------------------------------+-----------+--------+-------------------+
[   51.534528] cloud-init[1155]: ci-info: ++++++++++++++++++++++++++++++++++Route IPv6 info+++++++++++++++++++++++++++++++++++
[   51.543976] cloud-init[1155]: ci-info: +-------+--------------------------+---------------------------+-----------+-------+
[   51.552187] cloud-init[1155]: ci-info: | Route |       Destination        |          Gateway          | Interface | Flags |
[   51.560231] cloud-init[1155]: ci-info: +-------+--------------------------+---------------------------+-----------+-------+
[   51.568184] cloud-init[1155]: ci-info: |   0   | fd2e:6f44:5dd8:c956::9f  |             ::            |   enp3s0  |   U   |
[   51.576194] cloud-init[1155]: ci-info: |   1   | fd2e:6f44:5dd8:c956::/64 |             ::            |   enp3s0  |   U   |
[   51.584119] cloud-init[1155]: ci-info: |   2   |        fe80::/64         |             ::            |   enp3s0  |   U   |
[   51.592097] cloud-init[1155]: ci-info: |   3   |           ::/0           | fe80::f816:3eff:fea6:4411 |   enp3s0  |   UG  |
[   51.599952] cloud-init[1155]: ci-info: |   5   |          local           |             ::            |   enp3s0  |   U   |
[   51.607837] cloud-init[1155]: ci-info: |   6   |          local           |             ::            |   enp3s0  |   U   |
[   51.615757] cloud-init[1155]: ci-info: |   7   |        multicast         |             ::            |   enp3s0  |   U   |
[   51.623673] cloud-init[1155]: ci-info: +-------+--------------------------+---------------------------+-----------+-------+
[   51.660201] cloud-init[1155]: 2024-10-12 07:35:29,188 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='fe80::a9fe:a9fe%25enp3s0', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f36e5042a50>: Failed to establish a new connection: [Errno -2] Name or service not known'))"), UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f36e5043380>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")] during request to http://169.254.169.254/openstack, raising last exception
[   51.706374] cloud-init[1155]: 2024-10-12 07:35:29,189 - url_helper.py[ERROR]: Timed out, no response from urls: ['http://[fe80::a9fe:a9fe%25enp3s0]/openstack', 'http://169.254.169.254/openstack']
[   51.718984] cloud-init[1155]: 2024-10-12 07:35:29,189 - util.py[WARNING]: No active metadata service found

Any ideas why it fails after backport?

holmanb · 2024-10-15T16:02:00Z

Any ideas why it fails after backport?

Based on the message "Failed to establish a new connection: [Errno -2] Name or service not known", I would guess that there is no network connection.

Based on the lack of IPv4 routes, I would assume that this is an ipv6-only test of some kind?

It is hard to tell without more information. What was the test environment? Is it reproducible? What network configuration did Openstack pass to cloud-init? Why aren't there any assigned IPv6 gateways in the output?

To diagnose this I would probably run cloud-init collect-logs (from the mounted image, if it isn't possible to log in) and check for warning or error logs as well as examine what configurations came from the Openstack IMDS.

xiachen-rh · 2024-10-16T14:25:53Z

Hi @ani-sinha @holmanb
I created an instance on openstack with an ipv6-ony network. cloud-init version is 24.1.4+this patch.
Here are the collect logs,
241-ipv6only-cloud-init.tar.gz

The openstack IPv6 metadata service connection failed during the first boot, but it can be connected when I login the VM.
[root@localhost ~]# curl http://[fe80::a9fe:a9fe%25enp3s0]/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
2020-10-14
latest

Let me know if require more information.

ani-sinha · 2024-10-17T10:02:50Z

Hi @ani-sinha @holmanb I created an instance on openstack with an ipv6-ony network. cloud-init version is 24.1.4+this patch. Here are the collect logs, 241-ipv6only-cloud-init.tar.gz

The openstack IPv6 metadata service connection failed during the first boot, but it can be connected when I login the VM. [root@localhost ~]# curl http://[fe80::a9fe:a9fe%25enp3s0]/openstack 2012-08-10 2013-04-04 2013-10-17 2015-10-15 2016-06-30 2016-10-06 2017-02-22 2018-08-27 2020-10-14 latest

Let me know if require more information.

Here is the case which succeeds with Ubuntu:

2024-10-12 09:27:07,068 - url_helper.py[DEBUG]: [0/1] open 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack' with {'url': 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 10.0, 'headers': {'User-Agent': 'Cloud-Init/24.3.1-0ubuntu0~24.04.2'}} configuration
2024-10-12 09:27:08,394 - url_helper.py[DEBUG]: Read from http://[fe80::a9fe:a9fe%25enp3s0]/openstack (200, 105b) after 1 attempts

Here is the one which fails with RHEL VM

024-10-16 08:21:51,337 - url_helper.py[DEBUG]: [0/1] open 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack' with {'url': 'http://[fe80::a9fe:a9fe%25enp3s0]/openstack', 'stream': False, 'allow_redirects': True, 'method': 'GET', 'timeout': 10.0, 'headers': {'User-Agent': 'Cloud-Init/24.1.4-17.el10.xiachen202410102239'}} configuration
2024-10-16 08:21:51,490 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='fe80::a9fe:a9fe%25enp3s0', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1559e3fe60>: Failed to establish a new connection: [Errno -2] Name or service not known'))"),

In the Ubuntu VM we have

ubuntu@xiachen-ubuntu:~$ ip -6 addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1422 state UP qlen 1000
    inet6 fd2e:6f44:5dd8:c956::35a/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fefd:47ea/64 scope link 
       valid_lft forever preferred_lft forever
ubuntu@xiachen-ubuntu:~$ ip a show enp3s0
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1422 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:fd:47:ea brd ff:ff:ff:ff:ff:ff
    inet6 fd2e:6f44:5dd8:c956::35a/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fefd:47ea/64 scope link 
       valid_lft forever preferred_lft forever

For RHEL we have

[root@localhost ~]# ip -6 addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1422 state UP qlen 1000
    inet6 fd2e:6f44:5dd8:c956::258/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe2a:ddb2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]# ip a show enp3s0
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1422 qdisc fq_codel state UP group default qlen 1000
    link/ether fa:16:3e:2a:dd:b2 brd ff:ff:ff:ff:ff:ff
    inet6 fd2e:6f44:5dd8:c956::258/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe2a:ddb2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

I am wondering whether Ubuntu uses some special ipv6 config that we are missing in RHEL?
One difference I see is that for fe80::/10 Ubuntu does not add noprefixroute but we do. Can this be the cause?

In any case, this change seems to break us and it will be problematic for us when we rebase to latest upstream even if we ended up fixing RHEL (if the issue is in RHEL).

frittentheke · 2024-10-17T12:57:27Z

One difference I see is that for fe80::/10 Ubuntu does not add noprefixroute but we do. Can this be the cause?

Yes @ani-sinha as this breaks routing between your VM's link-local and the metadata service listening to fe80::a9fe:a9fe.

ani-sinha · 2024-10-17T15:13:44Z

One difference I see is that for fe80::/10 Ubuntu does not add noprefixroute but we do. Can this be the cause?

Yes @ani-sinha as this breaks routing between your VM's link-local and the metadata service listening to fe80::a9fe:a9fe.

Can you please explain how? I am not a ipv6 expert but what change does noprefixroute bring @frittentheke ?

frittentheke · 2024-10-17T15:49:06Z

@ani-sinha

Can you please explain how? I am not a ipv6 expert but what change does noprefixroute bring @frittentheke ?

See torvalds/linux@761aac7 for some background.
But likely this actually is a red herring though, sorry about that. Maybe, if you care, let's see the IPv6 routes (ip -6 route) on the non-functional system ... but rather follow my other assumption:

What makes me wonder is the error message Failed to establish a new connection: [Errno -2] Name or service not known. If there was not route it should say no route to host or similar. I much rather believe the Python used does not understand the interface scope.

Please check the versions of Python requests or more importantly the urllib3 package for this PR / commit:

Introduce IPv6 zone identifier support. urllib3/urllib3#1013

If that is missing in your case, then that is likely the reason why the URL string http://[fe80::a9fe:a9fe%25enp3s0]/openstack is not properly interpreted as a bare IPv6 address with a scope appended. It's rather treated as a hostname.

ani-sinha · 2024-10-18T15:23:00Z

Please check the versions of Python requests or more importantly the urllib3 package for this PR / commit:

Introduce IPv6 zone identifier support. urllib3/urllib3#1013

If that is missing in your case, then that is likely the reason why the URL string http://[fe80::a9fe:a9fe%25enp3s0]/openstack is not properly interpreted as a bare IPv6 address with a scope appended. It's rather treated as a hostname.

@frittentheke
We seem to have the latest code that is present in upstream related to this change:
https://github.com/urllib3/urllib3/blob/main/src/urllib3/connectionpool.py#L1140

def _normalize_host(host, scheme):
    """         
    Normalize hosts for comparisons and use with sockets.
    """     
                
    host = normalize_host(host, scheme)
            
    # httplib doesn't like it when we include brackets in IPv6 addresses
    # Specifically, if we include brackets but also pass the port then
    # httplib crazily doubles up the square brackets on the Host header.
    # Instead, we need to make sure we never pass ``None`` as the port.
    # However, for backward compatibility reasons we can't actually
    # *assert* that.  See http://bugs.python.org/issue28539
    if host.startswith("[") and host.endswith("]"):
        host = host[1:-1]
    return host

We seem to have the version 1.26.19
https://github.com/urllib3/urllib3/releases/tag/1.26.19

frittentheke mentioned this pull request Jun 20, 2024

Srv6d openstack dualstack #1805

Merged

3 tasks

frittentheke changed the title ~~OpenStack: Append interface for IPv6 link-local metadata address~~ fix (openstack): Append interface for IPv6 link-local metadata address Jun 20, 2024

frittentheke marked this pull request as ready for review June 20, 2024 18:53

holmanb reviewed Jun 20, 2024

View reviewed changes

frittentheke force-pushed the fixIPv6OpenStack branch from 7bad298 to a79edf7 Compare June 20, 2024 19:21

frittentheke force-pushed the fixIPv6OpenStack branch from a79edf7 to 6131d39 Compare June 20, 2024 19:45

holmanb reviewed Jun 20, 2024

View reviewed changes

cloudinit/sources/DataSourceOpenStack.py Outdated Show resolved Hide resolved

frittentheke force-pushed the fixIPv6OpenStack branch from 6131d39 to 75d5fbd Compare June 21, 2024 07:13

frittentheke requested a review from holmanb June 21, 2024 07:15

frittentheke changed the title ~~fix (openstack): Append interface for IPv6 link-local metadata address~~ fix (openstack): Append interface / scope_id for IPv6 link-local metadata address Jun 21, 2024

holmanb self-assigned this Jun 21, 2024

holmanb requested changes Jun 21, 2024

View reviewed changes

cloudinit/sources/DataSourceOpenStack.py Outdated Show resolved Hide resolved

frittentheke force-pushed the fixIPv6OpenStack branch from 75d5fbd to af4ac7f Compare June 21, 2024 19:10

fix test

a14c390

holmanb approved these changes Jun 24, 2024

View reviewed changes

holmanb merged commit 2ea2765 into canonical:main Jun 24, 2024
23 checks passed

frittentheke deleted the fixIPv6OpenStack branch June 25, 2024 13:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix (openstack): Append interface / scope_id for IPv6 link-local metadata address #5419

fix (openstack): Append interface / scope_id for IPv6 link-local metadata address #5419

frittentheke commented Jun 20, 2024 •

edited

Loading

holmanb left a comment •

edited

Loading

holmanb Jun 20, 2024

frittentheke Jun 20, 2024 •

edited

Loading

holmanb Jun 20, 2024

frittentheke commented Jun 20, 2024 •

edited

Loading

tomra commented Jun 21, 2024 •

edited

Loading

holmanb left a comment

frittentheke commented Jun 21, 2024

frittentheke commented Jun 24, 2024

holmanb left a comment

ani-sinha commented Oct 15, 2024

holmanb commented Oct 15, 2024 •

edited

Loading

xiachen-rh commented Oct 16, 2024 •

edited

Loading

ani-sinha commented Oct 17, 2024

frittentheke commented Oct 17, 2024 •

edited

Loading

ani-sinha commented Oct 17, 2024

frittentheke commented Oct 17, 2024 •

edited

Loading

ani-sinha commented Oct 18, 2024 •

edited

Loading

fix (openstack): Append interface / scope_id for IPv6 link-local metadata address #5419

fix (openstack): Append interface / scope_id for IPv6 link-local metadata address #5419

Conversation

frittentheke commented Jun 20, 2024 • edited Loading

Proposed Commit Message

Additional Context

Test Steps

Checklist

Merge type

holmanb left a comment • edited Loading

Choose a reason for hiding this comment

holmanb Jun 20, 2024

Choose a reason for hiding this comment

frittentheke Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

holmanb Jun 20, 2024

Choose a reason for hiding this comment

frittentheke commented Jun 20, 2024 • edited Loading

tomra commented Jun 21, 2024 • edited Loading

holmanb left a comment

Choose a reason for hiding this comment

frittentheke commented Jun 21, 2024

frittentheke commented Jun 24, 2024

holmanb left a comment

Choose a reason for hiding this comment

ani-sinha commented Oct 15, 2024

holmanb commented Oct 15, 2024 • edited Loading

xiachen-rh commented Oct 16, 2024 • edited Loading

ani-sinha commented Oct 17, 2024

frittentheke commented Oct 17, 2024 • edited Loading

ani-sinha commented Oct 17, 2024

frittentheke commented Oct 17, 2024 • edited Loading

ani-sinha commented Oct 18, 2024 • edited Loading

frittentheke commented Jun 20, 2024 •

edited

Loading

holmanb left a comment •

edited

Loading

frittentheke Jun 20, 2024 •

edited

Loading

frittentheke commented Jun 20, 2024 •

edited

Loading

tomra commented Jun 21, 2024 •

edited

Loading

holmanb commented Oct 15, 2024 •

edited

Loading

xiachen-rh commented Oct 16, 2024 •

edited

Loading

frittentheke commented Oct 17, 2024 •

edited

Loading

frittentheke commented Oct 17, 2024 •

edited

Loading

ani-sinha commented Oct 18, 2024 •

edited

Loading