RPKI not *all* IPs are checked for ROA?

[bwbroersma]

See example https://internet.nl/site/amazon.nl/3077099/:

Web server	IPv6 address	IPv4 address
amazon.nl	None	52.95.120.64
...	-	52.95.116.117
...	-	54.239.33.93

For all IP addresses of your web server an RPKI Route Origin Authorisation (ROA) is published.

Webserver	IP address	RPKI Route Origin Authorization
amazon.nl	54.239.33.93	yes

All IP addresses of your web server have a Valid validation state. The route announcement of these IP addresses is matched by the published RPKI Route Origin Authorisation (ROA).

Web server	BGP Route Prefix	BGP Route Origin ASN	RPKI Origin Validation state
amazon.nl	54.239.33.0/24	AS16509	valid
...	54.239.32.0/21	AS16509	valid

[bwbroersma]

The RPKI code has the correct loop:

Internet.nl/checks/tasks/rpki.py

Lines 274 to 275 in 0e5a6d1

	for host in hostset:
	for ip in host.routing:

However at least here:

Internet.nl/checks/tasks/rpki.py

Lines 125 to 126 in 0e5a6d1

	web_registered = check_registry("web_rpki", web_callback, shared.resolve_a_aaaa)
	batch_web_registered = check_registry("batch_web_rpki", batch_web_callback, shared.batch_resolve_a_aaaa)

Both via this code:

Internet.nl/checks/tasks/shared.py

Lines 47 to 64 in 0e5a6d1

	@shared_task(
	bind=True,
	soft_time_limit=settings.SHARED_TASK_SOFT_TIME_LIMIT_HIGH,
	time_limit=settings.SHARED_TASK_TIME_LIMIT_HIGH,
	base=SetupUnboundContext,
	)
	def resolve_a_aaaa(self, qname, *args, **kwargs):
	return do_resolve_a_aaaa(self, qname, *args, **kwargs)
	@batch_shared_task(
	bind=True,
	soft_time_limit=settings.BATCH_SHARED_TASK_SOFT_TIME_LIMIT_HIGH,
	time_limit=settings.BATCH_SHARED_TASK_TIME_LIMIT_HIGH,
	base=SetupUnboundContext,
	)
	def batch_resolve_a_aaaa(self, qname, *args, **kwargs):
	return do_resolve_a_aaaa(self, qname, *args, **kwargs)

Call this function:

Internet.nl/checks/tasks/shared.py

Lines 153 to 154 in 0e5a6d1

	def do_resolve_a_aaaa(self, qname, *args, **kwargs):
	"""Resolve A and AAAA records and return a single result for each type."""

So maybe replacing this with code that returns the complete set.

Another problem seems to be the use of do_resolve_a_aaaa in do_resolve_ns_ips for NS that have multiple A or AAAA records:

Internet.nl/checks/tasks/shared.py

Line 198 in 0e5a6d1

yield (rr, do_resolve_a_aaaa(self, rr))

[baknu]

This seems to be a bug. The intent of the RPKI subtest was to check all IP's also of the servers:

• Land RPKI tests in Internet.nl #613
• https://github.com/internetstandards/Internet.nl/releases/tag/v1.5.0

Odd to discover this only now. Do we want to have this fixed in 1.9 (because from that release on RPKI will have score impact)?

[bwbroersma]

Some useful scripting to use the https://tranco-list.eu/ to find stuff:

curl -sSfLA '' "https://tranco-list.eu/download_daily/$(curl 'https://tranco-list.eu/latest_list' -sSfA '' -o /dev/null -w "%{redirect_url}\n" | cut -d/ -f5)" | bsdtar -Oxf- | cut -d, -f2 | tr -d '\r' > top-1m.list
head -n1000 top-1m.list | sed 's/$/ NS/g' | xargs dig +noall +answer > top1000ns
awk '{print $5}' top1000ns | sed 's/\.$/ A/g' | sort -u | xargs dig +noall +answer | awk '{l[$1]=l[$1]$5" ";n[$1]++}END{for(d in l){if(n[d]>1)print n[d], d, l[d]}}' | sort -nr > top1000ns-multi-ipv4

E.g. cloudflare.com both has 2 IPv4 and IPv6 addresses for the website and all nameservers, while the multiple IP addresses are displayed in the IPv6 nameserver section it's missing in the ROA existence.