You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Alloy uses a headless service and queries DNS for SRV records to discover peer instances, which can be used to join the cluster via memberlist-based gossip protocol provided by the ckit library.
These queries were in some reported cases returning zero results, which is evidenced by a periodic logging of msg=”rejoining peers”, peers=””
The standard Go DNS library used by Alloy was returning no results for the query.
In contrast, queries run using dig CLI utility were still returning valid results.
This indicates that there was a bug or implementation difference between Go DNS library and other DNS clients.
The DNS queries are limited in size. The query is first attempted via UDP, when the response from DNS server indicates that the message was truncated, the query is retried over TCP.
However, even the TCP query can be truncated. This is because the TCP messages still wrap the RFC1035 DNS message which is prefixed with a two byte length field, so the maximum length of DNS response over TCP is 65536.
When a TCP message is truncated, the Truncated flag is set to 1.
However, as seen above, there are still useful Answer RRs in the response, which can be used on the “best effort” basis by the clients.
dig does this, but Go didn’t due to this bug: golang/go#64896
The fix for this issue should be available in go1.23
But why are the DNS query responses so large? When Alloy cluster has, for example, 90 instances and for each of these instances, there are 10 different ports registered in the headless service, there will be 900 SRV records registered in DNS.
Issue
Alloy uses a headless service and queries DNS for SRV records to discover peer instances, which can be used to join the cluster via memberlist-based gossip protocol provided by the ckit library.
These queries were in some reported cases returning zero results, which is evidenced by a periodic logging of msg=”rejoining peers”, peers=””
The standard Go DNS library used by Alloy was returning no results for the query.
In contrast, queries run using dig CLI utility were still returning valid results.
This indicates that there was a bug or implementation difference between Go DNS library and other DNS clients.
The DNS queries are limited in size. The query is first attempted via UDP, when the response from DNS server indicates that the message was truncated, the query is retried over TCP.
However, even the TCP query can be truncated. This is because the TCP messages still wrap the RFC1035 DNS message which is prefixed with a two byte length field, so the maximum length of DNS response over TCP is 65536.
When a TCP message is truncated, the Truncated flag is set to 1.
![image](https://private-user-images.githubusercontent.com/17101802/346123535-aa030c30-8937-4412-9437-cd35588f5528.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQzMDgxMDksIm5iZiI6MTcyNDMwNzgwOSwicGF0aCI6Ii8xNzEwMTgwMi8zNDYxMjM1MzUtYWEwMzBjMzAtODkzNy00NDEyLTk0MzctY2QzNTU4OGY1NTI4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MjIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODIyVDA2MjMyOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNhMjkyODhjNmUyZTYzYWRjNzU3ZmY2ZjdjNDVlYTBhZWViNzAyYmUyYjExMGNiMTYwYjU4Y2QwMmYwODc1ZjMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.VVtI16sUJRtN7_u9hWOHewJ0hgMjYjqlLNyWpKyPHIQ)
However, as seen above, there are still useful Answer RRs in the response, which can be used on the “best effort” basis by the clients.
dig does this, but Go didn’t due to this bug: golang/go#64896
The fix for this issue should be available in go1.23
But why are the DNS query responses so large? When Alloy cluster has, for example, 90 instances and for each of these instances, there are 10 different ports registered in the headless service, there will be 900 SRV records registered in DNS.
This is quite common - initially, the Alloy helm chart only supported one port, but support for more ports was added later.
There are 9 ports defined in the k8s-monitoring helm chart.
Proposed fixes
There are a few tasks that should be taken to resolve this issue:
Tasks
The text was updated successfully, but these errors were encountered: