Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dns, http]: support IDNA domains #1192

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[dns, http]: support IDNA domains #1192

wants to merge 2 commits into from

Conversation

InsanePrawn
Copy link

@InsanePrawn InsanePrawn commented Feb 8, 2024 •
edited
Loading

based on #640, added support to the HTTP prober.

Works on my machine now(TM)

curl 'http://localhost:9115/probe?module=http_2xx&target=http://www.académie-française.fr/&debug=true'




Logs for the probe:
ts=2024-02-08T18:02:54.867186246Z caller=main.go:189 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Beginning probe" probe=http timeout_seconds=119.5
ts=2024-02-08T18:02:54.867269101Z caller=http.go:322 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Domain internationalized" unicode=www.académie-française.fr ascii=www.xn--acadmie-franaise-npb1a.fr
ts=2024-02-08T18:02:54.867291323Z caller=http.go:328 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Resolving target address" target=www.xn--acadmie-franaise-npb1a.fr ip_protocol=ip4
ts=2024-02-08T18:02:54.868394409Z caller=http.go:328 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Resolved target address" target=www.xn--acadmie-franaise-npb1a.fr ip=212.95.66.79
ts=2024-02-08T18:02:54.868466915Z caller=client.go:259 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Making HTTP request" url=http://212.95.66.79/ host=www.académie-française.fr
ts=2024-02-08T18:02:54.923353365Z caller=client.go:505 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Received redirect" location=https://www.academie-francaise.fr/
ts=2024-02-08T18:02:54.923550915Z caller=client.go:259 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Making HTTP request" url=https://www.academie-francaise.fr/ host=
ts=2024-02-08T18:02:54.923568167Z caller=client.go:259 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Address does not match first address, not sending TLS ServerName" first=212.95.66.79 address=www.academie-francaise.fr
ts=2024-02-08T18:02:55.035395335Z caller=handler.go:119 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Received HTTP response" status_code=200
ts=2024-02-08T18:02:55.057272708Z caller=handler.go:119 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Response timings for roundtrip" roundtrip=0 start=2024-02-08T19:02:54.868545632+01:00 dnsDone=2024-02-08T19:02:54.868545632+01:00 connectDone=2024-02-08T19:02:54.897483221+01:00 gotConn=2024-02-08T19:02:54.897621701+01:00 responseStart=2024-02-08T19:02:54.923206089+01:00 tlsStart=0001-01-01T00:00:00Z tlsDone=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2024-02-08T18:02:55.057305229Z caller=handler.go:119 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Response timings for roundtrip" roundtrip=1 start=2024-02-08T19:02:54.923630574+01:00 dnsDone=2024-02-08T19:02:54.924789154+01:00 connectDone=2024-02-08T19:02:54.954272586+01:00 gotConn=2024-02-08T19:02:55.010455876+01:00 responseStart=2024-02-08T19:02:55.035305076+01:00 tlsStart=2024-02-08T19:02:54.954323051+01:00 tlsDone=2024-02-08T19:02:55.010441198+01:00 end=2024-02-08T19:02:55.057234436+01:00
ts=2024-02-08T18:02:55.057341838Z caller=main.go:189 module=http_2xx target=http://www.académie-française.fr/ level=info msg="Probe succeeded" duration_seconds=0.190124914



Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 0.00112117
# HELP probe_duration_seconds Returns how long the probe took to complete in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 0.190124914
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HELP probe_http_content_length Length of http content response
# TYPE probe_http_content_length gauge
probe_http_content_length -1
# HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirects
# TYPE probe_http_duration_seconds gauge
probe_http_duration_seconds{phase="connect"} 0.058559501
probe_http_duration_seconds{phase="processing"} 0.050433588
probe_http_duration_seconds{phase="resolve"} 0.0022797599999999996
probe_http_duration_seconds{phase="tls"} 0.056118147
probe_http_duration_seconds{phase="transfer"} 0.02192935
# HELP probe_http_last_modified_timestamp_seconds Returns the Last-Modified HTTP response header in unixtime
# TYPE probe_http_last_modified_timestamp_seconds gauge
probe_http_last_modified_timestamp_seconds 1.707412895e+09
# HELP probe_http_redirects The number of redirects
# TYPE probe_http_redirects gauge
probe_http_redirects 1
# HELP probe_http_ssl Indicates if SSL was used for the final redirect
# TYPE probe_http_ssl gauge
probe_http_ssl 1
# HELP probe_http_status_code Response HTTP status code
# TYPE probe_http_status_code gauge
probe_http_status_code 200
# HELP probe_http_uncompressed_body_length Length of uncompressed response body
# TYPE probe_http_uncompressed_body_length gauge
probe_http_uncompressed_body_length 34425
# HELP probe_http_version Returns the version of HTTP of the probe response
# TYPE probe_http_version gauge
probe_http_version 1.1
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 4.009199703e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_ssl_earliest_cert_expiry Returns last SSL chain expiry in unixtime
# TYPE probe_ssl_earliest_cert_expiry gauge
probe_ssl_earliest_cert_expiry 1.710547199e+09
# HELP probe_ssl_last_chain_expiry_timestamp_seconds Returns last SSL chain expiry in timestamp
# TYPE probe_ssl_last_chain_expiry_timestamp_seconds gauge
probe_ssl_last_chain_expiry_timestamp_seconds 1.710547199e+09
# HELP probe_ssl_last_chain_info Contains SSL leaf certificate information
# TYPE probe_ssl_last_chain_info gauge
probe_ssl_last_chain_info{fingerprint_sha256="d35ce7d9983fb599ffdd092908ac11a66ec0ce5ffab6ff001ff06d12b0abaf32",issuer="CN=RapidSSL Global TLS RSA4096 SHA256 2022 CA1,O=DigiCert\\, Inc.,C=US",subject="CN=www.academie-francaise.fr",subjectalternative="www.academie-francaise.fr,academie-francaise.fr"} 1
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1
# HELP probe_tls_version_info Returns the TLS version used or NaN when unknown
# TYPE probe_tls_version_info gauge
probe_tls_version_info{version="TLS 1.2"} 1



Module configuration:
prober: http
http:
  preferred_ip_protocol: ip4
  ip_protocol_fallback: true
  follow_redirects: true
  enable_http2: true
tcp:
  ip_protocol_fallback: true
icmp:
  ip_protocol_fallback: true
  ttl: 64
dns:
  ip_protocol_fallback: true
  recursion_desired: true

@InsanePrawn InsanePrawn mentioned this pull request Feb 8, 2024
Open
mem
mem reviewed Mar 8, 2024
View reviewed changes
prober/utils.go
@@ -142,3 +144,18 @@ func ipHash(ip net.IP) float64 {
}
return float64(h.Sum32())
}

func internationalizeDNSDomain(logger log.Logger, domain string) string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be happier if there was a test for this function.

Basically test these cases:

  • IPv4 address
  • IPv6 address
  • regular domain name
  • domain name with non-ASCII characters in UTF form ("www.académie-française.fr")
  • domain name with non-ASCII characters in ASCII form (punycode) ("www.xn--acadmie-franaise-npb1a.fr")

prober/utils.go
@@ -60,6 +61,7 @@ func chooseProtocol(ctx context.Context, IPProtocol string, fallbackIPProtocol b
fallbackProtocol = "ip6"
}

target = internationalizeDNSDomain(logger, target)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

below the target string is used is several places to log results.

Question: do I really want to see the actual target that was used (www.xn--acadmie-franaise-npb1a.fr) or do I want to see the target as entered by the user (www.académie-française.fr)? The first one is useful for debugging, the second one might be less surprising. Maybe this needs documenting?

@github-actions github-actions bot added the stale label Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mem mem mem left review comments

Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@InsanePrawn @mem @roidelapluie