IDNA proposal #2874

tlimoncelli · 2024-03-18T00:42:34Z

No description provided.

tlimoncelli · 2024-03-21T13:33:34Z

Mentioning a number of people that I suspect have IDNA domains or knowledge about how IDNA works (based on skimmy bugs and PRs)

@adamus1red
@dkim1970
@flz
@juliusrickert
@killerbees19
@Kusado
@louis-lau
@masterzen
@mderriey
@pmoroney
@tresni
@Yannik

I'd love to receive feedback about this proposal!

Yannik · 2024-03-21T14:33:43Z

@tlimoncelli

I think your example for the asci==unicode exampe might be wrong:

#7: + CREATE foo.example.com MX 10 xn--p1ai.com (рф.com) (ttl=14400)

Was this what you intended?

Yannik · 2024-03-21T14:42:55Z

I noticed you created both CREATE and MODIFY examples for ascii (unicode) and unicode (ascii), but only MODIFY examples for ascii and unicode. How is that to be understood?

tlimoncelli · 2024-03-21T14:43:46Z

@tlimoncelli

I think your example for the asci==unicode exampe might be wrong:
#7: + CREATE foo.example.com MX 10 xn--p1ai.com (рф.com) (ttl=14400)
Was this what you intended?

Ah, good point!

The first one is where the label is ascii==unicode but the target is ascii!=unicode. I'll update the comment.

Thanks for finding that!

Tom

tlimoncelli · 2024-03-21T15:20:47Z

I noticed you created both CREATE and MODIFY examples for ascii (unicode) and unicode (ascii), but only MODIFY examples for ascii and unicode. How is that to be understood?

I've added more examples. I don't think I've covered every combination, but my goal is to show typical examples not every possible example.

I've also added examples where we use {} and ⟬⟭ and ❮❯. I think using unicode chars to highlight unicode domains would be cool (maybe too clever?).

adamus1red · 2024-03-21T19:27:34Z

While I'm not against having the ASCII and UTF on the output lines, I do worry it might make the output too busy.
Wouldn't simply using the .Name value be a better since it should then pretty much match what is in the dnscontrol configuration?

pmoroney · 2024-03-21T20:57:49Z

LGTM

tlimoncelli · 2024-03-21T21:02:00Z

@adamus1red wrote:

While I'm not against having the ASCII and UTF on the output lines, I do worry it might make the output too busy.
Wouldn't simply using the .Name value be a better since it should then match what is in the dnscontrol configuration?

That's an interesting point! I guess my thought is that showing both versions helps with debugging.

adamus1red · 2024-03-21T21:17:10Z

That's an interesting point! I guess my thought is that showing both versions helps with debugging.

@tlimoncelli maybe a compromise would be if the output was the same as what the DNS provider or Registrar used.

I know I've had issues where the DNS is using UTF but the registrar is using ASCII. I.e. namecheap uses ascii, so for registrar stuff using namecheap use ascii punycode and the DNS is cloudflare which uses UTF, so the output uses UTF.

louis-lau · 2024-03-21T21:17:30Z

The only IDNA domain I have is for fun, so I don't have a strong preference. I'll give my input nonetheless :). If you want to show both, I think I like B better, as it feels more consistent to me. Anything not in brackets will always be ASCII that way.

I'd probably go with showing what the original user input was, with a flag to only show ASCII if needed. It's less information to parse, and the user should be familiar with it as that's the way it's listed in their config. I could see points being made for showing both, but I've always liked things more distraction free and less dense.

I think the Unicode brackets are a little too clever, perhaps even a little confusing ;)

Yannik · 2024-03-23T15:02:49Z

First of all, improving IDNA handling would be a great improvement to dnscontrol.

Regarding output, the one thing I definitely do not like is having the ascii output come first, because it is the one least likely to be understood/mentally associated with the relevant domain.

I think simply using the original user input has merit, pairing that with a toggle to additionally show ascii seems fine to me.
However, I also wouldn't mind the unicode (ascii) output.

dkim1970 · 2024-03-23T15:20:59Z

First of all, improving IDNA handling would be a great improvement to dnscontrol.

Regarding output, the one thing I definitely do not like is having the ascii output come first, because it is the one least likely to be understood/mentally associated with the relevant domain.

I think simply using the original user input has merit, pairing that with a toggle to additionally show ascii seems fine to me. However, I also wouldn't mind the unicode (ascii) output.

I'm seconding this suggestion, by displaying the "human readable" format I think the barrier for using IDN's with dnscontrol is getting lowered.

Because the IDNA format is not human readable, especially when it comes to non-latinized languages.

tlimoncelli · 2024-03-23T16:41:33Z

This is excellent feedback! It's getting me excited!

Question: In what situations would people want to see something besides the .Name (the user input) version?

adamus1red · 2024-03-23T18:14:00Z

Question: In what situations would people want to see something besides the .Name (the user input) version?

What about if the registrar or dns provider use something different than the .Name value, then include the version they are using in brackets?

louis-lau · 2024-03-23T20:16:50Z

Personally, I think whatever the dns provider does isn't relevant to the cli output. Behind the scenes at every provider, it's all punycode anyway.

Yannik · 2024-03-23T20:44:12Z

Personally, I think whatever the dns provider does isn't relevant to the cli output. Behind the scenes at every provider, it's all punycode anyway.

Agreed, having the display format handled outside of the provider is to be preferred IMO.

masterzen · 2024-03-24T12:19:11Z

I don't have experience with IDNA at all.
My $0.02: I do agree that showing the ascii version is useful only for debugging, what users want to see is if their unicode domain (or however it was entered in dnsconfig.js) is being processed and how.

tlimoncelli · 2024-03-25T13:18:22Z

Hi folks!

2 ideas:

Support multiple formats?

There's been a lot of discussion about ascii (unicode) vs unicode (ascii). It might be possible to add a command line flag that selected the format. No promises, but it might be possible. In that case, I'd recommend the default be userinput and add a flag for debugging that shows unicode (ascii) or userinput (ascii).

I'll know more if this is possible when I start coding.

An idea that would break less existing code

Existing code expects .Name to be ASCII (the current code runs dc.Punycode() for all providers, which rewrites .Name to be ASCII). Rather than require every use of .Name to change to .NameASCII, maybe the names should be: .Name (ASCII, to be compatible with old code), .NameORIG (how the user input the string), .NameUNICODE, and .NameDisplay.

flokli · 2025-05-11T18:25:00Z

documentation/design-idna.md

+```
+models.DomainConfig:
+
+   .Name: the name from D() after downcased via unicode.ToLower()


I'd prefer to always put the punycode variant in here (which is identical to the non-punycode if it's a ASCII-only domain).
This should require the least amount of changes in all providers and "do the right thing" out of the box 99% of the time, without breaking pure-ASCII domains.

It's also least surprising for users - how they write it shouldn't affect how the provider should treat it.
Which encoding the user provided in the D call is nothing individual providers need to worry about.

flokli · 2025-05-11T18:26:12Z

documentation/design-idna.md

+models.DomainConfig:
+
+   .Name: the name from D() after downcased via unicode.ToLower()
+   .NameASCII: The name stored after calling ToASCII() (with ACE prefix if any Unicode chars are present)


flokli · 2025-05-11T18:27:43Z

documentation/design-idna.md

+
+   .Name: the name from D() after downcased via unicode.ToLower()
+   .NameASCII: The name stored after calling ToASCII() (with ACE prefix if any Unicode chars are present)
+   .NameUnicode: The name stored after calling ToUnicode()


IDN seems more fitting (Internationalized Domain Name)

flokli · 2025-05-11T18:29:20Z

documentation/design-idna.md

+   .Name: the name from D() after downcased via unicode.ToLower()
+   .NameASCII: The name stored after calling ToASCII() (with ACE prefix if any Unicode chars are present)
+   .NameUnicode: The name stored after calling ToUnicode()
+   .NameDisplay: if .NameASCII != .NameUnicode, store as "ascii (unicode)"


This is confusing. Shouldn't this be the bigger slice that Name and IDN would be subslices of?

flokli · 2025-05-11T18:31:51Z

documentation/design-idna.md

+
+Here are some example outputs:
+
+NOTE: Feedback needed!  Do you prefer "a" or "b"?  Is there an even better format I should consider?  Should we use `{}` instead of `()`?


I prefer b and (). We should also specify NameDisplay only is a string with both variants if the domain is non-ASCII.

KaiSchwarz-cnic · 2025-05-12T07:39:14Z

documentation/design-idna.md

+
+Here are some example outputs:
+
+NOTE: Feedback needed!  Do you prefer "a" or "b"?  Is there an even better format I should consider?  Should we use `{}` instead of `()`?


band ()

KaiSchwarz-cnic · 2025-05-12T07:40:47Z

documentation/design-idna.md

+models.DomainConfig:
+
+   .Name: the name from D() after downcased via unicode.ToLower()
+   .NameASCII: The name stored after calling ToASCII() (with ACE prefix if any Unicode chars are present)


KaiSchwarz-cnic · 2025-05-12T07:41:10Z

documentation/design-idna.md

+```
+models.DomainConfig:
+
+   .Name: the name from D() after downcased via unicode.ToLower()


punycode variant in here ftw.

KaiSchwarz-cnic · 2025-05-12T07:53:38Z

documentation/design-idna.md

+   .Name: the name from D() after downcased via unicode.ToLower()
+   .NameASCII: The name stored after calling ToASCII() (with ACE prefix if any Unicode chars are present)
+   .NameUnicode: The name stored after calling ToUnicode()
+   .NameDisplay: if .NameASCII != .NameUnicode, store as "ascii (unicode)"


So, it will be either

mydomain.com if it is not about an IDN and

xn--p1ai.com (рф.com) otherwise.

Sounds good to me.

KaiSchwarz-cnic · 2025-05-12T07:54:58Z

documentation/design-idna.md

+
+   .Name: the name from D() after downcased via unicode.ToLower()
+   .NameASCII: The name stored after calling ToASCII() (with ACE prefix if any Unicode chars are present)
+   .NameUnicode: The name stored after calling ToUnicode()


While I enjoy playing with short names that make things clear - so IDN would be perfect, but from developer perspective, having that property to start also with NameXXX makes it probably better accessible via IDE as the Name properties appear right after each other.

.NameIDN isn't applicable as it states "name" twice out

.NameUnicode is ok though

.NameInternationalized eventually as well

My favs are by that .IDN or .NameUnicode while .NameUnicode is still my personal pick. Unicode and ASCII keywords are often used whenever it comes to IDN translation libraries. While IDN and Punycode keywords are usually used on Domain/DNS Provider side. That's at least what I noticed over the years. Still, that shouldn't give a hint for making a decision.

KaiSchwarz-cnic · 2025-05-12T08:12:03Z

Also, IDN isn't IDN if we compare .de and .com. Some TLD Providers support different IDNA Standards (IDNA2003 vs. IDNA2008, UTS46). Translating an IDN might by that end in a different punycode variant.

Let me provide some example in here from the HEXONET Provider's ConvertIDN API Command:

[COMMAND]
COMMAND = ConvertIDN
DOMAIN0 = ärzte.com
DOMAIN1 = ärzte.de
EOF

[RESPONSE]
CODE = 200
DESCRIPTION = Command completed successfully
PROPERTY[ACE][0] = xn--rzte-koa.com
PROPERTY[ACE][1] = xn--rzte-koa.de
PROPERTY[IDN][0] = ärzte.com
PROPERTY[IDN][1] = ärzte.de
EOF

No big difference in here. But let us pick one with german special characters:

[COMMAND]
COMMAND = ConvertIDN
DOMAIN0 = fußball.com
DOMAIN1 = fußball.de
EOF

[RESPONSE]
CODE = 200
DESCRIPTION = Command completed successfully
PROPERTY[ACE][0] = fussball.com
PROPERTY[ACE][1] = xn--fuball-cta.de
PROPERTY[IDN][0] = fussball.com
PROPERTY[IDN][1] = fußball.de
EOF

Let us ignore that .com is covering that differently and let us use the punycode variant returned for .de as a .com domain name xn--fuball-cta.com. While the IDN translation is from technical perspective correct, it won't work together with the TLD Provider as of a different supported IDNA Standard. By that, I highly think that this needs to be considered as well when going for a IDNA proposal. The above API Command is mapping the response to a working variant.

Mhmm... DNSControl again runs on "existing" data configured by the user. By that, the input should be considered as "correct" (would be very stupid otherwise) and by that, we can consider this special discussion probably as superfluous...
Or should DNSControl then exit with an error in case a potential IDN Precheck fails?

Mhmm 2 ... The DNS/Domain Provider should finally be capable of handling that on their own (returning an error message static out that the provided domain/dnszone name is invalid) and you guys do not have to worry about all that. Sorry that I bumped this up :-)

tlimoncelli added 3 commits March 17, 2024 20:41

IDNA proposal

3195c39

fixup!

169ec20

Merge branch 'main' into tlim_puny

d0814a8

tlimoncelli added 3 commits March 21, 2024 10:44

Merge branch 'main' into tlim_puny

5f93e32

Address Yannik's note

91d8ac6

more examples

bb4d88d

Merge branch 'main' into tlim_puny

2689cd8

flokli mentioned this pull request May 11, 2025

Refactor Punycode (IDNA/ACE) handling #2097

Open

flokli reviewed May 11, 2025

View reviewed changes

KaiSchwarz-cnic reviewed May 12, 2025

View reviewed changes


		Here are some example outputs:

		NOTE: Feedback needed! Do you prefer "a" or "b"? Is there an even better format I should consider? Should we use `{}` instead of `()`?

IDNA proposal #2874

Are you sure you want to change the base?

IDNA proposal #2874

Conversation

tlimoncelli commented Mar 18, 2024

Uh oh!

tlimoncelli commented Mar 21, 2024

Uh oh!

Yannik commented Mar 21, 2024

Uh oh!

Yannik commented Mar 21, 2024

Uh oh!

tlimoncelli commented Mar 21, 2024

Uh oh!

tlimoncelli commented Mar 21, 2024

Uh oh!

adamus1red commented Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pmoroney commented Mar 21, 2024

Uh oh!

tlimoncelli commented Mar 21, 2024

Uh oh!

adamus1red commented Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

louis-lau commented Mar 21, 2024

Uh oh!

Yannik commented Mar 23, 2024

Uh oh!

dkim1970 commented Mar 23, 2024

Uh oh!

tlimoncelli commented Mar 23, 2024

Uh oh!

adamus1red commented Mar 23, 2024

Uh oh!

louis-lau commented Mar 23, 2024

Uh oh!

Yannik commented Mar 23, 2024

Uh oh!

masterzen commented Mar 24, 2024

Uh oh!

tlimoncelli commented Mar 25, 2024

Support multiple formats?

An idea that would break less existing code

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KaiSchwarz-cnic commented May 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

adamus1red commented Mar 21, 2024 •

edited

Loading

adamus1red commented Mar 21, 2024 •

edited

Loading

KaiSchwarz-cnic commented May 12, 2025 •

edited

Loading