LINE SEPARATOR U+2028

[Ekmansoft]

Is the unicode character LINE SEPARATOR (U+2028) (aka UTF-8 0xE2 0x80 0xA8) allowed in a UTF-coded GEDCOM 5.5.x file?

Geni.com can create such files for download if the character is included in some INDI.NOTE sections.

I think this should not be allowed, but what do you say?

[tychonievich]

Is U+2028 allowed? Yes, it has been allowed since Unicode support was introduced in GEDCOM 5.3. U+2029 (paragraph separator) is also allowed.
In versions 5.x U+0085 (next line) was also allowed, but support for that control code was removed in 7.0.0.

I think this should not be allowed

Why not? I don't immediately see any problems caused by supporting these characters, but I'm assuming you do?

My opinion is that applications should should not modify user-supplied content without their consent unless what they supplied violates the spec. If a user knew enough to enter U+2028 in a payload, I think an application should assume the user wanted that specific character.

I have much less opinion about the merits of having U+2028 and U+2029 supported in future versions of the spec. They have do distinct semantic meaning that is not explicit in CONT and that meaning can have impact on semantically-aware algorithms and user interfaces, but very few applications seem to take advantage of that semantic difference today so we could probably prohibit them with minimal disruption if necessary.

[Ekmansoft]

In principle I agree, but I will try to explain the reason why I started this topic.

First, the reason why this character appears in these comments is that some user has uploaded a gedcom from some genealogy software that uses it for some automatic formatting. When visible on the geni.com website it is shown as an (unknown) control character, and not affecting formatting.

Second, when loading such a file into an editor (say visual studio code for example), it complains that the file has inconsistent line feeds present.

I agree that there is probably no need to affect older specs, but in future versions of 7.0 and later I think it would be preferable to format characters like this in some other way.

[dthaler]

My view: I don't think the GEDCOM spec should change to disallow specific Unicode characters based on what appear to be bugs in individual applications/sites, that feedback should be directed to the buggy application/site. If there is an argument that a Unicode character is generically problematic for ALL apps/sites, then it should be disallowed. So far it sounds like this issue is in the former category.

[Ekmansoft]

Hmmm... How does it work for other special formatting Unicode characters?
I'm not saying it should be forbidden, but maybe it should be escaped in some way so it's not interfering with linefeeds?

[tychonievich]

We currently ban all but a specific few Cc-class characters, which includes the special formatting characters that are not combiners or spacers. Other characters we allow, and leave it to applications to handle them appropriately.

Because the set of unicode characters is so large, it is unfortunately not surprising that a pair of applications (in this case geni.com and VSC) handle a few of them differently, a problem that is shared by almost all text-processing applications including email clients, web browsers, text editors, word processors, etc. While I'd love to be able to fix this, I don't think we're likely to make more progress than others have.