CRLF issue on Windows - request for some advice on fixing it

[abmclin]

Hello, I recently compiled Scryer under msys64 on my Windows machine and ran into an issue that is already documented in #553 and #543. The implementation doesn't handle Windows EOL CRLF correctly. I stepped through the Rust code in my debugger and tracked the issue to the next_token function in lexer.rs. The actual error that is generated when Scryer tries to read a file (either .scryerrc when saved with CRLF) or any file loaded via [<filename>] is coming from get_back_quoted_string which initially caused me some confusion since my test file doesn't have back quoted strings. Eventually I realized why get_back_quoted_string is being called is because it's the last clause in the series of if statements in name_token so when next_token doesn't recognize the \r character of a CRLF marker, it delegates to name_token to figure out \r, the since the if conditions don't match \r the last clause, namely get_back_quoted_string gets executed which is what generates the ultimate error.

While I can work around this issue by saving my files using the Unix convention, I thought I'd take a crack at fixing this problem and opening a pull request. But I'd like some advice on the best way to fix this.

I can see two possible ways here;

1. Fix next_token so when it sees a \r it needs to skip it and check the next character to see if it's a \n then it's recognized as end of line and proceeds from there.

2. Arrange it so whenever a file is read into memory, all CRLF markers are converted to LF so the lexer processes it correctly.

Fix no. 1 seems better to me, but do you have any preferences or a better idea?

[UWN]

(I am not a Wind- ows user) but maybe this helps.

It seems that the syntax in general needs to be extended. There, just allow all control characters as layout characters. In 6.4.3 Layout characters there is:

layout char (* 6.5.4 *)
   = space char (* 6.5.4 *)
   | horizontal tab char (* 6.5.4 *)
   | new line char (* 6.5.4 *) ;

So at first sight, there must not be any further layout character. On the other hand, the actual PCS (6.5) may also contain extended characters and in fact, note that the control escape sequences "\a\b\r\f" are not necessarily in the PCS (with ASCII and Unicode they are present). Only "\t\n" have to be in the PCS and thus the rather restricted definition of layout char. Currently, in Scryer:

?- member(C,"\a\b\r\f\t\n"), catch(number_chars(N,[C|"9"]), error(Error, _), true).
   C = '\a', Error = syntax_error(unexpected_char)
;  C = '\b', Error = syntax_error(unexpected_char)
;  C = '\r', Error = syntax_error(unexpected_char)
;  C = '\f', N = 9
;  C = '\t', N = 9
;  C = '\n', N = 9
;  false.

SICStus, GNU, Trealla all give N = 9 and thus accept these control characters as layout. IF and Ichiban do not accept \a and \b as layout characters but otherwise also N = 9.

Scryer already accepts \v, so why not the others too.

So for regular syntax this could solve the problem.

The other one that still persists are linebreaks within single quoted or double quoted characters. Thus an atom like:

?- 'The first line;\
and its continuation' = X.
   X = 'The first line;and its continuation'.
?- 'The first line;\
and its continuation' = 'The first line;and its continuation'.
   true.

Here,

continuation escape sequence (* 6.4.2 *)
   = backslash char (* 6.5.5 *),
     new line char (* 6.5.4 *) ;

could be extended to accept also layout characters in between. This would solve also the 3270 (and similar devise) issue with spaces at the end of lines.

[UWN]

(Scryer already supports both \f and \v)

[abmclin]

Thanks for the examples! I'll start writing up test cases to help me figure out how to address the above issues. I'm not sure what you meant by "6.4.3" or "PCS" though, may you clarify please?

[EricGT]

not sure what you meant by "6.4.3"

6.4.3 is a section number from

ISO/IEC 13211-1:1995. Part 1: General core. Published 1995-06-01. (ref)

It is never free and the price seems to have jumped in the last few years.

UWN can correct me on this but I think each member nation is allowed to set the price and the U.S. site usually has the best prices.

https://webstore.ansi.org/Standards/INCITS/INCITSISOIEC132111995R2017 Currently $116

It is nice to have when the discussions reference it but not strictly necessary as many Prolog programmers who visit these sites have a copy and can answer questions about it.

Here is a bit more and the notes might be of benefit.

A control escape sequence denotes the control character indicated by the name of the symbolic control char, iff that control character is an extended character of the processor character set (6.5).

control escape sequence (* 6.4.2.1 *)
  = backslash char (* 6.5.5 *),
   symbolic control char (* 6.4.2.1 *) ;

symbolic control char (* 6.4.2.1 *)
  = symbolic alert char (* 6.4.2.1 *)
  | symbolic backspace char (* 6.4.2.1 *)
  | symbolic carriage return char (* 6.4.2.1 *)
  | symbolic form feed char (* 6.4.2.1 *)
  | symbolic horizontal tab char (* 6.4.2.1 *)
  | symbolic new line char (* 6.4.2.1 *) 
  | symbolic vertical tab char (* 6.4.2.1 *) ;

symbolic carriage return char (* 6.4.2.1 *)
  = "r" ;

NOTES
1 A new line char is not allowed in a quoted character.

2 \ cannot be followed by a space in a quoted token, and a new line char occurs in a quoted token only as part of a
continuation escape sequence (6.4.2), so an atom ‘a\ 
b' does not conform to this syntax unless \ is followed by a
new line char in which case the atom is equivalent to the atoms ’ ab' and ab.

3 The representations of the symbolic control characters are those recommended by the International Standard for C (ISO/IEC 9899).

also

6.5.4 Layout characters

new line char (* 6.5.4 *)
= implementation dependent ;

as for PCS, it seems to be processor character set (PCS) which is noted in the ISO standard.

I don't plan to keep quoting more as it eventually leads to quoting the entire document. It is an interesting read but I don't know of anyone who has followed the standard to the letter.

---

It seems to boil down to this. If you want to know what the standard states and be able to defend what you write with quotes from the standard then you will have to get a copy of the standard.

[UWN]

Getting the document of 1995 as an INCITS document (without the corrigenda) costs exactly USD 30. See instructions here. And get the "historical" version. There are many other ways to get the standard, and prices vary up to a factor of 20.

[UWN]

a suitable substitution for the actual ISO standard?

No. There are actual differences for built-ins, Its preparation was finished before the actual standard.

[EricGT]

At

https://global.ihs.com/

I find the historical one for $30

https://global.ihs.com/doc_detail.cfm?&input_doc_number=INCITS%2FISO%2FIEC%2013211%2D1%3A1995%20%28R2007%29&input_doc_title=&document_name=INCITS%2FISO%2FIEC%2013211%2D1&item_s_key=00461478&item_key_date=040630&origin=DSSC

and the the full one for $60

https://global.ihs.com/doc_detail.cfm?&item_s_key=00461478&item_key_date=921231&input_doc_number=INCITS%2FISO%2FIEC%2013211%2D1%3A1995%20%28R2007%29&input_doc_title=

which is better than $116.

The way I see the difference is that you are paying an extra $30 to have the free parts bundled together. I paid for the historical one and then downloaded the other free parts.

---

As for

ISO/IEC 13211-2:2000. Part 2: Modules.

I would not get it unless needed. I do not have a copy and have never needed a copy, AFAIK no Prolog uses it for implementing modules.

[UWN]

As for modules, the best seems to be to go for SICStus. 13211-2 permits various instances many of which have not found any reception.

[abmclin]

I've downloaded the $30 historical version and the three free corrigendas. I think I'm set now, thank you for the details.