raw string literals not preserving newlines, as suggested in docs

[benatkin]

I noticed that raw strings aren't copying the newlines as is suggested in the docs. The relevant code seems to be here: https://github.com/rhaiscript/rhai/blob/main/src/tokenizer.rs#L1265

The docs suggesting it does preserve the newlines: https://rhai.rs/book/ref/strings-chars.html#raw-strings

Reproducing this behavior with rhai-run, installed with cargo install rhai:

# cat > printit.rhai
let hello = ###"Hello

world"###;

print(hello);
# rhai-run printit.rhai
Helloworld
#

Note that Helloworld is just one word, not separated by newlines.

An example with backquotes where the newlines are preserved:

# cat > backquote.rhai
let hello = `Hello

world`;

print(hello);
# rhai-run backquote.rhai
Hello

world

This is too bad because it makes it less convenient to embed Markdown with fenced code blocks, where each of the 6 backquotes per fenced code block would need to be duplicated.

I get the idea that raw strings are pretty new and that's why this hasn't been noticed yet. I definitely think it should preserve newlines, so the code should be updated rather than the docs. I also think because it's new and has been documented like this, it ought to be OK to put it in a minor release. It probably shouldn't go into a patch release because it's definitely a change of behavior.

[schungx]

Yes this is definitely a bug.

The idea of raw strings is that anything in between comes out verbatim.

I think we probably missed testing the multiline case.

Do you know how to fix it? If so you can open a PR.

https://github.com/rhaiscript/rhai/blob/main/src/tokenizer.rs#L1267

Seems to be this line that forgot to store the newline into the buffer.

Otherwise I'll have to wait until I'm free to get to it...

[schungx]

Fixed in #941. Please check.

[benatkin]

Installed from git and confirmed that it fixes the issue! Thanks!

A release would be nice but for now I'll have cargo install from git for what I'm working on.