Tagless multiline strings

[MichalMarsalek]

Two years ago a created a nested format with strings, lists and dictionaries for my personal use that was surprisingly similar to NestedText. While it was a bit less elegant than NestedText and still required some escaping, there was one thing that was imo nicer than in NestedText - it supported unannounced multiline strings. In my understanding from quickly reading the docs, I don't think there would be any ambiguity in allowing

key: a dictionary value
     that spans
     multiple lines
another key: another value

and

- a list value
  that spans
  multiple lines
- another value

as shorter alternatives to

key:
  > a dictionary value
  > that spans
  > multiple lines
another key: another value

and

-
  > a dictionary value
  > that spans
  > multiple lines
- another value

I'd love if this was added into the syntax if I'm not missing any clashes.

PS: I added NestedText to https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats .¨
EDIT: Well it was reverted. Apparently it needs a Wikipedia page first.

[KenKundert]

Michal,
We considered this approach to multiline strings when we were first coming up with NestedText and rejected it for the following reasons:

1. The indentation for the multiline string value in a dict item would be excessive for long keys. This causes two problems. If one was trying to maintain a reasonable text width, the multiline string would be squashed into the few remaining columns. Also, each value in a dictionary would have an indent level that depended on the length of its key. For example:

       This is a very, very, very, very, very, very, very, very long key: Due to
                                                                          the
                                                                          indent
                                                                          this
                                                                          value
                                                                          starts
                                                                          in
                                                                          column
                                                                          75 and
                                                                          so can
                                                                          can
                                                                          only
                                                                          contain
                                                                          one or
                                                                          two
                                                                          words
                                                                          in
                                                                          each
                                                                          line.
       another key: Another value. This one has much more space than the 
                    previous one.  Here the string is can be much wider because 
                    the indent is much smaller.  But it might be confusing that 
                    two values in the same dictionary have vastly different 
                    indentation levels.

2. There is no clear demarcation of the end of the string, so one cannot distinguish between blank lines at the end of the value and blank NestedTest comment lines that follow the strings. We would have to choose between allowing multiline strings that end in blank lines and allowing blank lines as comments in the NestedText file. We could not support both.

3. I don't believe I can configure Vim to automatically indent the second line of the multiline value in the way needed by your proposal.

The approach we took (using the string tag '>') does not suffer from any of these problems, and also allows us to support blank lines as comments within a multi-line string. For example:

# part one ...
> The first line of the multiline string.

# part two ...
> The second line of the multiline string

As you say, your current proposal does not conflict with the existing syntax, so it would be possible to support both approaches. But we are really trying to keep the language small and simple, and having multiple ways of doing the same thing conflicts with that goal.

Thank you for your suggestion. I understand you desire to find an alternative to the current key-based approach to multiline strings. It seems artificial to me too. But despite spending considerable time on it, I have yet to come up with anything that provides one simple solution to all of the problems I considered.

-Ken

[MichalMarsalek]

Ken,
thank you for taking the time to reply.

1. There's no reason to require the continuation lines' indentation to match the start of the first line of the text value. Just as anywhere else in NestedText, the amount of the added indentation is arbitrary, so a user might chose to align it, but usually you would stick to your normal 2-4 spaces.

another key: Another value. This one has much more space than the 
  previous one.  Here the string is can be much wider because 
  the indent is much smaller.  But it might be confusing that 
  two values in the same dictionary have vastly different 
  indentation levels.

2. While I'm aware, that > allows you to do that, not allowing comments in the middle of a string value is not something I would consider a dealbreaker. Note that empty lines outside of a multiline string context would not be affected. I don't really know any language that does this (manually joining multiple single line string literals does not count).
   If a user desires to write comments inside their string values, they can utilize the > approach (which I'm not suggesting to remove - not only for backwards compat but also my suggestion doesn't cover all strings).
3. See 1.

But we are really trying to keep the language small and simple, and having multiple ways of doing the same thing conflicts with that goal.

Well there are already multiple ways to do the same things. And that's fine. One way that's really short and elegant and works for most cases and another that's a bit more verbose, but works even for all the edge cases (like is the case for dictionaries). When I was thinking about the necessary changes to the (Javascript) parser, I came to the conclusion that it would be reasonably simple.
Yes, you would no longer be able to determine the type of each line individually using its tag, but that's fine imo.
What you need to do is:

1. Create a new line type - "string continuation".
2. When you encounter a line that follows a "list item"/"dictionary item" with "end of line string" value line and the current line is larger indentation, you flag the current line as "string continuation". You can do that since otherwise, an indentation increase is illegal here.
3. When you encounter a line that follows a "string continuation" line with the same indentation, you flag the current line as "string continuation" line. (If the current line has higher "apparent indentation", you treat the "added" indentation as verbatim spaces and reset the indentation to that of the previous line).
4. In other cases, parse the line as you do now (including detecting comment and empty lines which is only done now).
5. You check the indentation levels and combine the lines just as you do now.

The fact that the Pretty printing section even exists in the docs imho really supports my case. The multiline strings serialization is the only thing that is not pretty in NestedText.

[MichalMarsalek]

I just realised that empty/comment lines are still possible inside a tagless multiline string, it just has to be indented less than the string:

key: First line of a multiline string value.
     Second line of a multiline string value.
# Since this line cannot be a part of the string value, it is a comment.
     Third line of a multiline string value.

That being sad I would maybe find this form a bit confusing.

[MichalMarsalek]

I see that there's no .Net library linked. I can write and maintain one. Maybe this weekend. I will probably include this proposal as a toggleable option to experiment with it a bit and see how it works. If my proposal will not end up getting added to the standard, my library will of course by default work as per the standard (not using to emit, throw on parse), but I will be able to use it with this proposal for my personal projects by switching a boolean flag.

[LewisGaul]

I just realised that empty/comment lines are still possible inside a tagless multiline string, it just has to be indented less than the string:

key: First line of a multiline string value.
     Second line of a multiline string value.
# Since this line cannot be a part of the string value, it is a comment.
     Third line of a multiline string value.

Just because you can, doesn’t mean you should :)

From https://nestedtext.org/en/latest/file_format.html:

Each line in a NestedText document is assigned one of the following types: comment, blank, list item, dictionary item, string item, key item or inline. Any line that does not fit one of these types is an error.

A fundamental part of the design is that you can tell what type of line you're looking at without the context of any of the surrounding lines. This holds true whatever you have in your string, e.g. a line: > foo: bar is part of a multiline string, but if you drop the > it could be a dict key-value, and this is only determinable by attributing further significance to the preceding whitespace in the context of previous lines. This fact of the design makes parsing simpler, both for software and for humans.

[MichalMarsalek]

Just because you can, doesn’t mean you should :)

Yes, I agree, as I said I find this confusing.

A fundamental part of the design is that you can tell what type of line you're looking at without the context of any of the surrounding lines.

Sure, I am aware of this, I even mention this in my comment.

It's common for languages to have parser modes. Once you enter a multiline string, some things behave slightly differently and that's fine. I don't think my proposal would make it harder for humans to read as it strives to be very intuitive.

Also, I'm not sure it is that fundamental. Sure it holds (perhaps as a consequence of making sure no escaping is needed) and is kinda convenient, but making an exception seems like nothing fundamental would break in the long run. Well, certainly it is less fundamental than "no scalar types except strings supported" or "no escaping needed".

[KenKundert]

Perhaps I am not being clear. So let me take a different approach.

Consider this example.

key 1: line 1 of value 1
       line 2 of value 1

key 2: value 2

Is the blank line that follows line 2 of value 1 included in the value or not?

In other words, what is the value of key 1? Is it ...

> line 1 of value 1
> line 2 of value 1
>

Or is it ...

> line 1 of value 1
> line 2 of value 1

As a second question, you say that the indentation of the subsequent lines need not match the indent level of the first line. So this is valid:

key 1: line 1 of value 1
    line 2 of value 1

Okay, if that is valid, then how does one represent an out-dented paragraph?

Consider this example:

aa: header ...
    point 1
    point 2

Does this represent:

aa:
    > header ...
    > point 1
    > point 2

or does it represent:

aa:
    > header ...
    >     point 1
    >     point 2

[MichalMarsalek]

1. The latter. Lines with decreased indentation are never part of the value. They are either ignored (if they are white space only or # comments) or end the value (otherwise).
2. The former. My proposal is not meant as a replacement of > - I'm aware that it cannot represent all string values.

[KenKundert]

But didn't you say

There's no reason to require the continuation lines' indentation to match the start of the first line of the text value. Just as anywhere else in NestedText, the amount of the added indentation is arbitrary, so a user might chose to align it, but usually you would stick to your normal 2-4 spaces.

Doesn't that contradict your # 1 response?

Isn't that what I did with this example:

key 1: line 1 of value 1
    line 2 of value 1

[MichalMarsalek]

I'm sorry what exactly are you referring to? What are the two statements that are contradictory?

[KenKundert]

The latter. Lines with decreased indentation are never part of the value. They are either ignored (if they are white space only or # comments) or end the value (otherwise).

This is a really difficult conversation because your proposal is so squishy and poorly defined. That is why I am trying to communicate with concrete examples. Allow me to try again. You gave the following example above:

another key: Another value. This one has much more space than the 
    previous one.  Here the string is can be much wider because 
    the indent is much smaller.  But it might be confusing that 
    two values in the same dictionary have vastly different 
    indentation levels.

What does that actually represent? It could be:

another key: 
    > Another value. This one has much more space than the 
    > previous one.  Here the string is can be much wider because 
    > the indent is much smaller.  But it might be confusing that 
    > two values in the same dictionary have vastly different 
    > indentation levels.

Or it could be:

another key: 
    > Another value. This one has much more space than the 
    >     previous one.  Here the string is can be much wider because 
    >     the indent is much smaller.  But it might be confusing that 
    >     two values in the same dictionary have vastly different 
    >     indentation levels.

Which is it? And why should it be one rather than the other? It seems ambiguous to me.

[MichalMarsalek]

Communicating with concrete examples is fine by me, but if you prefer, I can try to write a formal spec instead, although I believe most questions should be clarified by the pseudocode I provided in my comment.

another key: Another value. This one has much more space than the 
    previous one.  Here the string is can be much wider because 
    the indent is much smaller.  But it might be confusing that 
    two values in the same dictionary have vastly different 
    indentation levels.

just as

another key: Another value. This one has much more space than the 
             previous one.  Here the string is can be much wider because 
             the indent is much smaller.  But it might be confusing that 
             two values in the same dictionary have vastly different 
             indentation levels.

is the equivalent of

another key: 
    > Another value. This one has much more space than the 
    > previous one.  Here the string is can be much wider because 
    > the indent is much smaller.  But it might be confusing that 
    > two values in the same dictionary have vastly different 
    > indentation levels.

Anything else would violate

the amount of the added indentation is arbitrary

Especially your other option for its meaning

another key: 
    > Another value. This one has much more space than the 
    >     previous one.  Here the string is can be much wider because 
    >     the indent is much smaller.  But it might be confusing that 
    >     two values in the same dictionary have vastly different 
    >     indentation levels.

doesn't make any sense to me. The spaces just denote a block (indentation). How could they be part of the resulting value? That would imply

another key: Another value. This one has much more space than the 
previous one.  Here the string is can be much wider because 
the indent is much smaller.  But it might be confusing that 
two values in the same dictionary have vastly different 
indentation levels.

is equivalent to

another key:
  > Another value. This one has much more space than the 
  > previous one.  Here the string is can be much wider because 
  > the indent is much smaller.  But it might be confusing that 
  > two values in the same dictionary have vastly different 
  > indentation levels.

which is just some ambiguous nonsense.

[KenKundert]

Okay, so how do you represent the following?

key 1:
    >
    > value 1
key 2:
    > value 2 line 1
    >     value 2 line 2
key 3:
    >
    > - value 3
key 4:
    >
    > value 4: four

[MichalMarsalek]

None of those are representable using the proposed syntax, so you'd fallback to the general > version.

[MichalMarsalek]

You already have two ways to represent strings:

key: string value

and

key:
  > string value

My proposal is to take the one that cannot represent every string (the first one) and make it able to represent some more kinds of strings by allowing the value to continue on the next line (but of course it won't be every string).

Both in the current version and in a version where my proposal is implemented, the following statements hold:

1. There are two ways to represent a string (some strings).
2. The first way doesn't cover all strings, while the second does.

So these two points shouldn't be reasons for rejecting the idea.

This is different than #34 which (apart from all 3 suggested syntaxes being ambiguous) suggested a third syntax. My proposal just extends one of the syntaxes by making use of a syntax which is currently illegal (a nested block following a list/dictionary item) and thus has no assigned meaning.