@milisims what's the state of this PR? Should I maybe give it a test or it's still WIP?

The flattening of (expr) into (sym) (str) and (num) is definitely concrete and staying, and (sym) has the anonymous nodes for ascii symbols still, so those changes will be stable.

For the other change, my goal is to facilitate querying ambiguous markup. What I did here was add the "pre" "mid" and "post" signals (empty nodes) as a part of (sym). Basically, if the symbol is immediately before, in between, or after alnum characters, the relevant anonymous node is shown. So a bold query can be simplified to (paragraph (sym "*" "pre") @start (sym "*" "post") @stop).

However, right now /*a*/ is parsed as (sym "/").(sym "*" "pre").(str).(sym "*" "post").(sym "/"), which has pros and cons. This lets the user be explicit easily about allowing or not the use of double markup like that, but if they do want double markup then it renders the "pre" and "post" kind of useless. Additionally, it's already not helping parsing objects like links, because the double symbols.

So, there's three solutions.

1. Drop the pre/mid/post signals, or leave it as is
2. Propagate the signals: /*a*/ is parsed as (sym "/" "pre").(sym "*" "pre").(str).(sym "*" "post").(sym "/" "post")
3. Add a pre/post field (sym "char" pre: "str/sym/num", post: "str/sym/num")

Since it's optional and very cheap to have the signals, I'm not really interested in dropping them. The second is a barely more expensive, but I think it's simplest to use/understand. 3 gives the most control, but take for example: a]/*a*/. Option 2 says the symbols in the middle are "mid", option 3 requires additional processing to figure it out. I like the idea of some combination of propagating signals and using a pre/post field, because they can be negated usefully in a query.

Currently I'm just writing some play queries to figure out a scheme that makes the most sense, if there is one. Have any thoughts?

From the given options, I think I would prefer option 2, but option 3 would also be helpful. My current implementation works, but it's far from perfect and has a bunch of bugs and edge cases. I'm looking for a way to simplify it, and both of these options should help, but option 2 seems like I might be able to drop custom parsing completely.

Whichever way you choose, even the currently implemented one, should be helpful to some extent.

I started preparing a branch with these changes, and I ran into one issue with tags parsing.
This content:

* Test :tag:
  - Test

Is parsed like this:

(document [0, 0] - [2, 0]
  subsection: (section [0, 0] - [2, 0]
    headline: (headline [0, 0] - [1, 0]
      stars: (stars [0, 0] - [0, 1])
      item: (item [0, 2] - [0, 12]
        (str [0, 2] - [0, 6])
        (sym [0, 7] - [0, 8])
        (str [0, 8] - [0, 11])
        (sym [0, 11] - [0, 12])))
    body: (body [1, 0] - [2, 0]
      (list [1, 0] - [2, 0]
        (listitem [1, 2] - [2, 0]
          bullet: (bullet [1, 2] - [1, 3])
          contents: (paragraph [1, 4] - [2, 0]
            (str [1, 4] - [1, 8])
            (nl [1, 8] - [2, 0])))))))

But if it's a single line:

* Test :tag:

It parses it properly:

(document [0, 0] - [1, 0]
  subsection: (section [0, 0] - [1, 0]
    headline: (headline [0, 0] - [1, 0]
      stars: (stars [0, 0] - [0, 1])
      item: (item [0, 2] - [0, 6]
        (str [0, 2] - [0, 6]))
      tags: (tag_list [0, 6] - [0, 12]
        tag: (tag [0, 8] - [0, 11])))))

Generally, it seems that adding a second line with any type of content immediately breaks parsing the tags.

Tags issue is fixed, thanks!
I ran into another one with checkboxes and links:

This content:

- [[Test]]

Generates this tree:

(document [0, 0] - [1, 0]
  body: (body [0, 0] - [1, 0]
    (list [0, 0] - [1, 0]
      (listitem [0, 0] - [1, 0]
        bullet: (bullet [0, 0] - [0, 1])
        checkbox: (checkbox [0, 1] - [0, 9]
          status: (status [0, 3] - [0, 8]))
        contents: (paragraph [0, 9] - [1, 0]
          (sym [0, 9] - [0, 10])
          (nl [0, 10] - [1, 0]))))))

It treats the link as a checkbox