Draft: some rewrite thing

[sowelipililimute]

This PR is some sort of mess of me deciding to make the scanner have better errors, then ending up rewriting the scanner, then ending up rewriting the parser that consumes that scanner, then ending up rewriting the evaluators that consume ASTs from the parser.

[0xTim]

I've had a quick scan through and will do a more thorough review in the week - but to start it looks like a lot of the existing tests have been changed or commented out or just deleted - is this just while rewriting or are you planning on these changes being permanent?

[tonyarnold]

I'd love better error reporting in Leaf, so I'm really supportive of any efforts to achieve this @pontaoski ❤️ Thanks for your explorations!

That being said, I've had a look through the proposed changes, and it's a large amount to change in one go. There could be an amazing set of improvements in here, but it's putting the onus on the reviewer to wade through thousands of lines of code and unstructured commits to work that out.

Are you willing to consider breaking the PR into a series of smaller changes and improvements? Perhaps there is a logical sequence of individual improvements that could be separated out and proposed as individual PRs in a chain?

I'm happy to help do this if you'd like.

[sowelipililimute]

in my experience regarding error reporting in parsers, you kinda need the capabilities to be worked in from the start of the design, otherwise redoing the whole thing from the ground up is much easier to understand, since you only need to validate that a brand new lexer+parser is doing the right thing is way easier than trying to validate that you've correctly hacked on better error reporting to an existing design.

additionally, unfortunately with the way leaf is designed, there was an extremely tight coupling between the data structures involved in the lexer+parser+renderer+evaluator, so any change of lexer+parser necessitates a lot of changes in the latter.

that being said, i've adjusted the file structure to make the git diff render all of the rewrites as new files instead of attempting to diff with the files they replace, which should help with legibility.

[tonyarnold]

That makes sense. I also appreciate that you filed this PR so that these conversations could happen around how best to integrate your changes, so thank you for being open to that.

To get this PR to a point where it can be properly reviewed and merged, I think there are a few things that need to happen:

• Tidy up the PR description/title/etc - as you saw with Adds a new #with() tag to make it easier to embed and extend tags #111, those get used in the release notes when PRs are merged. If you're confident in the changes you've made and their effect, I think you can describe them succinctly and clearly with that purpose in mind 👍🏻
• Let @0xTim know via the inline review comment about the tests that have been modified, removed and commented out - I assume that's all just part of the churn of you exploring these changes.
• If you can, please separate out deliberate syntax changes and new features such as the requirement for parentheses and the nil boolean handling into separate PR(s). Correct me if I'm wrong, but your goal here was a rewrite of the parser/lexer/etc to provide a better design, with improved error reporting? If that's the case, let's focus this PR on just that, and move anything that doesn't work toward that goal into their own PRs.

Sorry for the slow replies today, it's been a long work day.

[sowelipililimute]

poke