[Suggestion] Use a parser generator to formalize the sourcepawn grammar

[Sarrus1]

Right now, the LSP behind my vscode extension uses tree-sitter to generate the AST of sourcepawn files. It works very well, but if the sourcepawn syntax changes, I have to edit the grammar file.

Would it be possible to use the parser generated by tree-sitter for spcomp instead of a custom implementation?

I have also created a lexer for sourcepawn.
As well as a preprocessor.
Here is the link to the tree-sitter grammar edited by @nilshelmig and me.

[dvander]

This is an impressive work, nicely done.

Unfortunately, the opportunity to borrow this (or another parser-generator tool) for spcomp is low. The pipeline from "character" to "token" to "AST" is very, very intertwined since SourcePawn is such an idiosyncratic language. Macro expansion and preprocessing happen during lexing. It's not a separate pass. Stuff like newdecls requires a bizarre amount of lookahead to deal with source compatibility.

Worse, the current design is actually too compartmentalized. It needs to be even more intertwined.

Issue #844 is why. Our semantic analysis pass has no sane way to modify the AST to fix this bug. Most languages have immutable ASTs, either by (1) performing all analyses as part of parsing (clang does this), or (2) building a complete new AST after each pass.

I'd like to move the compiler in the direction of (1), so that the parser is very tightly coupled with semantics.

That said, it's been (and being) designed so that steps after parsing are optional. So if there is any way we can output or format the AST that is useful for your tooling, let me know, we can probably do it.

[assyrianic]

why not use my grammar doc from sptools?

[Sarrus1]

why not use my grammar doc from sptools?

The tree-sitter grammar handles both the old and new syntax.

[assyrianic]

why not use my grammar doc from sptools?

The tree-sitter grammar handles both the old and new syntax.

waste of effort to support old syntax, especially when it's gonna be removed in the next SourceMod transition and future.

[Sarrus1]

Some includes are still using the old syntax. Not supporting it would mean no intellisense for those includes.