Lexing Teal.

[daelvn]

Hello,

I'm making a project that involves lexing different Lua-based languages, and Teal is one of them. I discovered the lpeg-lexer branch on this repository and was surprised to find an already working lpeg.re-based lexer, which is miraculously exactly what I was looking for. However, upon closer inspection, I noticed that the lexer doesn't include any such keywords like is, record, enum, type... These are instead treated as identifiers. Is this how the lexer is supposed to be used?

Apart from lexing Teal using lpeg.re, I am looking to extend the leafo/lua-syntaxhighlight repository with Teal support, for HTML highlighting, and while it isn't complicated, I need to know what are all changes over base Lua that Teal makes, apart from keywords.

As a third point: how easy would it be to repurpose the lpeg-lexer to be a simple Lua lexer? Is it simply about removing : as an operator? What else would it involve?

Thank you.

Update: Seems like this has escalated a whole lot! I thought I could get away with only the lexer, but it turns out that I also need a parser and ideally, to be able to extract type information (as detailed in #180). Teal becoming modular might help a lot with what I want to do.

[euclidianAce]

I noticed that the lexer doesn't include any such keywords like is, record, enum, type... These are instead treated as identifiers. Is this how the lexer is supposed to be used?

Since Teal is built on top of Lua, the decision was made to be able to maintain compatability, only in specific circumstances are those words recognized as keywords. So the following is entirely valid Teal, (but shouldn't be encouraged)

local record = "hi"
print(record)
global global = "local"
print(global)
local global, enum = "global", "enum"
print(global, enum)

I am looking to extend the leafo/lua-syntaxhighlight repository with Teal support

Is this all you are planning? You can probably get away with just saying those are keywords in the lexer then.
If you want to do more, the lexer is exposed in the form of tl.lex(string): {Token}, {Token}, (where the second array are the errors) so if it's okay to have Teal itself as a dependency you can just use it.
Or if I can do some self advertisement, if you need to do some parsing you can also use teal's parser via tl.parse_program or tl.process_string, or I also recommend tree-sitter. I maintain the teal parser here and lua bindings (with teal typedefs) here

[daelvn]

Since Teal is built on top of Lua, the decision was made to be able to maintain compatability, only in specific circumstances are those words recognized as keywords. So the following is entirely valid Teal, (but shouldn't be encouraged)

Oh I see, so it is the parser that makes a difference between parsing Lua and parsing Teal, right? Would there be a way to force teal.parse_program to use Lua, or is it not apt?

Is this all you are planning?

Well, the actual plan is more contrived; the plan is to parse Teal (and other Lua-based languages), extracting comments as part of the AST preferrably, to automatically generate documentation for Teal projects, with the possibility to also extract types and include them in the documentation. That alone sounds like the simplest part, in comparison to the other part I'd need a parser for: generating highlighted source code that isn't only colored, but also links identifiers to their definitions, shows a variable's type on hover... It's basically documentation generation but much more involved, and similar to Haddock. Now that I know the parser is exposed, I will probably use that, but I would also appreciate being able to query the types of certain identifiers to include them in the result, without having to infer the types myself.

I also recommend tree-sitter. I maintain the teal parser here and lua bindings (with teal typedefs) here

Will definitely check out! Thank you.

[euclidianAce]

generating highlighted source code that isn't only colored, but also links identifiers to their definitions, shows a variable's type on hover... It's basically documentation generation but much more involved

tree-sitter was practically built for that kind of stuff. For example neovim is experimentally using it for syntax highlighting and there is some decent semantic highlighting plugins out there using it as well as some documentation generation. And a more concrete and relevant example is how my own teal cli finds dependencies using ltreesitter
the following code extracts calls to require with a string literal argument:

local ts <const> = require("ltreesitter")
local teal_parser <const> = ts.require("tree-sitter-teal-parser", "teal")

-- ...

-- tree-sitter queries are written in a lisp that matches against the trees the parser generates
local dep_query <const> = teal_parser
   :query[[ (function_call
             . (identifier) @func_name
             . (arguments (string) @module_name)
             (#eq? @func_name "require")
             (#insert_mod_name! @module_name)) ]]

local function get_dependencies(file_content: string): {string}
   local tree <const> = teal_parser:parse_string(file_content)
   local modules <const>: {string} = {}

   dep_query:with{ ["insert_mod_name!"] = function(mod_name: string)
                table.insert(modules, mod_name)
             end}
            :exec(tree:root()) -- run the query over the parsed tree

   return modules
end

and I think I put a very simple "doccomment extractor" somewhere in the docs for ltreesitter