Thoughts / Concerns / Questions

[vitiral]

Hey, I really love the language. I particularly like how it preserves Lua's design and "feel" while addressing the most common issues.

I'm working on some of my own languages (http://civboot.org) and Lua libraries, including a recursive-descent parser library (https://github.com/civboot/pegl) which in a 370 line library file makes expressing Lua's syntax a breeze -- Lua's syntax is expressed in a bit more than 200 lines of code (not quite complete but I think I'm close).

I'm also writing my own "base library" that includes a type system (https://github.com/civboot/civlib/blob/main/lua/civ.lua but don't read the code, I was learning Lua while I wrote it), the basics of which is something like:

-- creates a new type with `__call` overridden to be a "constructor" which calls `setmetatable(t, MyType)`
-- MyType table has a few fields like `#tys` which hold the types of the fields
-- runtime type checking can be enabled/disabled through a global flag (changes which `__call` gets set,
-- so zero runtime performance cost)
MyType = newRecord("MyType", {field1=Num, field2=List(Int)})

-- creating methods is done as normal: it's just a metatable
MyType.myMethod = function(self, a) ... end

I have a few thoughts/suggestions

• My PEGL library makes writing a recursive descent parser (with decent error messages I think!) much more concise and readable, while the library itself is very lean (300 LoC). I noticed that tl.lua spent about 3000 lines parsing, I think with PEGL it would be more like 300-400 (plus another 300 for the PEGL library itself if you wanted to stay in a single file) -- I currently parse Lua itself in a bit more than 200 lines (although it doesn't quite work yet).

• Looking at tl.lua I see no way to declare a TL type for lua to use. I'm looking for something like newRecord above. This is confusing to me, since one of the big benefits of including something like TL into your lua project would be the ability to stay with Lua for some things but get the benefits of expressing types (with auto setmetatable and string formatting etc provided)

• In general I see very few library functions exported by TL. Is there a function for standard record formatting? For record equality? Does TL even use setmetatable?

• https://github.com/teal-language/tl/blob/master/docs/developing.md seems to indicate that TL is bootstrapping itself. Why? Isn't lua a "good enough" language for writing a compiler in? I have a hard time believing the win is worth the overhead mentioned in that doc.

[lenscas]

I currently parse Lua itself in a bit more than 200 lines (although it doesn't quite work yet).

Keep in mind that teal has to parse teal code, not lua code (Or maybe it needs to do both? Forgot) so that would add some of the extra lines. In addition, you would be surprised how many extra lines some of the finer details can bring. So depending on where your parser exactly is I don't find it that big of a surpise that the teal one is so much bigger than yours.

Looking at tl.lua I see no way to declare a TL type for lua to use. I'm looking for something like newRecord above. This is confusing to me, since one of the big benefits of including something like TL into your lua project would be the ability to stay with Lua for some things but get the benefits of expressing types (with auto setmetatable and string formatting etc provided)

I am not entirely sure I understand what you exactly want... Maybe .d.tl files are what you are looking for? Though that sounds like it goes the wrong way of what you want.

In general I see very few library functions exported by TL. Is there a function for standard record formatting? For record equality? Does TL even use setmetatable?

The API exposed by teal is for the time being very limited on purpose as it would be way too unstable. Also, formatting teal code isn't something that teal should be doing anyway (I think?)

https://github.com/teal-language/tl/blob/master/docs/developing.md seems to indicate that TL is bootstrapping itself. Why? Isn't lua a "good enough" language for writing a compiler in? I have a hard time believing the win is worth the overhead mentioned in that doc.

Most languages bootstrap themselves. It is a good way to test the compiler. It also helped the maintainer already by finding type errors. IIRC they mentioned how rewriting from lua to teal fixed many bugs in the compiler due to the better typesafety and how only nil related problems are left when it comes to stuff that can be checked at compile time.

[vitiral]

formatting teal code isn't something that teal should be doing anyway

Sorry, I meant formatting teal records (like assert(tostring(myPoint) == "Point{x=42, y=7}")

I'll respond to the other points when I get an example of what I want published to luarocks :D

[vitiral]

I am not entirely sure I understand what you exactly want... Maybe .d.tl files are what you are looking for? Though that sounds like it goes the wrong way of what you want.

This is what I want: https://github.com/civboot/metaty

Lua already has a powerful (runtime) type system, it's just annoying to express types in it. IMO it would be good to build on the system that exists and have a language+syntax (like teal) that compiles to a Lua-native type library.

If teal included something like metaty then it could just compile records directly to metaty calls. Of course, it would still do it's normal type checking/etc statically, but at runtime its types would just be metaty values.

... formatting

FYI metaty has cleaner formatting of Lua values (not just records but also raw tables), a huge pain point of Lua for me personally.

Most languages bootstrap themselves (...)

I think in a lot of languages it makes sense, however when the language is just a transpiler and the language it's trans-piling to is such a powerful and expressive language I think it makes less sense.

What if instead of teal being built in teal it was built in lua with type libraries (like metaty)?

It's y'alls project so it's your decision, I just wanted to push back on that point.

Feel free to close if you like and thanks for the discussion so far!

[vitiral]

I moved the project to https://github.com/civboot/civlua/blob/main/metaty/README.md

I've added function types as well (only runtime checking right now).

[hishamhm]

Hi! Thanks for the feedback. @lenscas's original reply already covered a lot of ground, so I'll just go through your original message and add some finer points from the author's perspective.

I noticed that tl.lua spent about 3000 lines parsing

Not only parsing but also building the AST. It also keeps the project self-contained with no dependencies (otherwise I'd be using LPeg or something). I took a look at your library; it looks cool, but since it's a grammar interpreter, I don't think performance would be on par with the handwritten recursive descent parser, and our current performance isn't ideal. I probably could make the parser shorter with a metaprogramming-based generator, but then code would be dynamic and I wouldn't have the Teal type-checker to help me write it, and I find the current parser code easy to maintain.

I'm looking for something like newRecord above

Teal doesn't provide a, let's call it, "metatable-use-pattern" model of its own (to not call it an object model). Everyone has different opinions on how one of those should look like; for discussions with links to further issues, start here: #97.

Isn't lua a "good enough" language for writing a compiler in?

Well, you like having static types when coding, why can't I have them too? 😁

On a more serious note, Lua is a scripting language; I have written entire applications in it, and it is on the back of that experience that I decided to create Teal — to make a Lua-like application-development language, first and foremost, for my own use! Plus everything @lenscas said. :)

[vitiral]

Thanks! I added discussion there. And thanks a bunch for luarocks, it's awesome 😄

Not only parsing but also building the AST.

I'm confused -- don't you normally get an AST after parsing? Maybe it's not the final one (i.e. maybe you still want to do operator precedence) but it will be a tree of nodes right? (FYI that's what you would get with pegl)

it's a grammar interpreter, I don't think performance would be on par with the handwritten recursive descent parser

I would be very surprised if the performance were any different, or if any differences couldn't be fixed. For instance right now Or{... a whole bunch of keywords ...} is very slow, but I plan on using a trie-of-tables approach for keyword matching.

In general I've heard that parsing is very rarely a significant portion of the cost of a compiler, so saving 10-20% is rarely worthwhile. But even still, I think pegl should be close to hand-written recursive descent -- and of course you could unroll the bottlenecks into true recursive descent in such cases.

but then code would be dynamic and I wouldn't have the Teal type-checker to help me write it, and I find the current parser code easy to maintain.

It's a matter of taste. Personally I prefer short and readable code that automatically handles most error cases for you but I can see the value in expanding it out.

One of the reasons I wrote pegl is because I also love expanding the code out for some use-cases and pegl just boils down to a concise recursive descent parser (which you can always override with true recursive descent when needed).

[hishamhm]

I'm confused -- don't you normally get an AST after parsing?

Not by definition; parsing means traversing a grammar to determine if the input matches the language; AST-building is something else that may or may not happen simultaneously (e.g. yacc grammars generate a parser, but for building an AST you need to add C code to the production rules). It's nice if your tool does generate a tree, but a reified representation of the parse tree that describes the traversal of the grammar (which is what a generic parsing tool generally does when it builds a tree) is often not what you want for later compiler steps (that's where the A in AST comes from, you want an abstracted view of the syntax, not a 1-1 view).

[vitiral]

Gotcha, I think I understand. I don't personally understand the point of parsing without simultaneously creating some kind of tree (even if it needs to be modified in later stages).

I think I'll be making my own minimal Lua implementation for my Civboot project. You may or may not be interested: https://github.com/civboot/civboot/blob/main/blog/0013-civboot-languages.md

[vitiral]

I went on a bit of a code safari and have a conclusion: runtime type checking should only be for concrete types: it should not support generics or function types. See my write up here: civboot/civlua#2

If teal were to export runtime types I would recommend the constraints only support specific values (either native "string" etc types or a specific metatable). Anything that is generic/interface/etc on a type/record should be effectively an Any type at runtime (type erasure).'

I tried to implement such a system and it's complexity and runtime cost outweighs it's utility. On the other hand, records have very little complexity (especially considering the gains of debugging) and absolutely zero runtime cost since you can selectively enable runtime checks and runtime checks were only happening in the constructor/__index/__newindex methods which can easily be swapped depending on whether checking is enabled or not.

Just some thoughts if you do decide to go that route. Thanks for the great discussion!