Could we add the ability to define some user-defined "processor/transformer" in the compiler?

[skunkiferous]

This slightly relates to discussion "Exporting type annotations to lua?" #613

What I mean is, once the teal file has been parsed, and turned into an AST, but before this AST is turned into Lua, allow a "user-defined" function to have a look at the AST, and optionally modify it. I'm imagining this as a "compiler flag" rather that something "in" the teal files themselves.

Use-Case:

1. Using the type info to add "comment nodes" about the record/enum/.. types in to generated Lua, as suggested in Discussion Official website? #163
2. Generate a "doc file" (Discussion "document generation from code" document generation from code #396)
3. Generate a machine-parseable XML/JSON file about the types, for example, to generate a "schema" used to validate input data files
4. Adding data-fields with the type info, (fields in record, params of functions, ...) to Lua, so you can perform "reflection" at runtime on the type, rather that a specific instance of it.
5. Optionally adding "arg checks" to functions, based on the type info
6. Optionally checking that only "specified" fields can be modified, and only with the right type of values. I'm not sure if this is possible by only modifying the AST; maybe the Lua gen would need to be modified too.

And probably more usages.

I've had a quick look at the compiler. It's rather "lightly documented", but I think this could run "just before calling pretty_print_ast()".

Unfortunately, I can't judge if modifying the AST would "mess something up" if it happens "while files are processed" rather than "after everything was validated".

Also, I can imagine that's not a can of worms you would want to open, as you'd then have to deal with questions relating to "my AST transformer breaks the compiler".

[skunkiferous]

As a simple example, I like to use CSV files as input. Even if I hard-wire the fields of the records I am using to read the file in the import code, how do I know that a string is a valid value for my enum. I really don't want to list the tags again after I did it in the enum itself. I'd like to "ask" the enum to check, and I'd like to therefore be able to generate such a method automatically for my enums, since TL doesn't do it for me.

[lenscas]

sounds like a combination of Myriad for F# (https://github.com/MoiraeSoftware/myriad) and Rust's procedural macros. Specifically the attribute macro kind.

[hishamhm]

Also, I can imagine that's not a can of worms you would want to open, as you'd then have to deal with questions relating to "my AST transformer breaks the compiler".

Actually I would be more worried about the opposite: questions like "your changes to the compiler broke my AST transformer". Exposing the AST would limit my ability to freely evolve the compiler internals.

[lenscas]

while that is true, a way to "query" the project for its types, exposed functions, etc would not necessarily have this problem and would still offer a solution to the 2 biggest ones.

It would solve number 3

Generate a machine-parseable XML/JSON file about the types, for example, to generate a "schema" used to validate input data files

It could allow you to get something like 4 going

Adding data-fields with the type info, (fields in record, params of functions, ...) to Lua, so you can perform "reflection" at runtime on the type, rather that a specific instance of it.

[hishamhm]

The good news is that we already have a good amount of infrastructure to allow for that: tl types does dump a JSON format with type information for a file. What we don't have yet is an ergonomic programmatic interface for that data.

[skunkiferous]

I wasn't aware of the "tl types" command. The "Running" section of the TL readme doesn't mention it. I tried it on some small file, and I can't really make any sense of the output. If the format is documented somewhere, I did not find it.

I came to an alternative idea. If having access to the internal AST is bad, because modifying the compiler would break "user code" depending on the AST, but OTOH, the "types" format is hard to parse, why not use an existing simple format instead.

Teal already has "definition files". The format is more complicated then JSON, but easily understood by humans. And we ideally want an "AST format", that is easily understood by humans, so that users can easily use it. So, why not define a simple grammar that matches the "definition files", a "public" data-model to match it, and a small library that does the parsing and returns the model. It does not need to be tied to the compiler impl, and does not need to "validate" anything. As long as a file can be parsed, then the model can be generated. Now, the we "only" need the tl compiler to optionally export the parsed/compiled file as a (much simpler) definition file.

Now, in my Teal code, I could just load that "definition parser" library, read my own generated "definition file", and extract the "reflection data" from it, OR from any other module I want to use, even if they were not even written in Teal, as long as a "definition file" was created for that module, which is likely since I'm using it in Teal itself.